Here are some results of testing performance of different Docker Inter Container Communication (ICC) techniques.
Techniques I tested:
| iptables | Routing settings that give containers externally visible IP address as described here: blog.codeaholics.org/2013/giving-dockerlxc-containers-a-routable-ip-address/ |
| pipework | github.com/jpetazzo/pipework – a tool for assigning external IP addresses to containers. It has two modes: using macvlan interface and using veth pairs. I used the second one – veth pairs. |
| Docker link | Docker feature for inter container communication (ICC). It doesn’t assign containers external IPs. docs.docker.com/userguide/dockerlinks/ |
| Open vSwitch |
openvswitch.org Used version 2.1.0 with kernel support. |
Tested on Dell Poweredge D320 server with Xeon 1.8GHz, 4GB (1333MHz) RDIMM, 7200RPM SATA HDD
OS Ubuntu server 12.04.4 LTS
kernel 3.8.0-39
Docker version 0.11.
Network performance was tested with iperf with the following client command:
iperf -c $ServerIP -P 1 -i 1 -p 5001 -f g -t 5
Performance in Gbit/s, average for one container.
The following 3 setups was tested:
- One server container and multiple client containers with 1 iperf process in each container.
- Multiple servers and multiple clients. 1 iperf process in each container.
- One server container with one iperf server, multiple containers with multiple iperf clients in each container.
One server and multiple clients
ICC performance between one container with one iperf server and multiple containers with one iperf client each. Average performance per container in Gbit/s.
| client containers | 1 | 2 | 4 | 8 | 16 | 20 |
|---|---|---|---|---|---|---|
| iptables | 10.0 | 8.0 | 4.2 | 2.1 | 1.0 | 0.8 |
| pipework | 11.6 | 8.3 | 5.1 | 2.7 | 1.4 | 1.0 |
| link | 12.0 | 8.1 | 4.9 | 2.5 | 1.2 | 1.0 |
| ovs | 13.2 | 11.0 | 6.6 | 3.3 | 1.8 | 1.4 |
1 server – multiple clients
Multiple servers and multiple clients
ICC between multiple containers each with one iperf server inside, and multiple containers with one iperf client each. Client number i connects to the server number i (mod n), where n is the number of servers.
Below are the average performance results in Gbit/s.
| servers | 1 | 2 | 4 | 20 |
|---|---|---|---|---|
| clients | 1 | 2 | 4 | 20 |
| iptables | 10.0 | 8.3 | 4.3 | 0.8 |
| pipework | 11.6 | 10.0 | 5.5 | 1.3 |
| link | 12.0 | 9.1 | 5.1 | 1.1 |
| ovs | 13.2 | 10.8 | 5.6 | 1.4 |
Multiple servers and multiple clients with 1 iperf process inside.
One server and multiple containers with multiple iperf clients inside
Average performance per container in Gbit/sec.
Number of containers x number of iperf clients in one container
| 1 x 1 | 1 x 2 | 1 x 4 | 1 x 16 | 2 x 1 | 4 x 1 | 16 x 1 | 2 x 2 | 4 x 4 | 16 x 16 | |
|---|---|---|---|---|---|---|---|---|---|---|
| pipework | 11.16 | 9.81 | 3.38 | 1.07 | 9.33 | 4.99 | 1.52 | 4.63 | 1.20 | 0.04 |
| link | 12.10 | 7.70 | 3.30 | 0.80 | 8.20 | 4.90 | 1.20 | 4.30 | 1.10 | 0.00 |
| ovs | 13.06 | 11.14 | 6.36 | 1.54 | 11.24 | 6.63 | 1.99 | 6.40 | 1.50 | 0.06 |
Conclusion
Running iperf clients in different containers gives better performance compared with the same number of clients running in one container (compare 1×4 and 4×1).