Link aggregation is something that has greatly interested me the last few months and I’ve been anxious to play with it on my own network setup. For those not familiar with link aggregation there is a lot of good documentation on Wikipedia to get caught up to speed. In my lab I wanted to setup link aggregation on a box with a NC3134 which is a dual NIC (2x 10/100 full duplex ports). I couldn’t do proper 802.3ad because I don’t have a switch which supports it…yet. However, since I am running Ubuntu server I found I could still set it up using a different mode. The mode I decided to configure the bond as is mode 0 otherwise known as the round-robin bond mode. This type is described as “Round-robin policy: Transmit packets in sequential order from the first available slave through the last. This mode provides load balancing and fault tolerance.” (Linux Horizon)
The setup for bonding the NICs is actually quite simple.
sudo apt-get install ifenslave sudo nano /etc/network/interfaces
Comment out your current configuration lines using the “#” before all lines. The following was my setup for the bond address configuration:
auto bond0
iface bond0 inet static
address 192.168.2.200
netmask 255.255.255.0
broadcast 192.168.2.255
gateway 192.168.2.1
slaves all
bond-mode 0
bond-miimon 100
This configuration will use all of my network devices in the bond (which in this case is just 2) and give the IP address 192.168.2.200. Note that I specified bond-mode 0. There are several other bond types (some of which require a 802.3ad capable switch) and it would be worth your while to read up on them to find which would be best for you.
Another thing to note in this configuration is that I used bond-miimon 100. This means that the links will be checked for failures every 100 milliseconds. Setting this value to 0 disables checking for link failure.
Save and exit the configuration file. From here you can take the interfaces down by issuing:
sudo ifdown eth0 sudo ifdown eth1
At this point I connected the 2nd Ethernet cable and restarted the networking services
/etc/init.d/networking restart
Simply type ifconfig and you should see bond0 in the list and listed as UP BROADCAST RUNNING MASTER MULTICAST and all others as UP BROADCAST RUNNING SLAVE MULTICAST. Next we will test performance of the connection to ensure that the bond is functioning as it should.
Using iperf I can test the bandwidth utilized between the server and client in memory (which takes the drive write bottleneck out of the equation). To setup iperf see my previous article on using iperf.
I have 192.168.2.101 setup as the server and 192.168.2.200 (the bonded NICs) as the client. I issue the following command to test my connection:
Iperf –c 192.168.2.101 –i 2
The output was as follows:
------------------------------------------------------------ Client connecting to 192.168.2.101, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.2.200 port 33878 connected with 192.168.2.101 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 2.0 sec 44.8 MBytes 188 Mbits/sec [ 3] 2.0- 4.0 sec 44.9 MBytes 188 Mbits/sec [ 3] 4.0- 6.0 sec 44.9 MBytes 188 Mbits/sec [ 3] 6.0- 8.0 sec 43.9 MBytes 184 Mbits/sec [ 3] 8.0-10.0 sec 44.8 MBytes 188 Mbits/sec [ 3] 0.0-10.0 sec 223 MBytes 187 Mbits/sec
You can see that now the connection is reaching ~188Mbits out of the theoretical 200Mbits that the bond provides (remember we are using 2x 100mbit lines). This is exactly the result we want.
Next, I wanted to test the fault of one line. I disconnected one one of the cables from the dual NIC and ran the iperf connection test again.
------------------------------------------------------------ Client connecting to 192.168.2.101, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.2.200 port 33879 connected with 192.168.2.101 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 2.0 sec 22.5 MBytes 94.4 Mbits/sec [ 3] 2.0- 4.0 sec 22.4 MBytes 94.1 Mbits/sec [ 3] 4.0- 6.0 sec 22.4 MBytes 94.0 Mbits/sec [ 3] 6.0- 8.0 sec 22.5 MBytes 94.2 Mbits/sec [ 3] 8.0-10.0 sec 22.4 MBytes 94.0 Mbits/sec [ 3] 0.0-10.0 sec 112 MBytes 94.2 Mbits/sec
You can see that the bandwidth has fallen back down to 94Mbits but the line still worked correctly without any traffic loss.
That’s it! Pretty simple to setup and test the bond. If you are creating a bond purely for the bandwidth increase keep in mind that system bottlenecks can still hinder performance – your hard drive write speed being the most likely.
With such a simple setup for this networking task it shows another benefit Linux servers can provide in comparison to Windows servers which (to my knowledge) do not natively support link aggregation.
The question I asked myself after this setup was: How does the 802.3ad configuration (mode 4) differ from mode 0 which doesn’t require switches with 802.ad support? After some further research I found that while mode 0 uses sequential transmission, it can often receive packets out of order which creates the need for segment retransmission (as mentioned here). It seems that mode 4 is able to receive all packets in order which produces an overall increase in efficiency. I would be curious to see some performance benchmarks between the two and plan to do so once I have a managed switch – so stay tuned!