Wednesday, July 28, 2010

Configure LACP etherchannel on Xen Server 5.5 with Cisco switch

The default bonding mode for Citrix XenServer 5.5 is 1, or active-backup (slb).  This mode only provides fault tolerance, since only one NIC in the bond is active at a time.  For more heavily utilized production systems, an active-active configuration is preferable.  The two bonding modes that provide full send/receive balancing across all NICs are 4 (802.3ad) and 6 (alb).  Mode 4 provides link aggregation according to the 802.3ad specification, and requires an etherchannel to be configured on the switch.
The following is an example of how to configure an LACP/802.3ad bond with XenServer 5.5 and a Cisco 3750 switch running IOS.  
First, configure your switchports for an LACP etherchannel.  Note: in this case, we are making the etherchannel/Bond a trunk port which will only allow VLANs 2 - 11.  We have two NICs from our XenServer that make up the bond0 interface, plugged int Gi1/0/1 and Gi2/0/1.
// Configure the EtherChannel first
interface Port-channel1
description Etherchannel Team for XenServer1
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport trunk allowed vlan 2-11
switchport mode trunk
spanning-tree portfast
spanning-tree bpduguard enable
!
// now configure the individual switchports
interface GigabitEthernet1/0/1
description XenServer1 - eth0
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport trunk allowed vlan 2-11
switchport mode trunk
speed 1000
duplex full
channel-protocol lacp
channel-group 2 mode active
!
interface GigabitEthernet2/0/1
description XenServer1 - eth1
switchport trunk encapsulation dot1q
switchport trunk native vlan 2
switchport trunk allowed vlan 2-11
switchport mode trunk
speed 1000
duplex full
channel-protocol lacp
channel-group 2 mode active
!
// now make sure things are configured properly:
show etherchannel 2 summary
// this should output something like the following:
...
Number of channel-groups in use: 4
Number of aggregators:           4
Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
2      Po2(SU)         LACP      Gi1/0/1(P) Gi2/0/1(P)
// Note, you may see the two nics in (I) instead of (P) until you configure and reboot your XenServer
Now, make the following changes to your XenServer host.
1. Create a Bonded interface from XenCenter.  Select your server, then the NICs tab and choose 'Create Bond'.  Add the PIFs you want
2.  The Citrix recommended option.  Run the following command on the XenServer host console.  Make sure to use the UUID of the Bonded interface.
    xe pif-param-set uuid= other-config:bond-mode=802.3ad
    (note: to find your PIF's uuid: xe pif-list host-uuid=<host uuid> )
    (note: to find your Host's uuid: xe host-list )
2a. Some of our users have reported problems getting the offical recommendation working, and suggested an alternate method:
On the XenServer host, edit the following file:
    /opt/xensource/libexec/interface-reconfigure

find the following line:
        "mode":   "balance-slb",
and replace it with
        "mode":   "802.3ad",
Then save and restart the XenServer.  This will change the default bonding method for all Xen bonds to "802.3ad".
3. To make sure things are working as expected, run (replace X with your bond number):
    cat /proc/net/bonding/bondX

    You should see something similar to this:
Ethernet Channel Bonding Driver: v3.1.2 (January 20, 2007)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 31000
Down Delay (ms): 200
802.3ad info
LACP rate: slow
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 17
        Partner Key: 2
        Partner Mac Address: <Switch's etherchannel MAC address>
Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: <eth0's MAC address>
Aggregator ID: 1
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: <eth1's MAC address>
Aggregator ID: 1


This document has been coppied "AS IT IS" from here

3 comments:

Anonymous said...

Just read your post.
You might also want to check-out this:
http://forums.citrix.com/message.jspa?messageID=1465624


With 5.6 it seems to be:
xe pif-param-set uuid= other-config:bond-mode=4
xe pif-param-set uuid= other-config:bond-lacp_rate=1

flrichar said...

Great article! I've done similar things with other environments. LACP/802.3ad is awesome.

Double-check your config though. You establish PortChannel1 then you add interfaces to PortChannel2.

flrichar said...

Great article! I've done similar things with other environments. LACP/802.3ad is definitely awesome.

I would double-check your config tho, for the folks playing at home who copy your config word for word.

In the first paragraph you establish PortChannel1, but then proceed to add interfaces to PortChannel2.