Milestone with NIC teaming and creating Port Channel for ESX host

Our new Network engineer has created port Channel for extra bandwidth and additional redundancy. He created two port channel 11 and 12 and trunked xxx to yyy vlan on those interface. But he never mention this to me and I was hard time figuring this out heck. Finally couple of folk Scott Lowe and René Jorissen came for my rescue. What was happening was once he set the port channel on the switch I did not made any changes on ESX (Yeah Damn VCP stupid I am ) So here is what he set on switch side for two different ESX host. Remember I am using two NIC for VM networking vSwitch.
!
interface Port-channel11
description **ESX PortChannel11**
switchport trunk encapsulation dot1q
switchport trunk allowed vlan xxx-yyy
switchport mode trunk
!
interface Port-channel12
description ** ESX Port Channel **
switchport trunk encapsulation dot1q
switchport trunk allowed vlan xxx-yyy
switchport mode trunk
!
interface GigabitEthernet1/0/1
description **ESX CHAN 11**
switchport trunk encapsulation dot1q
switchport trunk allowed vlan xxx-yyy
switchport mode trunk
channel-group 11 mode on
spanning-tree portfast


interface GigabitEthernet3/0/1
description **ESX CHAN 11**
switchport trunk encapsulation dot1q
switchport trunk allowed vlan xxx-yyy
switchport mode trunk
channel-group 11 mode on
spanning-tree portfast


!
interface GigabitEthernet2/0/1
description **ESX CHAN 12**
switchport trunk encapsulation dot1q
switchport trunk allowed vlan xxx-yyy
switchport mode trunk
channel-group 12 mode on
spanning-tree portfast


interface GigabitEthernet4/0/1
description **ESX CHAN 12**
switchport trunk encapsulation dot1q
switchport trunk allowed vlan xxx-yyy
switchport mode trunk
channel-group 12 mode on
spanning-tree portfast

So what I came to know is that NIC teaming has to be done for both inbound and outbound traffic. Once you set "channel-group <NO> on " which means neither pagp nor lacp are used .For pagp you have the options mode auto and mode desirable. If both sides configure mode auto, the channel won’t come up. So at least one end should be configured with mode desirable. For lacp you have the options passive and active. Like pagp, if both ends are configured with mode passive the channel won’t come up. At least one end should be configured with mode active

Both PAGP(Port aggregation protocol ) and LACP (link aggregation control protocol) are EtherChannel configure mode.Little summary about it can be found at end of this document

Some says the Load balancing mechanism has to match on both ESX side as well as Switch side but according some except there is also exception to this where working config was ESX uses the IP hashing load-balancing algorithm and Cisco Catalyst 3750 switches used src-mac-addressing. But in a layer 3 environment the source mac-address is mostly the same, so the switches only use one link from a configured port-channel, while ESX could use multiple connections.

If 802.3ad is used on the physical switches, then it will be required to use “Route based on ip hash” on the vSwitches in order for connectivity to work. Keep in mind that this will only help improve the distribution of traffic across the links, not necessarily improve the throughput of any single point-to-point connection.

To check the load balancing at Switch side

• Please check this from enable mode "show etherchannel load-balance" This will report the current load balancing algorithm in use by the switch

• This need to match this from host side and current option for load balancing mechanism at ESX side are
  a) Route based on the originating virtual port ID
  b) Route based on IP hash
  c) Route based on Source Mac HASH
  d) Use explicit fail over order
  
  Incase we have to set based on our own choice please set the load balancing mechanism at switch side from global config mode (This may differ based on IOS version or switch version)
  

port-channel load-balance src-dst-ip (to enable IP-based load balancing)
port-channel load-balance src-mac (to enable MAC-based load balancing)

VMware consultants always choose Route based on IP hashes as load-balancing algorithm. This means that load-balancing happens on layer 3 of the OSI model (source-destination-IP)

Once this is done verify it from ESX host
1. esxtop [enter]
2. s2 (schedule interval of 2 seconds) [enter]
3. n [network]

The load should be spread fairly even across the different switch ports en vmnics

EtherChannel configure mode with PAgP

PAgP facilitates the automatic creation of EtherChannel links by exchanging packets between channel-capable ports. The protocol learns the capabilities of port groups dynamically and informs the neighboring ports.

After PAgP identifies correctly paired channel-capable links, it groups the ports into a channel. The channel is then added to the spanning tree as a single bridge port. A given outbound broadcast or multicast packet is transmitted out one port in the channel only, not out every port in the channel. In addition, outbound broadcast and multicast packets transmitted on one port in a channel are blocked from returning on any other port of the channel.

There are four user-configurable channel modes: on, off, auto, and desirable. PAgP packets are exchanged only between ports in auto and desirable mode. Ports configured in on or off mode do not exchange PAgP packets. For switches to which you want to form an EtherChannel, it is best to have both switches set to desirable mode. This gives the most robust behavior if one side or the other encounters error situations or is reset. The default mode of the channel is auto. Both the auto and desirable modes allow ports to negotiate with connected ports to determine if they can form a channel. The determination is based on criteria such as port speed, trunking state, and native VLAN. Ports can form an EtherChannel when they are in different channel modes as long as the modes are compatible. This list provides examples:


A port in desirable mode can successfully form an EtherChannel with another port
that is in desirable or auto mode.
A port in auto mode can form an EtherChannel with another port in desirable
mode.
A port in auto mode cannot form an EtherChannel with another port that is also in
auto mode, since neither port initiates negotiation.
A port in on mode can form a channel only with a port in on mode because ports
in on mode do not exchange PAgP packets.
A port in off mode cannot form a channel with any port.
Only the combination auto-desirable, desirable-desirable and on-on will allow a channel to be formed.

EtherChannel can be

Ports in the channel can only be assigned to one VLAN
Ethernet channels can be set up for fast Ethernet, Gigabit Ethernet, and 10 Gigabit
Ethernet.
Ethernet channels can indeed be set up as trunks.
Ports within a Fast Ether Channel need to have identical duplex and speed
settings.
Port Aggregation Protocol (PAGP) facilitates the automatic creation of Fast Ether
channels links.


The PAgP modes are explained below.

Off: PAgP will not run. The channel is forced to remain down.
Auto: PAgP is running passively. The formation of a channel is desired; however, it is not initiated.
Desirable: PAgP is running actively. The formation of a channel is desired and initiated.
On: PAgP will not run. The channel is forced to come up.

EtherChannel configure mode with LACP

The link aggregation control protocol (LACP) is defined in IEEE 802.3ad. Link aggregation control protocol data unit (LACPDU) is used for exchanging information among LACP-enabled devices.

After LACP is enabled on a port, the port sends LACPDUs to notify the remote system of its system LACP priority, system MAC address, port LACP priority, port number, and operational key. Upon receipt of an LACPDU, the remote system compares the received information with the information received on other ports to determine the ports that can operate as selected ports. This allows the two systems to reach agreement on the states of the related ports

When aggregating ports, link aggregation control automatically assigns each port an operational key based on its rate, duplex mode, and other basic configurations. In an LACP aggregation group, all ports share the same operational key; in a manual or static LACP aggregation, the selected ports share the same operational key.