the MGMT interfaces failover between themselves to phone home. If an FI was to drop, the MGMT ports failover - but the CTRL does not failover. Best practice is to put port eth1 into FI0 and port ETH2 into FI1. If you choose not to put eth1 into the same FI - it works fine. But a loss of the ethernet port on eth1 would the array to failover CTRL after several heartbeat failures and if the other eth0 port was up.
most small nimble deployments don't attached eth1 and eth2 up for failover.
If you were building a fresh UCS setup, what would you do?
I have a huge problem right now with Nimble's KB-000202 document. This is apparently THE "Best Practice" document for a UCS build, and they only do one management interface in "Failover" mode, aside from other errors like MTU settings and upstream switch settings. What I really need to know, is what is going to happen when I have to reboot an FI? Are my host heartbeats going to freak out my HA?
One thing to keep in mind though. If you are using iSCSI boot from SAN volumes with VMWare hosts the FI failover will raise alerts on the ESXi hosts that require restarting the management agents to cleanup. It puts the boot volumes in RO mode until you do this but the VM's continue to run normally in memory the whole time.
This is the alert we get when we update firmware on the FI's and fail over to the second one with our iSCSI mounted storage.
Lost network connectivity on virtual switch
"iScsiBootvSwitch". Physical NIC vmnic2 is down.
9/17/2015 8:31:14 PM
Restarting the agents using one of the methods in this article will bring everything back to normal in VMware and no downtime or issue on the hosts.
Took us a while to track this down.