You only have to configure IP Addresses for the active ports - they will fail over from controller to controller in the event of a failover (eth1 to eth1, eth2 to eth2, and so forth).
Have you plugged in the same eth ports on controller A and B into the same switch? If you only have eth1 configured and plugged in on controller A, yet you have eth2 plugged in on controller B, then failover will not occur because you haven't mirrored the network configuration correctly.
You should have a configuration guide in the array box itself, and you can also get the Installation guide on the "Downloads" section of the support site (sign up here: https://infosight.nimblestorage.com).
If in doubt, give Support a call - they'll be happy to help out.
It all sounds as though it should be correct. You are right - Controller B assumes the same identity as Controller A (with respect to IP addressing and mgmt/data port distribution).
When you say you lost connectivity all together, does this mean management & servers' visibility of LUNs (or just one or the other)?
What do you mean by "ports are different for each controller"? Can you double check your VLAN configs on the switch, just to make sure that the ports from Controller B are in the same VLANs as their partner ports on Controller A? For example:
Controller A, Eth0 (management) - 10.1.1.50 --- VLAN1
Controller A, Eth1 (data 1) - 192.168.1.11 --- VLAN2
Controller A, Eth2 (data 2) - 192.168.1.12 --- VLAN2
Controller A, Eth3 (data 3) - 192.168.1.13 --- VLAN2
Controller B, Eth0 (management) - 10.1.1.50 (inherited) --- VLAN1
Controller B, Eth1 (data 1) - 192.168.1.11 (inherited) --- VLAN2
Controller B, Eth2 (data 2) - 192.168.1.12 (inherited) --- VLAN2
Controller B, Eth3 (data 3) - 192.168.1.13 (inherited) --- VLAN2
iSCSI Discovery address will be tagged to the management port.
It does sound like a networking config discrepancy to me, easily done though. Have you checked AutoSupport works from the array? That'll be useful for Support as/when diagnosing your issue.