14 Replies Latest reply: Sep 6, 2016 12:42 AM by Rick van Vliet RSS

    Connecting Nimble Storage to Cisco UCS

    mmclaughlin Wayfarer

      The Nimble Storage array can be connected directly to the Cisco UCS Fabric Interconnect (FI) using 10G cabling.

      The array is configured as a UCS Appliance Port.


      In a dual FI (A/B) configuration, you need to configure separate VLANs - one for each FI. For example, VLAN iSCSI-A for FI-A and VLAN iSCSI-B for FI-B.

      This results in configuring the Nimble Storage NICs into a dual subnet mode of operation. For proper failover management, it is recommended to connect array 10G NIC TG1 to FI-A and TG2 to FI-B.

        • Re: Connecting Nimble Storage to Cisco UCS

          Mike -- when would a customer make the decision of incorporating a switch such as the CIsco Nexus in their storage network? Also is connecting through the UCS FI always the recommended practice?

          • Re: Connecting Nimble Storage to Cisco UCS
            Dmitriy Sandler Adventurer

            Per the ever knowledgeable Huy Duong:


            Yes it is absolutely supported. You will need to assign the FI ports as “Appliance Ports” or in some Cisco Documentation, it’s also referred to as “NAS Ports”, the two terms can be used interchangeability


            Here are a list of Gotchas you need to know about:

            1. If you are going to do Boot from san, Make sure you create an IQN pool that has the word “microsoft” in them for bare metal windows install or VSS will fail.
            2. Under the “LAN” Tab, either create a “Network Control Policy” or edit the default of the “Action on uplink fail” to Warning, or it will not come up.
              1. UCS SM.jpg
            3. You will need to have two subnets, no exceptions if you want to plug into the FIs. The FIs operate as two separate switches and really don’t know about each other’s MAC address. Therefore you need to create two Vlans, each vlan to an FI.
            4. Create two NICS for your Service Profile, each one to a different FI. Disable Failover of the NIC (since the vlan will not exist on the other FI), MPIO will take care of this.



            • Re: Connecting Nimble Storage to Cisco UCS

              Can you elaborate on the "dual subnet" mode of operation?  I don't see this configuration how-to in the user guide.  It states to point initiators to the discovery IP address.  But I assume you would need two discovery IP addresses, or not use it in a dual subnet mode?  Thanks



                • Re: Connecting Nimble Storage to Cisco UCS
                  mmclaughlin Wayfarer

                  In a dual subnet model for ESXi connectivity to Cisco UCS, we are actually talking about dual data subnets so actually there may be 3 networks in you environment (management, data-A and data-B) where the A & B data subnets map onto the UCS Fabric Interconnects A & B respectively. On the Nimble array, there is only one officially defined iSCSI discovery address.


                  You have a couple of options - each with considerations:

                  1. perform iSCSI discovery on only one of the data subnets - this means new discovery may be limited should the iSCSI discovery network is down - data path won't be disrupted - just new provisioning
                  2. perform iSCSI discovery on the management subnet - you will need to make sure the host has connectivity to all three subnets somehow which may have undesired security considerations for access control
                  3. allow iSCSI discovery on the data network IP addresses directly - this is currently supported through an undocumented ability and subject to change in future releases if needed


                  For our current Cisco/VMware/Nimble reference architecture, we use method #1 above as shown in this screenshot:




                • Re: Connecting Nimble Storage to Cisco UCS

                  We just implemented a new UCS system along with 2 Nimble arrays last week. We used the appliance ports exactly as described. We did run into one gotcha during the process. We decided to use Jumbo Frames and found that we had to implement a QOS policy on the UCS with a MTU of 9216 in order for the connections to work properly.

                  4-30-2013 9-23-31 AM.jpg

                  • Re: Connecting Nimble Storage to Cisco UCS
                    Jay Zitcovich Newbie



                    I would classify the iSCSI traffic in the Gold class and add a nic for iSCSI traffic.  Build and apply a vnic policy and apply it to the iSCSI nic.


                    With the configuration shown from UCS Manager, client and iSCSI traffic are using the same class.. less than desirable from a performance standpoint.

                    UCS will not be able to differentiate between user traffic and iSCSI as shown in the configuration above..



                    • Re: Connecting Nimble Storage to Cisco UCS
                      Don Ngo Newbie

                      Hi Gents


                      Sorry to dig up an old thread but I thought my question is related to the networking setup of Nimble & Cisco UCS.


                      I am currently implementing a CS700 (with 4 x 10Gb SFP+ on each controller) for two sites and plan on replicating between them.


                      Currently the Nimble OS only allows replication over one subnet. My plan was to create a third VLAN purely for replication and tag this to all four 10GB interfaces on the controller. This VLAN will be a global VLAN on both fabrics.


                      Can anyone see if this is a workable solution?


                      Thanks in advance


                        • Re: Connecting Nimble Storage to Cisco UCS



                          It should technically work, as VLAN tagging allows you to share physical ports among multiple subnets. That being said, I don't know that I would necessarily tag all the physical ports with the replication VLAN. Under heavy replication load, this could potentially cause latency for iSCSI traffic and vice-versa. I would recommend picking 1 or 2 ports for replication and leaving at least 2 dedicated for only iSCSI.


                          That's my opinion, and others may have different ones.


                          Brandon Grieve

                          • Re: Connecting Nimble Storage to Cisco UCS
                            Rick van Vliet Adventurer

                            Hi Don,


                            This would work, we are using something similar, we replicate using a separate routed network to other datacenters and have tagged the replication vlan to to the array's NIC ports.

                            We are using all of the ports as well, but I can see Brandon's concern, setting a bandwidth limit might help in case you run into the latency issues on you iSCSI traffic.

                            For us the replication has never given any latency issues.