12 Replies Latest reply: Mar 15, 2014 2:14 PM by Dave Cramp RSS

    Jumbo frames end-to-end?  Or not?

    Scott Baldridge Newbie

      Seeking advice on this one.  I have jumbo enabled on my CS240, the switches, and my storage stack on the ESXi 5.1 hosts/vDS are all using jumbo.  I created a new port group for iSCSI VM traffic and it will use jumbo.


      Here is my question:  I have added two nics to a VM and configured them for iSCSI MPIO but I don't know if the Win2008R2 VM will need to use jumbo or not?  I searched here and didn't see an answer.




      Thanks in advance.


        • Re: Jumbo frames end-to-end?  Or not?
          Brandon Grieve Adventurer

          Hi Scott,


          You should enable jumbo frames across the entire network stack (host included) to realize the benefit that jumbo frames can offer. You can enable/use jumbo frames in Windows 2008 and later, and you will need to enable it on the network adapter(s) themselves. Here is a link to a basic article that outlines the steps involved:


          Q. How do I enable jumbo frames? | Windows content from Windows IT Pro


          For what it's worth, I would also submit that jumbo frames does not *always* equate to better performance. In most of the customer environments I'm exposed to, jumbo frames seems to be more common (and beneficial) where 10Gb Ethernet is in use and much less used in 1Gb environments.


          Hope this helps!


          Brandon Grieve

          Nimble Storage

          • Re: Jumbo frames end-to-end?  Or not?
            julez Adventurer

            I'd like to see a couple benchmark comparisons of Jumbo frames vs standard 1500 MTU on Nimble.

              • Re: Jumbo frames end-to-end?  Or not?


                I've done some testing in the past with jumbo vs standard, and found there's little (couple of percent) performance gain from using jumbo frames. Personally I recommend against jumbo frames for reasons of simplicity. I have done installs at 2 big environments where we found out that not all of the layers (a NIC in one environment, a switch port in another) were configured for jumbo frames. This resulted in lower then expected performance as you can imagine.


                Keep It Simple!


                • Re: Jumbo frames end-to-end?  Or not?
                  Nick Dyer Navigator

                  I have to +1 Arne Polman here. Unless you're looking to absolutely stress a 10GbE network with major sequential IO, Jumbo Frames will give little performance benefit vs the relative management & support overhead it takes to administer and troubleshoot the environment when it goes wrong - reason being it acts like a "house of cards"; one misconfiguration in the environment and the whole thing falls down.


                  Jumbo Frames should only really be considered on 10GbE networks (1GbE rarely sees any improvement), and only after the environment has been run at 1500 MTU to check the performance and latency first. In fact Support's best practice is to install & configure on 1500 MTU, and then look at switching to Jumbos only if there may be a good case to do so.


                  I think of KISS - Keep It Simple, Stupid

                • Re: Jumbo frames end-to-end?  Or not?
                  Alex Goltz Adventurer

                  Add me to the list of jumbo frames on iSCSI Initiators on 10Gbit.  Our heaviest workloads are SQL 2012 sequential reads.  At times our data warehouse VM will get close to 2 GB/sec if the cache hit ratio is near 100%.

                  This is probably the only scenario we have that gets the more mileage out of the additional payload size.

                  I would agree with Nick Dyer.  Test 1500 first, then move to 9000 if you really feel the calling to do so.


                  To whomever is interested:

                  One command I like to run on the guest VM to make sure the jumbo frame settings are correct in all areas is:  ping <array iSCSI IP> -f -l 8972                Don't use the iSCSI discovery IP. Use one of the actual data IPs.  And use a lowercase "L" for the second switch.

                  • Re: Jumbo frames end-to-end?  Or not?
                    Dave Cramp Newbie

                    I find it's best to explain Jumbo Frames as a way to reduce workload of switches and network adapters rather than something that's really going to increase performance.


                    Increasing the packet size can reduce the number of packets required to send traffic, which reduces the workload of switches and other network equipment (adapters etc..) especially in a TCP world where the equipment runs checksums and other sequence checking on every packet.


                    This is a lot more visible on 10Gb networks where the number of packets per second will typically be much higher than that of a 1Gbit network.