29 Replies Latest reply: Mar 21, 2014 11:27 AM by Nick Dyer Branched to a new discussion. RSS

    Importance of Path Change Settings in VMware

    Adam Herbert Adventurer

      With 1 gig networks it is important to tune per path settings for maximum throughput. The default of 1000 IOs per path can cause micro bursts of saturation and limit throughput. I've done a fair amount of testing and found that the best setting is to actually change paths based on the number of bytes sent per path. The reasoning is that it can be detrimental to change paths too often for small block IO. Setting the path change to bytes optimizes for both.

       

      Here is a real world example from a demo I recently conducted. This was done with 4 1G interfaces on both the Nimble array and the ESX host connected through a Cisco 3750X stack.

       

      SQLIO prior to path optimization :

      ServerToolTest DescriptionIO/sMB/sAvg. Latency
      SQL-06SQLIORandom 8k Writes, 8 threads with 8 qdepth for 120 sec12375974ms
      SQL-06SQLIORandom 8k Reads, 8 threads with 8 qdepth for 120 sec144561133ms
      SQL-06SQLIOSequential 64k Writes, 8 threads with 8 qdepth for 120 sec213013329ms
      SQL-06SQLIOSequential 64k Reads, 8 threads with 8 qdepth for 120 sec214713429ms

       

      SQLIO after path optimization :

      ServerToolTest DescriptionIO/sMB/sAvg. Latency
      SQL-06SQLIORandom 8k Writes, 8 threads with 8 qdepth for 120 sec268822101ms
      SQL-06SQLIORandom 8k Reads, 8 threads with 8 qdepth for 120 sec289642261ms
      SQL-06SQLIOSequential 64k Writes, 8 threads with 8 qdepth for 120 sec75244708ms
      SQL-06SQLIOSequential 64k Reads, 8 threads with 8 qdepth for 120 sec74744678ms

       

      Notice the large improvement in not only throughput but also the reduction in latency. The latency in the first test was due to the saturation of the 1G links.

       

      The optimization is done with the following command from the ESX 5.x console:

       

      for i in `esxcli storage nmp device list | awk '/Nimble iSCSI Disk/{print $7}' | sed -e 's/(//' -e 's/)//'`; do

        esxcli storage nmp psp roundrobin deviceconfig set -d $i -I 0 -t iops;

      done

       

      This can also be done with a PowerShell script that is posted here :

      Set VMware RoundRobin PSP through PowerCLI