Adam Herbert

Importance of Path Change Settings in VMware

Discussion created by Adam Herbert Employee on Mar 12, 2013
Latest reply on Mar 21, 2014 by Nick Dyer
Branched to a new discussion

With 1 gig networks it is important to tune per path settings for maximum throughput. The default of 1000 IOs per path can cause micro bursts of saturation and limit throughput. I've done a fair amount of testing and found that the best setting is to actually change paths based on the number of bytes sent per path. The reasoning is that it can be detrimental to change paths too often for small block IO. Setting the path change to bytes optimizes for both.

 

Here is a real world example from a demo I recently conducted. This was done with 4 1G interfaces on both the Nimble array and the ESX host connected through a Cisco 3750X stack.

 

SQLIO prior to path optimization :

ServerToolTest DescriptionIO/sMB/sAvg. Latency
SQL-06SQLIORandom 8k Writes, 8 threads with 8 qdepth for 120 sec12375974ms
SQL-06SQLIORandom 8k Reads, 8 threads with 8 qdepth for 120 sec144561133ms
SQL-06SQLIOSequential 64k Writes, 8 threads with 8 qdepth for 120 sec213013329ms
SQL-06SQLIOSequential 64k Reads, 8 threads with 8 qdepth for 120 sec214713429ms

 

SQLIO after path optimization :

ServerToolTest DescriptionIO/sMB/sAvg. Latency
SQL-06SQLIORandom 8k Writes, 8 threads with 8 qdepth for 120 sec268822101ms
SQL-06SQLIORandom 8k Reads, 8 threads with 8 qdepth for 120 sec289642261ms
SQL-06SQLIOSequential 64k Writes, 8 threads with 8 qdepth for 120 sec75244708ms
SQL-06SQLIOSequential 64k Reads, 8 threads with 8 qdepth for 120 sec74744678ms

 

Notice the large improvement in not only throughput but also the reduction in latency. The latency in the first test was due to the saturation of the 1G links.

 

The optimization is done with the following command from the ESX 5.x console:

 

for i in `esxcli storage nmp device list | awk '/Nimble iSCSI Disk/{print $7}' | sed -e 's/(//' -e 's/)//'`; do

  esxcli storage nmp psp roundrobin deviceconfig set -d $i -I 0 -t iops;

done

 

This can also be done with a PowerShell script that is posted here :

Set VMware RoundRobin PSP through PowerCLI

Outcomes