5 Replies Latest reply: Apr 23, 2013 6:09 AM by Frank Uittenbosch RSS

    The best way to test Linux ISCSI IOPs on Nimble?


      Hi All,


      We use Debian Squeeze Linux and Ubuntu 12.04 (both 64 bit/multiarch), and connect to our nimble via ISCSI multipathing. (2 x Gbit FD)

      The program i used to test the performance of the Nimble on Linux is: fio (flexible io tester)


      to start it i use a definition file from IOmeter, eg:

      fio iometer-file-access-server

      content of that file:


      description=Emulation of Intel IOmeter File Server Access Pattern
      # IOMeter defines the server loads as the following:
      # iodepth=1     Linear
      # iodepth=4     Very Light
      # iodepth=8     Light
      # iodepth=64    Moderate
      # iodepth=256   Heavy


      This gives us 15 K IOPS read and 15K  IOPS write.


      Does anyone have other programs to test this ? Any hints are welcome.



      Frank Uittenbosch

        • Re: The best way to test Linux ISCSI IOPs on Nimblestor ?
          Adam Herbert Adventurer

          FIO is a great tool to test performance. One thing to keep in mind is that at 15K IOPS at an 8K block size would saturate a 1G link.


          15K * 8KB = ~120MB/s = ~1Gbit/s


          You should check to make sure your multipath settings are configured properly and you may see some better performance. Check out this post about how you should set this up. MPIO Settings for Linux

            • Re: Re: The best way to test Linux ISCSI IOPs on Nimblestor ?

              Multipath is working, because when i rsync a file to the storage, and i run dstat, i see 220 MB/s getting transferred, whereas if i disconnect 1 cable, it is only 110.

              I will look into that config file BTW, because my current file is:


              multipath {
                              wwid 283d6b1680dbb029b6c9ce90074c1df7b
                              alias nimble_home
                              path_grouping_policy multibus
                              path_checker directio
                              path_selector "round-robin 0"
                              failback manual
                              rr_weight priorities
                              no_path_retry 5
                              rr_min_io 10


              Would this be a bad conf to have ?

            • Re: The best way to test Linux ISCSI IOPs on Nimblestor ?
              Ben Watson Adventurer

              For performance testing on Linux, my favourite tools are 'iogen' and 'Bonnie++'. There is also IOmeter, of course, which I'm pretty sure has a tar downloadable (though have only really used this on Windows).


              You can create lots of different workload simulations using 'iogen', as well as forking off child processes to give mega I/O transfers. Multipathing connections properly (it looks to me as though you have a good multipath.conf section - although consider using a smaller 'rr_min_io' setting perhaps and reviewing performance?) is, of course, key.


              Using iogen, I normally make [ext4] filesystems and mount to a local directory and target the I/O to that directory (with a separate directory for results), and vary the blocksize, child processes, read:write ratio and file size (N.B. every child process uses an individual file).


              Have had some phenomenal results using multiple clients spawning lots of child iogen processes onto Nimble storage.


              Other tests could be to use 'dd'...