0 Replies Latest reply: Aug 16, 2016 10:36 PM by Alexander Lawrence RSS

    How to: Setup NCM for Linux

    Alexander Lawrence Newbie
    Visibility: Open to anyone

      PacNW System Engineer for Nimble here! Noticed there was not a quick reference to how to setup NCM for Linux. Well, there is now!


      Step 1: Installing Linux NCM

       

      Note: Must be on one of the following: RHEL 6.5, RHEL 6.7, RHEL 7.0, and RHEL 7.1

      (I’ve tested on 6.7 and 7.1)

       

      Get NCM from InfoSight: https://infosight.nimblestorage.com/InfoSight/#software/Integration+Kits/Connection+Manager+%28NCM%29+for+Linux

       

      Install Dependencies:

       

      yum install device-mapper-multipath sg3_utils iscsi-initiator-utils

       

      Install mcm as root (or with sudo):

       

      chmod 755 [path-to-ncm]
      ./[path-to-ncm]
      
      
      

       

      ==============================================================================================================

      Post Installation Steps for User

      ==============================================================================================================

       

      Please add following LVM filters into "devices" section of /etc/lvm/lvm.conf
      
      preferred_names = ["^/dev/nimblestorage/", "^/dev/mpath/", "^/dev/mapper/mpath", "^/dev/[hs]d"]
      filter = [ "r|^/dev/nimblestorage/lower_tier_devices/.*|" ]
      global_filter = [ "r|^/dev/nimblestorage/lower_tier_devices/.*|" ]
      
      
      

       

      NOTE: For SAN Boot host, please re-generate initramfs using below command after adding LVM filters.

       

      dracut --add "multipath ncm" --force /boot/initramfs-`uname -r`-ncm.img
      where `uname -r` yeilds current running kernel version.
      
      
      

       

      Step 2: Configure iSCSI Network Adapters

       

      Setup Storage NICs on the linux guest (NICs that are on the same subnet(s) as your storage).

      Add these to your guest if you are virtual via VMWare/HyperV/Xen/KVM/etc... then configure them.

       

      Example:
      vim /etc/sysconfig/network-scripts/ifcfg-eno33559296
      
      TYPE=Ethernet
      BOOTPROTO=none
      IPV4_FAILURE_FATAL=no
      IPADDR="10.1.0.133"
      NETMASK="255.255.255.0"
      MTU="9000"
      IPV6INIT=no
      NAME=iscsi-2
      UUID=787d979e-a2ff-48eb-96ca-86e368eb254c
      DEVICE=eno50338560
      ONBOOT=yes
      
      
      


      You might need to bring the adapter up if you just added it.


      ifup eth0
      ifup eno33559296
      
      
      

       

      Step 3: Create an Initiator Group

       

      Get the IQN from your Linux guest:

       

      cat /etc/iscsi/initiatorname.iscsi

       

      Create a iSCSI initiator group on the nimble with the IQN

      Screen Shot 2016-07-06 at 9.34.15 PM.png

       

      Step 4: Create an Volume on Nimble

       

      Create a volume on the Nimble for your needs (perf policy, size, etc…) using the newly made iSCSI initiator group for access.


      Screen Shot 2016-07-06 at 9.36.20 PM.png

       

      Step 5: Connect to the Volume in Linux

       

      ncmadm  --rescan [iSCSI Discovery IP]

       

      (Example output)
      [root@ajl-rhel6 ~]# ncmadm --rescan 10.1.0.17
      Discovery IP :
      10.1.0.17
      List of discovered devices:
        target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.10:3260,2460
        target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.19:3260,2460
        target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.11:3260,2460
        target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.20:3260,2460
        target:iqn.2007-11.com.nimblestorage:ajl-linux6-ncm-das-v2d35dd4571510ccf.0000032c.5c64d18a portal:10.1.0.17:3260,2460
      
      Login to the new devices:
        target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.10:3260,2460
        target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.19:3260,2460
        target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.20:3260,2460
        target:iqn.2007-11.com.nimblestorage:nimble-cs420-grp-g2d35dd4571510ccf portal:10.1.0.11:3260,2460
        target:iqn.2007-11.com.nimblestorage:ajl-linux6-ncm-das-v2d35dd4571510ccf.0000032c.5c64d18a portal:10.1.0.17:3260,2460
      Rescan initiated. Use ncmadm -l to list Nimble device
      
      
      


      Showing the list command


      [root@ajl-rhel6 ~]# ncmadm -l
      size=150G uuid=2f7773dfa9b4d0ad36c9ce9008ad1645c mount-device: /dev/nimblestorage/AJL-Linux6-NCM-DAS-2f7773dfa9b4d0ad36c9ce9008ad1645c
      target: iqn.2007-11.com.nimblestorage:ajl-linux6-ncm-das-v2d35dd4571510ccf.0000032c.5c64d18a connection_mode=automatic
      ` + array_id=1 device=/dev/dm-3
         |-    7:0:0:0   sdb   8:16  session=5   sess_st=LOGGED_IN  dev_st=running    iface=default    tgtportal=10.1.0.17        data_ip=10.1.0.11      
         |-    8:0:0:0   sdc   8:32  session=6   sess_st=LOGGED_IN  dev_st=running    iface=default    tgtportal=10.1.0.17        data_ip=10.1.0.10      
      
      
      

       

      Step 6: Format a Multipath Disk

       

      Find it in /dev/mapper or /dev/nimblestorage (mpatha in this case).  If it is a different location then /dev/mapper you'll need to modify the below commands to your location.


      [root@ajl-rhel6 ~]# ls /dev/mapper/
      2f7773dfa9b4d0ad36c9ce9008ad1645c  control  mpatha  vg_ajlrhel6-lv_home  vg_ajlrhel6-lv_root  vg_ajlrhel6-lv_swap
      
      
      

       

      Create the filesystem

      fdisk /dev/mapper/[mpathXX]

      Create it how you want. My example is a primary partition, the first one, and set type to be LVM as I prefer LVMs. You might choose a different type or to use parted and GPT vs fdisk and MBR. I will be covering fdisk. If you do not want LVM, don't change the partition type, leave it as 83.

       

      Example:
      [root@ajl-rhel6 ~]# fdisk /dev/mapper/mpatha
      Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
      Building a new DOS disklabel with disk identifier 0xb4e794c7.
      Changes will remain in memory only, until you decide to write them.
      After that, of course, the previous content won't be recoverable.
      
      Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
      
      WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
               switch off the mode (command 'c') and change display units to
               sectors (command 'u').
      
      Command (m for help): m
      Command action
         a   toggle a bootable flag
         b   edit bsd disklabel
         c   toggle the dos compatibility flag
         d   delete a partition
         l   list known partition types
         m   print this menu
         n   add a new partition
         o   create a new empty DOS partition table
         p   print the partition table
         q   quit without saving changes
         s   create a new empty Sun disklabel
         t   change a partition's system id
         u   change display/entry units
         v   verify the partition table
         w   write table to disk and exit
         x   extra functionality (experts only)
      
      Command (m for help): n
      Command action
         e   extended
         p   primary partition (1-4)
      p
      Partition number (1-4): 1
      First cylinder (1-19581, default 1):
      Using default value 1
      Last cylinder, +cylinders or +size{K,M,G} (1-19581, default 19581):
      Using default value 19581
      
      Command (m for help): t 
      Selected partition 1
      Hex code (type L to list codes): l
      
      0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris       
      1  FAT12           39  Plan 9          82  Linux swap / So c1  DRDOS/sec (FAT-
      2  XENIX root      3c  PartitionMagic  83  Linux           c4  DRDOS/sec (FAT-
      3  XENIX usr       40  Venix 80286     84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
      4  FAT16 <32M      41  PPC PReP Boot   85  Linux extended  c7  Syrinx        
      5  Extended        42  SFS             86  NTFS volume set da  Non-FS data   
      6  FAT16           4d  QNX4.x          87  NTFS volume set db  CP/M / CTOS / .
      7  HPFS/NTFS       4e  QNX4.x 2nd part 88  Linux plaintext de  Dell Utility  
      8  AIX             4f  QNX4.x 3rd part 8e  Linux LVM       df  BootIt        
      9  AIX bootable    50  OnTrack DM      93  Amoeba          e1  DOS access    
      a  OS/2 Boot Manag 51  OnTrack DM6 Aux 94  Amoeba BBT      e3  DOS R/O       
      b  W95 FAT32       52  CP/M            9f  BSD/OS          e4  SpeedStor     
      c  W95 FAT32 (LBA) 53  OnTrack DM6 Aux a0  IBM Thinkpad hi eb  BeOS fs       
      e  W95 FAT16 (LBA) 54  OnTrackDM6      a5  FreeBSD         ee  GPT           
      f  W95 Ext'd (LBA) 55  EZ-Drive        a6  OpenBSD         ef  EFI (FAT-12/16/
      10  OPUS            56  Golden Bow      a7  NeXTSTEP        f0  Linux/PA-RISC b
      11  Hidden FAT12    5c  Priam Edisk     a8  Darwin UFS      f1  SpeedStor     
      12  Compaq diagnost 61  SpeedStor       a9  NetBSD          f4  SpeedStor     
      14  Hidden FAT16 <3 63  GNU HURD or Sys ab  Darwin boot     f2  DOS secondary 
      16  Hidden FAT16    64  Novell Netware  af  HFS / HFS+      fb  VMware VMFS   
      17  Hidden HPFS/NTF 65  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE
      18  AST SmartSleep  70  DiskSecure Mult b8  BSDI swap       fd  Linux raid auto
      1b  Hidden W95 FAT3 75  PC/IX           bb  Boot Wizard hid fe  LANstep       
      1c  Hidden W95 FAT3 80  Old Minix       be  Solaris boot    ff  BBT           
      1e  Hidden W95 FAT1
      Hex code (type L to list codes): 8e
      Changed system type of partition 1 to 8e (Linux LVM)
      
      Command (m for help): w
      The partition table has been altered!
      
      Calling ioctl() to re-read partition table.
      
      WARNING: Re-reading the partition table failed with error 22: Invalid argument.
      The kernel still uses the old table. The new table will be used at
      the next reboot or after you run partprobe(8) or kpartx(8)
      Syncing disks.
      
      
      

       

      At this point you have two options.

       

      Option 1: Reboot to clear up partition table.

       

      Option 2: Run

      kpartx -a /dev/mapper/[mpath]

       

        Example:  kpartx -a /dev/mapper/mpathb

       

      After reboot/kpartx you will now see a /dev/mapper/mapthap1

       

      Step 6: Create an LVM

      (if you do not want an LVM you can skip to step 7 and see the file system format command)

       

      I'm a fan of LVM's.  As such, that is what I did next.  See the example for pvcreate/vgcreate/lvcreate using all the space.  If you want to do differently check out redhat's guide.

       

      [root@ajl-rhel6 ~]# pvcreate /dev/mapper/mpathap1
      WARNING: Ignoring duplicate config value: preferred_names
        Physical volume "/dev/mapper/mpathap1" successfully created
      
      
      [root@ajl-rhel6 ~]# vgcreate IOtesting /dev/mapper/mpathap1
      WARNING: Ignoring duplicate config value: preferred_names
        Volume group "IOtesting" successfully created
      
      
      [root@ajl-rhel6 ~]# lvcreate -n IOTesting -l 100%FREE IOtesting
      WARNING: Ignoring duplicate config value: preferred_names
        Logical volume "IOTesting"created.
      
      
      

       

      Bring your LVM's online

      vgchange -a y [volume group name]

       

        Example: vgchange -a y IOtesting

       

      or all at once

       

      Example: vgchange -a y
      
      
      

       

      Step 7: Format the Filesystem

       

      mkfs.ext4 -L [somelabel] /dev/[vg]/[lv]

      I highly encourage the use of labels when formatting the filesystem as you can mount by them.  VERY useful when things jump around.  If you are not **** an LVM this is a must as your iscsi disks names can move around at boot time (eg /dev/sdc -> /dev/sde). Labels will make your life better.

       

      [root@ajl-rhel6 ~]# mkfs.ext4 -L IOTST /dev/IOtesting/IOTesting
      mke2fs 1.41.12 (17-May-2010)
      Discarding device blocks: done                           
      Filesystem label=
      OS type: Linux
      Block size=4096 (log=2)
      Fragment size=4096 (log=2)
      Stride=0 blocks, Stripe width=0 blocks
      9830400 inodes, 39320576 blocks
      1966028 blocks (5.00%) reserved for the super user
      First data block=0
      Maximum filesystem blocks=4294967296
      1200 block groups
      32768 blocks per group, 32768 fragments per group
      8192 inodes per group
      Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872
      
      Writing inode tables: done                           
      Creating journal (32768 blocks): done
      Writing superblocks and filesystem accounting information: done
      
      This filesystem will be automatically checked every 36 mounts or
      180 days, whichever comes first.  Use tune2fs -c or -i to override.
      Creating auto mount (fstab)
      
      
      

       

      Step 8: Create an fstab Entry

       

      Create a location to mount the new filesystem

      mkdir /[some location]


        Example: mkdir /IOTesting

       

      Edit fstab

      _netdev is critical. If you do defaults and not _netdev your LVM will not go active at boot time and you will have no filesystem to write to!

       

      vi /etc/fstab

       

      Mount by LVM point:

       

      /dev/IOtesting/IOTesting        /IOTesting      ext4    _netdev        0 0

       

      Mount by filesystem label:

       

      LABEL=TST2      /IOtst2 ext4    _netdev        0 0

       

      Verify and mount the entries created in fstab

       

      mount -a

       

      Step 9: View the Fruit of Your Labor

       

      [root@ajl-rhel6 ~]# df -h
      Filesystem            Size  Used Avail Use% Mounted on
      /dev/mapper/vg_ajlrhel6-lv_root
                             50G  2.3G   45G   5% /
      tmpfs                 1.9G     0  1.9G   0% /dev/shm
      /dev/sda1             477M   41M  411M  10% /boot
      /dev/mapper/vg_ajlrhel6-lv_home
                             45G   52M   43G   1% /home
      /dev/mapper/IOtesting-IOTesting
                            148G   60M  140G   1% /IOTesting
      /dev/mapper/IOtesting2-IOTesting2
                            148G   60M  140G   1% /IOtst2