Execution of iscsi Mulltipathing

[mihirborkar]

Here are the steps to enable iscsi multipathing :

1. We will need Execute iSCSI shell commands on multiple remote hosts (in addition to localhost) #173 to expose the tgt targets in every tgt servers.
   For example : the file /etc/tgt/conf.d/kumo-dan-installscript-img49.conf with the below configuration will have to be replicated.

<target try>
    driver iscsi
    bs-type rbd
    backing-store bmi-introspect/kumo-dan-installscript-img49
    bsopts ""conf=/etc/ceph/ceph.conf;id=bmi-introspect""
    write-cache off
    initiator-address ALL
</target>

2. Make changes in the /var/lib/tftpboot/*.ipxe File (Add the 2nd target in the ipxe file)
For Example :

  sleep 23
  set keep-san 1
  ifconf --configurator=dhcp net0
  sanboot iscsi:10.20.30.1:tcp:3260:1:kumo-dan-installscript-img37 \
                iscsi:10.20.30.2:tcp:3260:1:kumo-dan-installscript-img37
  boot

wherein 10.20.30.1 and 10..20.30.2 are the IP addresses of the respective tgt servers.

3. Boot the node with the above change(Reprovision the node)
On booting the node will have the entry of both the tgt servers in the /sys/firmware/ibft

[root@localhost ~]# cat /sys/firmware/ibft/target0/ip-addr 
10.20.30.1
[root@localhost ~]# cat /sys/firmware/ibft/target1/ip-addr 
10.20.30.2
[root@localhost ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0 48.8G  0 disk 
├─sda1   8:1    0  476M  0 part 
└─sda2   8:2    0 48.4G  0 part 
sdb      8:16   0 48.8G  0 disk 
├─sdb1   8:17   0  476M  0 part /boot
└─sdb2   8:18   0 48.4G  0 part /

4. Now a shell script which performs the following can be executed:
5. yum update
6. yum install device-mapper-multipath
7. mpathconf --enable --with_multipathd y
8. mpathconf --enable
9. Make the following change in /etc/multipath.conf :
   (uncommenting the defaults settings)

defaults {
        polling_interval        10
        path_selector           "round-robin 0"
        path_grouping_policy    multibus
        uid_attribute           ID_SERIAL
#       prio                    alua
        path_checker            readsector0
        rr_min_io               100
        max_fds                 8192
        rr_weight               priorities
        failback                immediate
        no_path_retry           5
        user_friendly_names     yes
        find_multipaths         yes
}

6. systemctl start multipathd; systemctl enable multipathd
7. Reboot

on logging into the node again:
another shell script will have to be executed that does the following :

1.multipath -a /dev/sdb (which is where my boot drive is)
2.dracut --force -H --add multipath
3.shutdown
4.reboot

Multipathing gets established and successful hand off takes place in the event of any of the tgt servers going down. This is how it looks like once we log inn to the node:

[root@localhost ~]# multipath -l
mpatha (360000000000000000e00000000010001) dm-0 IET     ,VIRTUAL-DISK
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 7:0:0:1 sda 8:0  active undef unknown
  `- 8:0:0:1 sdb 8:16 active undef unknown

[root@localhost ~]# multipath -ll
mpatha (360000000000000000e00000000010001) dm-0 IET     ,VIRTUAL-DISK
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 7:0:0:1 sda 8:0  active ready running
  `- 8:0:0:1 sdb 8:16 active ready running

[root@localhost ~]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda           8:0    0   10G  0 disk
+-mpatha    253:0    0   10G  0 mpath
  +-mpatha1 253:1    0  476M  0 part  /boot
  +-mpatha2 253:2    0    5G  0 part  /
sdb           8:16   0   10G  0 disk
+-mpatha    253:0    0   10G  0 mpath
  +-mpatha1 253:1    0  476M  0 part  /boot
  +-mpatha2 253:2    0    5G  0 part  /

At this point

1. A deep snapshot of the node from BMI can be taken to provision other nodes with multipathing enabled over it.

2. "N" TGT servers can be multipathed by simply discovering and logging into them using adm commands, those targets automatically get multipathed.

Problems with this approach :

1. Automating the shell scripts as two reboots are needed

2. Both the TGT servers are needed to be up for booting the node,
   otherwise the node does not boot up and gets stuck in dracut.
   https://docs.oracle.com/cd/E50245_01/E50246/html/ch06s101.html

As a solution to the above problem, multipathing with three tgt servers instead of two had been suggested by @SahilTikale and @apoorvemohan , that would look like this:

  sleep 23
  set keep-san 1
  ifconf --configurator=dhcp net0
  sanboot iscsi:**10.20.30.1**:tcp:3260:1:kumo-dan-installscript-img37 \
                iscsi:**10.20.30.2**:tcp:3260:1:kumo-dan-installscript-img37  \
                iscsi:**10.20.30.3**:tcp:3260:1:kumo-dan-installscript-img37
  boot

However, There seems to be some line limit on the numbers of arguments that can be passed in the ipxe file. #176