Configuring Device Mapper Multipathing with Linux

Oracle

Linux SAN Multipathing

What is Multipathing?

Multipathing is a fault-tolerance technology that allows you to have more than one physical path between a host (server) and an external storage device (SAN). In the event that any component (NIC, cable, switch) in the environment fails, the physical path is switched to a path that does not involve the failed component allowing continued communications. My weapon of choice when it comes to multipathing in Linux is Device Mapper-Multipath (DM-Multipath). DM-Multipathing is a fail-over AND load-balancing solution that is highly configurable.

Tutorial
This tutorial will include:

1. Oracle Enterprise Linux Server 6.6

2. DM-Mulitipath – device-mapper-multipath-0.4.9-80.0.1.el6_6.3.x86_64

To configure a system with DM-Multipath we need to install the package first. There are a couple options here…

Option 1: Install from a configured YUM repository.

# yum install device-mapper-multipath

Option 2: Download the package and manually install it…

# rpm -ivh device-mapper-multipath-0.4.9-80.0.1.el6_6.3.x86_64

Next, we need to configure DM-Multipath.

1. Edit the /etc/multipath.conf file and set parameters as necessary. Your storage provider probably has recommended settings and should be checked.

# vi /etc/multipath.conf

2. Load the necessary modules

# modprobe dm-multipath

3. Add dm-multipath to startup

# chkconfig multipathd on

4. Start multipath daemon

# service multipathd start

Now, the stage is set.
I like to take a quick inventory of the existing storage on the system prior to adding new storage to the server so the new LUN is easier to identify.

[root@OEL6-test eagle]# ll /dev/sd*
brw-rw---- 1 root disk 8,  0 Sep  7 10:11 /dev/sda
brw-rw---- 1 root disk 8,  1 Sep  7 10:12 /dev/sda1
brw-rw---- 1 root disk 8,  2 Sep  7 10:11 /dev/sda2
brw-rw---- 1 root disk 8, 16 Sep  7 10:12 /dev/sdb

[root@OEL6-test eagle]# lsblk
NAME                           MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda                              8:0    0   50G  0 disk 
├─sda1                           8:1    0  500M  0 part  /boot
└─sda2                           8:2    0 49.5G  0 part 
  ├─vg_oel6test-lv_root (dm-0) 252:0    0 43.5G  0 lvm   /
  └─vg_oel6test-lv_swap (dm-1) 252:1    0    6G  0 lvm   [SWAP]
sdb                              8:16   0    2G  0 disk 
└─3600144f09ff1616800005717cb580001 (dm-2)
                               252:2    0    2G  0 mpath
  └─3600144f09ff1616800005717cb580001p1 (dm-3)
                               252:3    0    2G  0 part  /test
sr0                             11:0    1 1024M  0 rom 

 [root@OEL6-test eagle]# cat /proc/scsi/scsi
Attached devices:
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: VBOX     Model: CD-ROM           Rev: 1.0
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: VBOX HARDDISK    Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 00
  Vendor: SUN      Model: Sun Storage 7000 Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 05

[root@OEL6-test eagle]# multipath -ll
3600144f09ff1616800005717cb580001 dm-2 SUN,Sun Storage 7000
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 3:0:0:0 sdb 8:16 active ready running

We are ready to add a LUN. Present a LUN to the host or have your storage administrator present a LUN to your server. I presented a 1GB LUN to the server and we are ready to proceed.

First, we need to rescan the server for the newly presented LUN(s).

[root@OEL6-test eagle]# echo "- - -" > /sys/class/scsi_host/host0/scan 
[root@OEL6-test eagle]# echo "- - -" > /sys/class/scsi_host/host1/scan 
[root@OEL6-test eagle]# echo "- - -" > /sys/class/scsi_host/host2/scan 
[root@OEL6-test eagle]# echo "- - -" > /sys/class/scsi_host/host3/scan

Alternatively, this can also be run as a for loop>

# for host in $(ls-1d /sys/class/scsi_host/*); do echo “- - -“ > ${host}/scan; done

Also, in the event that fiber channel is in use, the respective commands would be>

# echo "1" > /sys/class/fc_host/host1/issue_lip
# echo "1" > /sys/class/fc_host/host2/issue_lip

And the for loop>

# for host in $(ls -1d /sys/class/fc_host/*); fo echo “1” > ${host}/issue_lip; done

Now, let’s take another look…

 [root@OEL6-test eagle]# ll /dev/sd*
brw-rw---- 1 root disk 8,  0 Sep  7 10:11 /dev/sda
brw-rw---- 1 root disk 8,  1 Sep  7 10:12 /dev/sda1
brw-rw---- 1 root disk 8,  2 Sep  7 10:11 /dev/sda2
brw-rw---- 1 root disk 8, 16 Sep  7 10:18 /dev/sdb
brw-rw---- 1 root disk 8, 32 Sep  7 10:23 /dev/sdc

[root@OEL6-test eagle]# lsblk
NAME                           MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda                              8:0    0   50G  0 disk 
├─sda1                           8:1    0  500M  0 part  /boot
└─sda2                           8:2    0 49.5G  0 part 
  ├─vg_oel6test-lv_root (dm-0) 252:0    0 43.5G  0 lvm   /
  └─vg_oel6test-lv_swap (dm-1) 252:1    0    6G  0 lvm   [SWAP]
sdb                              8:16   0    2G  0 disk 
└─3600144f09ff1616800005717cb580001 (dm-2)
                               252:2    0    2G  0 mpath
  └─3600144f09ff1616800005717cb580001p1 (dm-3)
                               252:3    0    2G  0 part  /test
sdc                              8:32   0    1G  0 disk 
sr0                             11:0    1 1024M  0 rom  

[root@OEL6-test eagle]# cat /proc/scsi/scsi
Attached devices:
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: VBOX     Model: CD-ROM           Rev: 1.0
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: VBOX HARDDISK    Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 00
  Vendor: SUN      Model: Sun Storage 7000 Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 01
  Vendor: SUN      Model: Sun Storage 7000 Rev: 1.0
  Type:   Direct-Access                    ANSI  SCSI revision: 05

[root@OEL6-test eagle]# multipath -ll
3600144f09ff1616800005717cb580001 dm-2 SUN,Sun Storage 7000
size=2.0G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 3:0:0:0 sdb 8:16 active ready running

Now, the LUN has been discovered by the host, but it is still not visible in multipath. So, we need to add the new LUN to the multipath configuration.

First we need to discover the wwid of the new LUN

[root@OEL6-test eagle]# scsi_id -g -u /dev/sdc
3600144f09ff16168000057cfe9710001

Next, edit the /etc/multipath.conf file and add the wwid of the new LUN to the ‘blacklist exceptions’ and ‘multipaths’ portions of multipath.conf.
Ex: multipath.conf – you can see the new entries highlighted

[root@OEL6-test eagle]# cat /etc/multipath.conf

defaults {
  udev_dir                /dev
  polling_interval        10
  path_selector           "round-robin 0"
  path_grouping_policy    multibus
  getuid_callout          "/lib/udev/scsi_id --replace-whitespace --whitelisted --device=/dev/%n"
  path_checker            readsector0
  rr_min_io               100
  max_fds                 8192
  rr_weight               priorities
  failback                immediate
  no_path_retry           fail
  user_friendly_names     no
}
blacklist {
  devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
  devnode "^hd[a-z]"
  devnode "^dcssblk[0-9]*"
  # don't count normal SATA devices as multipaths
  device {
          vendor  "ATA"
  }
  wwid "*"
}
devices {
  device {
  }
}
blacklist_exceptions {
       wwid "3600144f09ff1616800005717cb580001"
       wwid "3600144f09ff16168000057cfe9710001"
}
multipaths {
       multipath {
  uid 0
  gid 0
       wwid "3600144f09ff1616800005717cb580001"
  mode 0600
       }
       multipath {
  uid 0
  gid 0
       wwid "3600144f09ff16168000057cfe9710001"
  mode 0600
       }
}

Finally, reload/restart the multipath daemon>

[root@OEL6-test eagle]# multipath -r
reload: 3600144f09ff1616800005717cb580001 undef SUN,Sun Storage 7000
size=2.0G features='0' hwhandler='0' wp=undef
`-+- policy='round-robin 0' prio=1 status=undef
 `- 3:0:0:0 sdb 8:16 active ready running
reload: 3600144f09ff16168000057cfe9710001 undef SUN,Sun Storage 7000
size=1.0G features='0' hwhandler='0' wp=undef
`-+- policy='round-robin 0' prio=1 status=undef
 `- 3:0:0:1 sdc 8:32 active ready running

Now, the new LUN has been added to the multipath configuration! All that is left is to partition, format and mount the LUN!

Want to automate this process? Contact us to learn how our team uses Shell Script to automate the DM-Multipath configuration process.

Related post: Building Test Environments with Oracle VirtualBox

About the author

Shan BaugherShan Baugher, Systems Administrator
Shan Baugher began working with Eagle in 2012 and is responsible for initiating, architecting, and managing complex IT infrastructures for clients, as well as the offices at Eagle’s headquarters. With over 8 years of project management and systems administration experience, Shan is well-versed in managing virtual environments of all shapes and sizes. He holds an assortment of Professional Microsoft Certifications including MCSE, MCSA, MCP, and MCTS and is also Certified as a Specialist in Oracle VM 3.0 for x86 implementation.

Comments