Why multipath marked a SAN LUN subpath as active/faulty where as it's visible to system?
Environment
- Red Hat Enterprise Linux(RHEL) 5
- Device Mapper Multipath (DM Multipath)
Issue
- After a SAN outage, one of lun marked as active/faulty where as it's visible to system:
sda: checker msg is "tur checker reports path is down"
mpath0 (36001438005deca3f0000400076e90000) dm-0 HP,HSV400
[size=50G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
\_ 0:0:2:1 sdc 8:32 [active][ready]
\_ 1:0:2:1 sdf 8:80 [active][ready]
\_ round-robin 0 [prio=30][enabled]
\_ 0:0:1:1 sdb 8:16 [active][ready]
\_ 1:0:0:1 sdd 8:48 [active][ready]
\_ 1:0:1:1 sde 8:64 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 0:0:0:1 sda 8:0 [active][faulty] <---
sdais marked as active/faulty:
# multipath
sda: checker msg is "tur checker reports path is down"
/sbin/scsi_id exitted with 1
cannot get the the wwid for sda <---
sda: checker msg is "tur checker reports path is down"
- Following command does not show wwid of the device:
scsi_id -g -u -s /block/sda
fdisk -lof the/dev/sdadevice shows partition tables.
Resolution
-
Issue observed with storage and resolved by disabling the the SAN Fibre switch port and re-enabling it.
-
With reference to error messages logged for the subpath:
sda()0:0:0:1: sd 0:0:0:1: SCSI error: return code = 0x00070000, try to verify if there is some issue present from SAN/Storage side for the configuration of sub path "sda" (0:0:0:1)? It is also advisable to engage storage vendor support to verify if there is any issue present from the storage side due to which the driver is receiving fibre channel response frames containing invalid conflicting information.
Root Cause
- Found
SCSI error: return code = 0x00070000error message logged in/var/log/messages. The SCSI error return codes of "0x00070000" in logs could be decoded as follows:
0x00 07 00 00
-------------
00 status byte : {likely} not valid, see other fields;
00 msg byte : {likely} not valid, see other fields;
07 host byte : DID_ERROR - internal error
00 driver byte : {likely} not valid, see other fields;
- The SCSI error return code of "0x00070000" (DID_ERROR) could be observed if there is some issue present from hardware or san-based issue is present within the storage subsystem such that received fibre channel response frames contains invalid or conflicting information that the driver is not able to use or reconcile. Please review the systems hardware, FC switch error counters, etc. to see if there is any indication of where the issue might lie.
Diagnostic Steps
- Below marked path is showed as failed:
mpath0 (36001438005deca3f0000400076e90000) dm-0 HP,HSV400
[size=50G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=100][active]
\_ 0:0:2:1 sdc 8:32 [active][ready]
\_ 1:0:2:1 sdf 8:80 [active][ready]
\_ round-robin 0 [prio=30][enabled]
\_ 0:0:1:1 sdb 8:16 [active][ready]
\_ 1:0:0:1 sdd 8:48 [active][ready]
\_ 1:0:1:1 sde 8:64 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 0:0:0:1 sda 8:0 [active][faulty] <---
- Looking at the status of same Lun on server, it appears to be active/running and visible to system:
Host: scsi0 Channel: 00 Id: 00 Lun: 01
Vendor: HP Model: HSV400 Rev: 1100
Type: Direct-Access ANSI SCSI revision: 05
- Trying to reload the multipath configuration using below command failed to bring the path online:
# service multipathd reload
# multipath -F
# multipath -v2
- Reinstate should resumes path
sdafrom failed state but that also failed:
# multipathd -k'reinstate sda'
- From the "/var/log/messages" file, lots of SCSI error messages observed for above sub path "sda" (0:0:0:1) which is marked as active/faulty :
Jun 11 09:30:53 localhost kernel: sd 0:0:0:1: Unhandled error code
Jun 11 09:30:53 localhost kernel: sd 0:0:0:1: SCSI error: return code = 0x00070000
Jun 11 09:30:53 localhost kernel: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
Jun 11 09:30:53 localhost kernel: sd 0:0:0:1: Unhandled error code
Jun 11 09:30:53 localhost kernel: sd 0:0:0:1: SCSI error: return code = 0x00070000
Jun 11 09:30:53 localhost kernel: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
Jun 11 09:30:53 localhost kernel: sd 0:0:0:1: Unhandled error code
Jun 11 09:30:53 localhost kernel: sd 0:0:0:1: SCSI error: return code = 0x00070000
Jun 11 09:30:53 localhost kernel: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
[...]
- The DID_ERROR status is assigned because the fibre channel response frame received by the driver from the hba has invalid or conflicting information. This means either an hba issue or san issue which is causing the bad response frames or status. Please go through How do I interpret scsi status messages in RHEL like "sd 2:0:0:243: SCSI error: return code = 0x08000002"? for detailed information about this issue.