Veritas Unstartable Volume

In this example of VXVM 4.0 on a Solaris 8 system, an array was temporarily unavailable, causing problems with a file system whose two plexes resided on the array.

bash-2.03# cd /files04
bash: cd: /files04: I/O error

The volume was in DISABLED ACTIVE state, and both plexes were in DISABLED RECOVER state.

v  vol04        -            DISABLED ACTIVE   29360128 SELECT    -        fsgen
pl vol04-01     vol04        DISABLED RECOVER  29367434 STRIPE    3/128    RW
sd appsdg01-04  vol04-01     cs_array07-f0 8392167 2797389 0/0    c1t0d0   ENA
sd appsdg07-01  vol04-01     cs_array03-f2 0   5594778  0/2797389 c4t2d0   ENA
sd appsdg07-04  vol04-01     cs_array03-f2 11189556 1396899 0/8392167 c4t2d0 ENA
sd appsdg02-04  vol04-01     cs_array07-f1 8392167 2797389 1/0    c1t1d0   ENA
sd appsdg10-02  vol04-01     cs_array06-f1 2797389 5594778 1/2797389 c5t1d0 ENA
sd appsdg10-05  vol04-01     cs_array06-f1 13986945 1396899 1/8392167 c5t1d0 ENA
sd appsdg03-04  vol04-01     cs_array07-f2 8392167 2797389 2/0    c1t2d0   ENA
sd appsdg11-02  vol04-01     cs_array06-f2 8392167 6991677 2/2797389 c5t2d0 ENA
pl vol04-02     vol04        DISABLED RECOVER  29367434 STRIPE    3/128    RW
sd appsdg04-02  vol04-02     cs_array07-f3 2797389 2797389 0/0    c1t3d0   ENA
sd appsdg04-05  vol04-02     cs_array07-f3 0   2797389  0/2797389 c1t3d0   ENA
sd appsdg04-06  vol04-02     cs_array07-f3 16784334 894159 0/5594778 c1t3d0 ENA
sd appsdg14-02  vol04-02     cs_array07-f6 12586455 3300129 0/6488937 c1t6d0 ENA
sd appsdg12-03  vol04-02     cs_array06-f3 5594778 2797389 1/0    c5t3d0   ENA
sd appsdg13-02  vol04-02     cs_array07-f4 12586455 5092038 1/2797389 c1t4d0 ENA
sd appsdg12-02  vol04-02     cs_array06-f3 16784334 894159 1/7889427 c5t3d0 ENA
sd appsdg05-02  vol04-02     cs_array03-f0 12586455 1005480 1/8783586 c4t0d0 ENA
sd appsdg09-02  vol04-02     cs_array06-f0 2797389 8392167 2/0    c5t0d0   ENA
sd appsdg09-06  vol04-02     cs_array06-f0 3591 1396899 2/8392167 c5t0d0   ENA

We confirmed that the storage array was available to the operating system.

# luxadm probe
Found Enclosure(s):
...
SENA               Name:cs_array06   Node WWN:5080020000038ba8  
  Logical Path:/dev/es/ses6
  Logical Path:/dev/es/ses7

# luxadm display cs_array06

                                   SENA           
                                 DISK STATUS
SLOT   FRONT DISKS       (Node WWN)          REAR DISKS        (Node WWN)
0      On (O.K.)         2000002037094289    On (O.K.)         200000203709422e
1      On (O.K.)         2000002037093aaf    On (O.K.)         2000002037094220
2      On (O.K.)         200000203709410b    On (O.K.)         2000002037093ddd
3      On (O.K.)         2000002037094254    On (O.K.)         200000203709422b
4      On (O.K.)         20000020370940da    On (O.K.)         2000002037094247
5      Not Installed                         Not Installed                    
6      On (O.K.)         2000002037093df0    On (O.K.)         200000203709383f

Next, we reattached the disks to the disk group they were in. You may want to run vxreattach -c diskname to check if a reattach is possible before attempting to reattach the disks.

# vxdisk list
...

-            -         cs_array06-f0 appsdg       failed was:c5t0d0s2
-            -         cs_array06-f1 appsdg       failed was:c5t1d0s2
-            -         cs_array06-f2 appsdg       failed was:c5t2d0s2
-            -         cs_array06-f3 appsdg       failed was:c5t3d0s2
-            -         cs_array06-r4 appsdg       failed spare was:c5t20d0s2
-            -         cs_array06-f4 appsdg       failed was:c5t4d0s2

# cd /usr/lib/vxvm/bin
# ./vxreattach c5t0d0s2
# ./vxreattach c5t1d0s2
# ./vxreattach c5t2d0s2
# ./vxreattach c5t3d0s2
# ./vxreattach c5t20d0s2
# ./vxreattach c5t4d0s2


We then followed the "Recovering an Unstartable Volume with a Disabled Plex in the RECOVER State" procedure in the Volume Manager Troubleshooting Guide.

1. Force plex vol04-01 into the OFFLINE state.
# vxmend -g appsdg -o force off vol04-01

2. Place plex vol04-01 into the STALE state.
# vxmend -g appsdg on vol04-01

3. There are no other clean plexes in the volume, so make plex vol04-01 DISABLED and CLEAN.
# vxmend -g appsdg fix clean vol04-01

4. Start the volume, and perform resynchronization of the plexes in the background.
# vxvol -g appsdg -o bg start vol04

At this point, the file system is unmounted, checked for file system consistency, and remounted.

# umount /files04

# mount /files04
UX:vxfs mount: ERROR: V-3-21268: /dev/vx/dsk/appsdg/vol04 is corrupted. needs checking

# fsck -F vxfs /dev/vx/rdsk/appsdg/vol04

log replay in progress
replay complete - marking super-block as CLEAN

# mount /files04

Back to brandonhutchinson.com.
Last modified: 2007/08/07