Make a soft partition (a partition within a metadevice) d35 is the soft, d5 is the real metadevice, 1G is the size metainit d35 -p d5 1G Make a stripe of 3 disk, w/ 1MB stripe size metainit d100 1 3 ctd1 ctd2 ctd3 -i 1000k Mirroring disks with DiskSuiteMirroring disks with DiskSuite The following example shows how to mirror two disks--c0t0d0 with c0t1d0--using Solstice DiskSuite 4.2.1 on Solaris 8. Please read the excellent Mirroring A Boot Drive Using Solstice Disksuite page for more information. The current partition table of c0t0d0: Part Tag Flag Cylinders Size Blocks 0 root wm 0 - 812 400.15MB (813/0/0) 819504 1 swap wu 813 - 1333 256.43MB (521/0/0) 525168 2 backup wm 0 - 17659 8.49GB (17660/0/0) 17801280 3 unassigned wm 1334 - 1354 10.34MB (21/0/0) 21168 4 var wm 1355 - 8522 3.45GB (7168/0/0) 7225344 5 usr wm 8523 - 14764 3.00GB (6242/0/0) 6291936 6 unassigned wm 14765 - 16845 1.00GB (2081/0/0) 2097648 7 home wm 16846 - 17658 400.15MB (813/0/0) 819504 1. Copy the partition table of the first disk to its mirror. prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t1d0s2 2. Create at least 2 DiskSuite state database replicas on each disk. A state database replica contains DiskSuite configuration and state information. metadb -a -f -c2 /dev/dsk/c0t0d0s3 /dev/dsk/c0t1d0s3 Description of metadb flags: -a -- adding -f -- force (needed the first time creating databases) -c2 -- create 2 databases in each slice 3. Mirror the root slice. metainit -f d10 1 1 c0t0d0s0 metainit -f d20 1 1 c0t1d0s0 metainit d30 -m d10 metaroot d30 (Use this command only on the root slice!) 4. Mirror the swap slice. metainit -f d11 1 1 c0t0d0s1 metainit -f d21 1 1 c0t1d0s1 metainit d31 -m d11 5. Mirror the var slice. metainit -f d14 1 1 c0t0d0s4 metainit -f d24 1 1 c0t1d0s4 metainit d34 -m d14 6. Mirror the usr slice. metainit -f d15 1 1 c0t0d0s5 metainit -f d25 1 1 c0t1d0s5 metainit d35 -m d15 7. Mirror the unassigned slice. metainit -f d16 1 1 c0t0d0s6 metainit -f d26 1 1 c0t1d0s6 metainit d36 -m d16 8. Mirror the home slice. metainit -f d17 1 1 c0t0d0s7 metainit -f d27 1 1 c0t1d0s7 metainit d37 -m d17 Run metastat metadevice for information on a metadevice. If you wish to delete a metadevice, run metaclear metadevice. 9. Update /etc/vfstab to mount the mirrors after boot. /etc/vfstab before changes: #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # #/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes - fd - /dev/fd fd - no - /proc - /proc proc - no - /dev/dsk/c0t0d0s1 - - swap - no - /dev/md/dsk/d30 /dev/md/rdsk/d30 / ufs 1 no logging /dev/dsk/c0t0d0s5 /dev/rdsk/c0t0d0s5 /usr ufs 1 no ro,logging /dev/dsk/c0t0d0s4 /dev/rdsk/c0t0d0s4 /var ufs 1 no nosuid,logging /dev/dsk/c0t0d0s7 /dev/rdsk/c0t0d0s7 /home ufs 2 yes nosuid,logging /dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 /opt ufs 2 yes nosuid,logging swap - /tmp tmpfs - yes - /etc/vfstab after changes: #device device mount FS fsck mount mount #to mount to fsck point type pass at boot options # #/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes - fd - /dev/fd fd - no - /proc - /proc proc - no - /dev/md/dsk/d31 - - swap - no - /dev/md/dsk/d30 /dev/md/rdsk/d30 / ufs 1 no logging /dev/md/dsk/d35 /dev/md/rdsk/d35 /usr ufs 1 no ro,logging /dev/md/dsk/d34 /dev/md/rdsk/d34 /var ufs 1 no nosuid,logging /dev/md/dsk/d37 /dev/md/rdsk/d37 /home ufs 2 yes nosuid,logging /dev/md/dsk/d36 /dev/md/rdsk/d36 /opt ufs 2 yes nosuid,logging swap - /tmp tmpfs - yes - 10. Reboot the system. lockfs -fa init 6 11. Attach the second submirror to the mirror. This will cause the data from the boot disk to be synchronized with the mirrored drive. metattach d30 d20 metattach d31 d21 metattach d34 d24 metattach d35 d25 metattach d36 d26 metattach d37 d27 You can run metastat to track the mirroring progress. 12. Change the crash dump device to the DiskSuite swap metadevice (Solaris 7 and later). dumpadm -d `swap -l | tail -1 | awk '{print $1}'` 13. Make the mirror disk bootable. installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s0 14. Determine the physical device path of the mirror disk. ls -l /dev/dsk/c0t1d0s0 lrwxrwxrwx 1 root root 50 Jan 16 10:20 /dev/rdsk/c0t1d0s0 -> ../../devices/pci@1f,0/pci@1,1/ide@3/dad@1,0:a 15. Create a mirror device alias for the mirror disk: eeprom "nvramrc=devalias mirror /pci@1f,0/pci@1,1/ide@3/dad@1,0:a" eeprom "use-nvramrc?=true" You can also configure the device alias from the Open Boot (ok) prompt: nvalias mirror /pci@1f,0/pci@1,1/ide@3/dad@1,0:a nvstore If there is ever a problem with the standard boot disk, run eeprom "boot-device=mirror" from the shell or boot-device=mirror from the Open Boot prompt. ================================================================================ vxdctl enable prtvtoc /dev/rdsk/c1t0d0s0 | fmthard -s - /dev/rdsk/c1t1d0s0 metadb -a -f /dev/dsk/c1t0d0s7 /dev/dsk/c1t1d0s7 metainit -f d0 1 1 c1t0d0s0 metainit -f d1 1 1 c1t1d0s0 metainit d2 -m d0 metaroot d2 metainit -f d3 1 1 c1t0d0s4 metainit -f d4 1 1 c1t1d0s4 metainit -f d5 -m d3 vi /etc/vfstab lockfs -fa init 6 #metadetach d5 d1 #metaclear d1 metattach d5 d4 metattach d2 d1 metainit -f d6 1 1 c1t0d0s1 metainit -f d7 1 1 c1t1d0s1 metainit d8 -m d6 metattach d8 d7 swap -a /dev/md/dsk/d8 swap -l vi /etc/vfstab ================================================================================ #!/bin/sh # #DiskSuite mirror shell script #I use the following shell script to mirror the $BOOT_DISK and $MIRROR_DISK after #manually creating the DiskSuite state database replicas. If slice 0 is not your #root slice, make sure to modify the script so that the $METAROOT command is #executed on the root slice. Be sure to comment out slices that you would not #like mirrored. The script assumes that DiskSuite database replicas are located #on slice 3, so this slice is initally commented out. #Back to brandonhutchinson.com. #Last modified: 02/04/2004 BOOT_DISK=c1t2d0 MIRROR_DISK=c1t1d0 METAINIT=/usr/sbin/metainit METAROOT=/usr/sbin/metaroot # Slice 0 (root slice) $METAINIT -f d10 1 1 ${BOOT_DISK}s0 $METAINIT -f d20 1 1 ${MIRROR_DISK}s0 $METAINIT d30 -m d10 $METAROOT d30 # Slice 1 $METAINIT -f d11 1 1 ${BOOT_DISK}s1 $METAINIT -f d21 1 1 ${MIRROR_DISK}s1 $METAINIT d31 -m d11 # Slice 2 will not be mirrored (represents entire disk) # Slice 3 is initially commented out; slice containing DiskSuite # database replicas in our environment #$METAINIT -f d13 1 1 ${BOOT_DISK}s3 #$METAINIT -f d23 1 1 ${MIRROR_DISK}s3 #$METAINIT d33 -m d13 # Slice 4 $METAINIT -f d14 1 1 ${BOOT_DISK}s4 $METAINIT -f d24 1 1 ${MIRROR_DISK}s4 $METAINIT d34 -m d14 # Slice 5 $METAINIT -f d15 1 1 ${BOOT_DISK}s5 $METAINIT -f d25 1 1 ${MIRROR_DISK}s5 $METAINIT d35 -m d15 # Slice 6 $METAINIT -f d16 1 1 ${BOOT_DISK}s6 $METAINIT -f d26 1 1 ${MIRROR_DISK}s6 $METAINIT d36 -m d16 # Slice 7 $METAINIT -f d17 1 1 ${BOOT_DISK}s7 $METAINIT -f d27 1 1 ${MIRROR_DISK}s7 $METAINIT d37 -m d17 # Make the mirror disk bootable installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/${MIRROR_DISK}s0 ================================================================================ Mirroring Disks with Solstice DiskSuiteMirroring Disks with Solstice DiskSuite Introduction This paper will present a short introduction to mirroring two disks using Solstice DiskSuite. Although not as robust as Veritas Volume Manager (VxVM) (also distributed by Sun as the "Sun Enterprise Volume Manager" (SEVM)), DiskSuite is nonetheless still a popular choice for doing basic disk mirroring. This tutorial will be presented using an actual mirroring session with comments and explanations interspersed. Installation The first step to setting up mirroring using DiskSuite is to install the DiskSuite packages and any necessary patches. The latest recommended version of DiskSuite is 4.2 for systems running Solaris 2.6 and higher. There are currently two packages and one patch necessary to install DiskSuite. They are: SUNWmd SUNWmdg 106627-04 (obtain latest revision) The packages should be installed in the same order as listed above. Note that a reboot is necessary after the install as new drivers will be added to the Solaris kernel. Also to make life easier, be sure to update your PATH and MANPATH variables to add DiskSuite's directories. Executables reside in /usr/opt/SUNWmd/sbin and man pages in /usr/opt/SUNWmd/man. The Environment In this example we will be mirroring two disks, both on the same controller. The first disk will be the primary disk and the second will be the mirror. The disks are: Disk 1: c0t0d0 Disk 2: c0t1d0 The partitions on the disks are presented below. There are a few items of note here. Each disk is partitioned exactly the same. This is necessary to properly implement the mirrors. Slice 2, commonly referred to as the 'backup' slice, which represents the entire disk must not be mirrored. There are situations where slice 2 is used as a normal slice, however, this author would not recommend doing so. The three unassigned partitions on each disk are configured to each be 4MB. These 4MB slices will hold the DiskSuite State Database Replicas, or metadb's. More information on the state database replicas will be presented below. Although a metadb only occupies 517KB of space, I prefer to create the slices slightly larger. Disk 1: c0t0d0s0: / c0t0d0s1: swap c0t0d0s2: backup c0t0d0s3: unassigned c0t0d0s4: /var c0t0d0s5: unassigned c0t0d0s6: unassigned c0t0d0s7: /export Disk 2: c0t1d0s0: / c0t1d0s1: swap c0t1d0s2: backup c0t1d0s3: unassigned c0t1d0s4: /var c0t1d0s5: unassigned c0t1d0s6: unassigned c0t1d0s7: /export The Database State Replicas The database state replicas serve a very important function in DiskSuite. They are the repositories of information on the state and configuration of each metadevice (A logical device created through DiskSuite is known as a metadevice). Having multiple replicas is critical to the proper operation of DiskSuite. There must be a minimum of three replicas. DiskSuite requires at least 51% of the replicas to be present in order to operate. Replicas should be spread across disks and controllers where possible. In a three drive configuration, at least one replica should be on each disk, thus allowing for a one disk failure. In a two drive configuration, such as the one we present here, there must be at least two replicas per disk. If there were only three and the disk which held two of them failed, there would not be enough information for DiskSuite to function. Here we will create our state replicas using the metadb command: # /usr/opt/SUNWmd/sbin/metadb -a -f /dev/dsk/c0t0d0s3 # /usr/opt/SUNWmd/sbin/metadb -a -f /dev/dsk/c0t0d0s5 # /usr/opt/SUNWmd/sbin/metadb -a -f /dev/dsk/c0t0d0s6 # /usr/opt/SUNWmd/sbin/metadb -a -f /dev/dsk/c0t1d0s3 # /usr/opt/SUNWmd/sbin/metadb -a -f /dev/dsk/c0t1d0s5 # /usr/opt/SUNWmd/sbin/metadb -a -f /dev/dsk/c0t1d0s6 The -a and -f options used together create the initial replicas. The -a option attaches a new database device and automatically edits the /etc/system file. The -f option creates the actual database replica. Initializing Submirrors Each mirrored meta device contains two or more submirrors. The meta device gets mounted by the operating system rather than the original logical device. Below we will walk through the steps involved in creating metadevices for our primary filesystems. Here we create the two submirrors for the / (root) filesystem, as well as a one way mirror between the meta device and its first submirror. # metainit -f d10 1 1 c0t0d0s0 # metainit -f d20 1 1 c0t1d0s0 # metainit d0 -m d10 The first two commands create the two submirrors. The -f option forces the creation of the submirror even though the specified slice is a mounted filesystem. The second two options 1 1 specify the number of stripes on the metadevice and the number of slices that make up the stripe. In a mirroring situation, this should always be 1 1. Finally, we specify the logical device that we will be mirroring. After mirroring the root partition, we need to run the metaroot command. This command will update the root entry in /etc/vfstab with the new metadevice as well as add the appropriate configuration information into /etc/system. Ommitting this step is one of the most common mistakes made by those unfamiliar with DiskSuite. If you do not run the metaroot command before you reboot, you will not be able to boot the system! # metaroot d0 Next, we continue to create the submirrors and initial one way mirrors for the metadevices which will replace the swap, and /var partitions. # metainit -f d11 1 1 c0t0d0s1 # metainit -f d21 1 1 c0t1d0s1 # metainit d1 -m d11 # metainit -f d14 1 1 c0t0d0s4 # metainit -f d24 1 1 c0t1d0s4 # metainit d4 -m d14 # metainit -f d17 1 1 c0t0d0s7 # metainit -f d27 1 1 c0t1d0s7 # metainit d7 -m d17 Updating /etc/vfstab The /etc/vfstab file must be updated at this point to reflect the changes made to the system. The / partition will have already been updated through the metaroot command run earlier, but the system needs to know about the new devices for swap and /var. The entries in the file will look something like the following: /dev/md/dsk/d1 - - swap - no - /dev/md/dsk/d4 /dev/md/rdsk/d4 /var ufs 1 yes - /dev/md/dsk/d7 /dev/md/rdsk/d7 /export ufs 1 yes - Notice that the device paths for the disks have changed from the normal style /dev/dsk/c#t#d#s# and /dev/rdsk/c#t#d#s# to the new metadevice paths, /dev/md/dsk/d# and /dev/md/rdsk/d#. Attaching the Mirrors Now we must attach the second half of the mirrors. Once the mirrors are attached it will begin an automatic synchonization process to ensure that both halves of the mirror are identical. The progress of the synchonization can be monitored using the metastat command. To attach the submirrors, issue the following commands: # metattach d0 d20 # metattach d1 d21 # metattach d4 d24 # metattach d7 d27 Final Thoughts With an eye towards recovery in case of a future disaster it may be a good idea to find out the physical device path of the root partition on the second disk in order to create an Open Boot PROM (OBP) device alias to ease booting the system if the primary disk fails. In order to find the physical device path, simply do the following: # ls -l /dev/dsk/c0t1d0s0 This should return something similar to the following: /sbus@3,0/SUNW,fas@3,8800000/sd@1,0:a Using this information, create a device alias using an easy to remember name such as altboot. To create this alias, do the following in the Open Boot PROM: ok nvalias altboot /sbus@3,0/SUNW,fas@3,8800000/sd@1,0:a For more information on creating OBP device aliases, refer to the following document: TechNote: Modifying the CD-ROM nvalias on an Ultra 10 (IDE based) System". It is now possible to boot off of the secondary device in case of failure using boot altboot from the OBP. For more information on DiskSuite, including configuring metatrans devices for filesystem transaction logging, see the second part of this document, Additional ================================================================================ FIX a failed root mirror This document is aimed at those attempting to setup root disk mirroring with Solaris[TM] Volume Manager on Solaris 9 x86. This should be used in conjunction with the Solaris Volume Manager Administration Guide: http://docs.sun.com/db/doc/816-4519 Many Sun customers mirror the root partition on their critical systems to minimize downtime and maximize availability. A mirrored root should provide the customer with a way to have their systems automatically boot from a secondary drive if the primary fails. Note: Solaris Volume Manager(SVM) supplied with Solaris 9 was previously known as Solstice DiskSuite[TM] (SDS). Document Body: Top Currently the task of setting up automatic failover on a SPARC(R) is relatively straight forward, however the same task on an x86 machine is considerably more convoluted and difficult. For a SPARC machine, only a couple of simple commands to set the nvalias in the NVRAM are needed to set up automatic failover to the secondary drive. To set up automatic failover on an x86 machine, the System Administrator needs to modify two files on the boot floppy (boot.rc and bootenv.rc) and ALWAYS boot from the floppy. Setting up automatic failover for a mirrored root: Currently for automatic failover to succeed this solution requires use of the boot floppy. The files boot.rc and bootenv.rc on the boot floppy need to be modified and autoboot must be set. The floppy is used as a primitive type of boot prom, as Solaris does not have control over the varied types of BIOS that can be found on these machines, to allow the system to automatically failover to the alternate boot device. To set up autofailover use the following procedure: 1) Create the boot floppy from Solaris CD2 using dd.exe. 2) Modify the /floppy/floppy0/solaris/boot.rc file: The following line in /floppy/floppy0/boot.rc: run /boot/solaris/bootconf.exe ${confflags} should be changed to: run /boot/solaris/bootconf.exe -n ${confflags} Add these lines immediately after the previous line: cd /options if .streq ( ${root-is-mounted}X , falseX ) echo echo 'Failed to mount primary bootpath:' echo " ${bootpath}" echo echo 'Trying alternate boot disk:' echo " ${altbootpath}" echo setprop bootpath "${altbootpath}" run /boot/solaris/bootconf.exe ${confflags} endif 3) Modify the /floppy/floppy0/solaris/bootenv.rc file as follows. Make sure auto-boot? is set to true: setprop auto-boot? True Set auto-boot-timeout to an appropriately low value: setprop auto-boot-timeout 5 Set altbootpath to your alternate submirror: setprop altbootpath Make sure the bootpath is set to the location of your primary root partition: setprop bootpath e.g. setprop auto-boot? true setprop auto-boot-timeout 5 setprop bootpath /pci@0,0/pci8086,2545@3/pci8086,1460@1d/pci8086,341a@7,1/sd@0,0:a setprop altbootpath /pci@0,0/pci8086,2545@3/pci8086,1460@1d/pci8086,341a@7,1/sd@1,0:a SVM Failed Disk Recovery When one of the disks fails, reboot the system with the boot floppy inserted. During the boot you will be told that there are state databases which no longer exist. You will then be prompted to give the root password for system maintenance. 1) Enter the root password for system maintenance. 2) Execute metadb to see which state databases have failed. 3) Delete references to the dead state databases: Metadb d (eg c0t1d0s3) 4) Reboot the system which should come up normally, however you will see some messages reporting that, for example, metadevice d21 is "unavailable needs maintenance" for each of the slices which were on the failed disk. 5) Execute metastat and make a note of the slices and their disk slice locations. 6) Shutdown the server, replace the failed drive with a replacement and power up again. 7) Create a copy of the good disk's partition information onto the new disk: fdisk W /var/tmp/fdisk.dat /dev/rdsk/c0t0d0p0 fdisk F /var/tmp/fdisk.dat /dev/rdsk/c0t1d0p0 8) Add bootblock as per answerbook to the new drive. http://docs.sun.com/db/doc/806-4073/6jd67r9h9?a=view#disksxadd-45774 e.g. installboot /usr/platform/`uname -i`/lib/fs/ufs/pboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk /dev/rdsk/c0t1d0s2 9) Copy the disk layout of the good disk with the new replacement for the failed disk: prtvtoc /dev/rdsk/ | fmthard s - /dev/rdsk/ 10) Now create new state databases on the new disk: metadb a c3 11) Finally you have to execute metareplace for each failed slice: metareplace -e e.g. metareplace -e d30 c0t1d0s0 metareplace -e d31 c0t1d0s1 metareplace -e d32 c0t1d0s4 metareplace -e d33 c0t1d0s5 metareplace -e d34 c0t1d0s7 You can monitor the progress of the re-sync by either using the metastat command or by firing up the Solaris Management Console (SMC) and double clicking on "The Computer" -> "Storage" -> "Enhanced Storage" -> "Volumes". Issues: There is an issue with reference to booting from the second half of an SVM mirror on certain systems x86 systems. For example, with a failed first half of a mirror on a V60x the system should boot from the second half of the mirror, but instead fails with an error like the following: unable to mount a Solaris root file system from the device: Disk target:SEGATE ST336607LSUN36G 0307 on Adaptec Ultra/320 on Board PCI bus 4, at Dev7, func1 Error message from mount: /pci@0,0/pci8086,2545@3/pci8086,1460@1d/pci8086,341a@7,1/sd@1,0:a Workaround: The easiest solution is to pull the failed first disk and put the second disk in its place. The replacement disk can then be put in the free slot and sync(ed) from there. ================================================================================