================================================================================
Good reference: 
http://unix-essentials.blogspot.com/2009/03/vcs-implementing-cluster-file-system.html 
================================================================================
Veritas Clustered File-System Installation

Other Pre-Install Requirements

Communications
RSH/SSH must be enabled between your primary server that you are loading from and all other members of the CFS collection and for functional use (explorer grabs and communication).

Heartbeat/IO Fencing Disk Requirements Disk
SCSI-3 reservation bit must be set on ALL Veritas controlled disks, 
not just the coordinator disks.

Network(Solaris)
On a sun box with multiple NICs, change the following EEPROM setting.
This setting will allow each NIC to have its own MAC address rather than shared.

	eeprom local-mac-address=true

Main.cf config
Have the most basic main.cf running (type.cf, cluster (no resources), systems)

	include "types.cf"

	cluster mycluter (
	        )

	system sys1 (
	        )

	system sys2 (
	        )


Make sure NTP is setup on all CFS systems.
So file mod/create dates don't conflict.

Check for SCSI-3 access to disks

	symdev -noport list | grep DEVICE | grep SCSI-3
		
		Change with "symconfigure".


	vxdisk scandisks
	vxfenadm -i /dev/rdsk/$DISK  #displays the SCSI disk query info
	
	vxdiskadd DISK1 DISK2 DISK3   #make vxcoordg

	/opt/VRTSvcs/vxfen/bin/vxfentsthdw -r -g vxcoorddg


Set fencing mode (raw, dmp, ...)

	cp /etc/vxfen.d/vxfenmode_scsi3_raw /etc/vxfenmode
	reboot

	#
	# vxfenconfig -U   #unconfigure fencing
	# vxfenconfig -c   #configure   fencing
	#
	#

###Set vxdg values (dd the following lines)

		###vi /etc/default/vxdg

			###enable_activation=true
			###default_activation_mode=shared-write

Do a configure CVM and CFS Manually on JUST ONE NODE:
	
	/opt/VRTSvxfs/cfs/bin/cfscluster config

	#===================================================================
	###
	###Add to /etc/llttab (if it fits for your site for performance reasons)
	###
	###	set-flow lowwater:200
	###	set-flow highwater:400
	###	set-flow window:200
	###
	###SOLARIS: For older version of VCS/CFS on Solaris
	###
	###	vi /kernel/drv/vcsmm.conf
	###
	###		add:
	###			name="vcsmm" parent="pseudo" slave_members=8192 instance=0;
	###
	#===================================================================


Setup Coordinator Disks - fencing disks
Verify with the EMC Storage group that ALL disks are SCSI-3 PR enabled.  
The following commands step you through creating the disk group for IO Fencing.

Do this on only one of the shared systems (it will error saying it is "owned
by another diskgroup" if you try on the other systems).

	disk1=c4t0d0
	disk2=c4t0d1
	disk3=c4t0d2

	vxdisksetup -i $disk1 format=cdsdisk
	vxdisksetup -i $disk2 format=cdsdisk
	vxdisksetup -i $disk3 format=cdsdisk

	vxdg init vxcoorddg $disk1 $disk2 $disk3

	vxdg -g vxcoorddg set coordinator=on

	#The coordinator attribute disallows add/del of disks to coordinatordg 
	#when "on"

	vxdg -g vxfencoorddg set coordinator=off

This is a ***DESTRUCTIVE*** test of the disks in the disk group

   Must give each disk if you don't give the diskgroup
   Or give the DG and it "learns" the disks

     /opt/VRTSvcs/vxfen/bin/vxfentsthdw -g vxcoorddg  
   OR
     /opt/VRTSvcs/vxfen/bin/vxfentsthdw 


   Or for "read-only" NONDESTRUCTIVE use "-r".  The "-n" says to use rsh.
   You will see skipped and one acceptable failure (for remove keys). 

	/opt/VRTSvcs/vxfen/bin/vxfentsthdw -r -n -g vxcoorddg

You need then to deport the diskgroup and set it to not import.
Do on each sytem to make sure it can import.

	vxdg deport vxcoorddg
	vxdg -t import vxcoorddg
	vxdg deport vxcoorddg
	
Create /etc/vxfendg on all systems

	echo "vxcoorddg" > /etc/vxfendg

Start the fencing daemons

	/etc/init.d/vxfen start
		HPUX: /sbin/init.d/vxfen start
	cat /etc/vxfentab

Update the veritas control deamon version.
This may not be needed.  Do a "vxdctl list" to see  (need version 70).
	vxdctl upgrade

Test keys

	vxfenadm -g all -f /etc/vxfentab

  Do on only one system at a time

	vxfenadm -m -k tmp -f /etc/vxfentab   #adds key
	vxfenadm -x -k tmp -f /etc/vxfentab   #removes key
 
	vxfenadm -a -k tmp -f /etc/vxfentab   #adds key
	vxfenadm -x -k tmp -f /etc/vxfentab   #removes key
  
	vxfenadm -g all -f /etc/vxfentab

If this doesn't work then try to make sure whether we could talk to these disks.
   
	vxfenadm -i /dev/rdsk/CTD    #devices listed in /etc/vxfentab 
	vxfenadm -t /dev/rdsk/CTD    #true and ready for testing

Modify the main.cf, below is an example edit.  This should have been done by 
the /opt/VRTSvxfs/cfs/bin/cfscluster config done earlier.

	more /etc/VRTSvcs/conf/config/main.cf

	#=================================================================
	include "types.cf"
	include "CFSTypes.cf"
	include "CVMTypes.cf"

	cluster mycluster (
	        UseFence = SCSI3
   			HacliUserLevel = COMMANDROOT
			CounterInterval = 5
		)

	//...more stuff...

	group cvm (
        SystemList = { dpspdb01 = 0, dpspdb02 = 1 }
        PrintTree = 0
        AutoFailOver = 0
        Parallel = 1
        AutoStartList = { dpspdb01, dpspdb02 }
        )

        CFSfsckd vxfsckd (
                )

        CVMCluster cvm_clus (
                CVMClustName = p-dps-need-hp
                CVMNodeId = { dpspdb01 = 0, dpspdb02 = 1 }
                CVMTransport = gab
                CVMTimeout = 200
                )

        CVMVxconfigd cvm_vxconfigd (
                Critical = 0
                CVMVxconfigdArgs = { syslog }
                )

        cvm_clus requires cvm_vxconfigd
        vxfsckd requires cvm_clus

	#=================================================================

After updating the main.cf  on all nodes, you will have to start and stop the cluster.

	hastop -all
	hastart

Start other daemons

	/etc/init.d/vcsmm start
	/etc/init.d/lmx start
	/etc/init.d/odm start

Use gabconfig -a to check the status of the drivers. 
Ports a, b, d, and o should show membership on node 0

	gabconfig -a

	# GAB Port Memberships
	==========================================
	Port a gen 4a1c0001 membership 01
	Port b gen g8ty0002 membership 01
	Port d gen 40100001 membership 01
	Port f gen f1990002 membership 01
	Port h gen d8850002 membership 01
	Port o gen f1100002 membership 01
	Port q gen 28d10002 membership 01
	Port v gen 1fc60002 membership 01
	Port w gen 15ba0002 membership 01
	
================================================================================
Setup CFS (Clustered File Systems)

	#
	# make sure you have CVM group
	#
	hastatus -sum

	vxdctl -c mode
	#
	# run the following from the MASTER node
	#

	/etc/vx/bin/vxclustadm nidmap
	/etc/vx/bin/vxclustadm -v nodestate
	cfscluster config
	cfscluster status

	vxdisksetup -i san_vc0_2 format=cdsdisk
	vxdisksetup -i san_vc0_3 format=cdsdisk
	vxdisksetup -i san_vc1_3 format=cdsdisk

	vxdg -s init datadg \
		datadg01=san_vc0_2 datadg02=san_vc0_3 datadg03=san_vc1_3

	cfsdgadm add datadg all=sw

	cfsdgadm display

	vxassist -g datadg make cache 200g
	vxassist -g datadg make grid 500g

	/opt/VRTS/bin/mkfs  -t vxfs /dev/vx/rdsk/datadg/cache
	/opt/VRTS/bin/mkfs  -t vxfs /dev/vx/rdsk/datadg/grid

	cfsmntadm add datadg cache /test/file/etl/appserv/cw/Cache all=rw
	cfsmntadm add datadg grid /test/file/etl/grid  all=rw

	cfsmntadm display -v
	hastatus -sum

================================================================================
Some file contents:

  /etc/vxfendg
      ####i.e.
      vxcoorddg

  /etc/vxfenmode
      vxfen_mode=scsi3
      scsi3_disk_policy=raw

  /etc/vxfentab
      ####i.e.
      /dev/rdsk/c4t2d1
      /dev/rdsk/c6t2d1
      /dev/rdsk/c4t2d2
      /dev/rdsk/c6t2d2
      /dev/rdsk/c4t2d3
      /dev/rdsk/c6t2d3

================================================================================
The output of the gabconfig -a command displays which cluster systems have membership with the modules that have been installed and configured thus far in the installation. The first line indicates that each system (0,1,n..) has membership with the GAB utility, which uses Port a. The ports listed, including port a, are configured for the following functions:

	==========================================
	Port  Function
	a     GAB
	b     I/O fencing
	d     ODM (Oracle Disk Manager)
	f     CFS (Cluster File System)
	h     VCS (had: high availability daemon)
	o     VCSMM driver
	q     QuickLog daemon
	v     CVM (Cluster Volume Manager)
	u     temporarily used by CVM membership joining
	v     CVM membership 
	w     vxconfigd (module for cvm - communication port)
	==========================================

================================================================================
Some vxdg set options

		vxdg -g DG set activation=[off,ro,rw,ew,sr,sw]
			(off, readonly, readwrite, exclusivewrite, sharedread, sharedwrite)

		vxdg -g DG set enable_activation=true
		vxdg -g DG set enable_activation_mode=[off,ro,rw,ew,sr,sw]

		vxdg -g DG set align=[1,8k]
			1block, 8K  #CDS needs 8K

		vxdg -g DG set cds=[on,off]]

		vxdg -g vxfencoorddg set coordinator=off
		vxdg -g `cat /etc/vxfendg` set coordinator=off

		cds, default_activivation_mode, enable_activation may be set in
		/etc/default/vxdg (vxconfigd -k to read w/o reboot)

================================================================================
Troubleshooting IO Fencing

Sometimes the fence disks can be left in unknown states, especially if servers reboot without properly shutting down.  The following commands are to help clear the "keys" that are kept on the disk to maintain consistency.

(The following command verifies the systems that are currently registered with the coordinator disks.)

	vxfenadm -g all -f /etc/vxfentab

If not setup, then:

	/opt/VRTSvcs/vxfen/bin/vxfentsthdw

Clear all the keys on the coordinator disks in addition to the data disks.

	/opt/VRTSvcs/vxfen/bin/vxfenclearpre

Now you can repair the faulted system and reboot ALL nodes of the cluster.

================================================================================
The SFCFS commands are:

	-cfscluster: cluster configuration command

	 	cfscluster status


	-cfsmntadm: adds, deletes, modifies, and sets policy on cluster 
	            mounted file systems

	-cfsdgadm: adds or deletes shared disk groups to/from a cluster 
	           configuration

	-cfsmount/cfsumount: mounts/unmounts a cluster file system on a shared 
	                     volume

	-fsclustadm: The fsclustadm command reports various attributes of a 
	             cluster file system.
				man fsclustadm
				
	-fsadm: The fsadm command can be invoked from the primary or secondary node.
            man fsadm

To determine the primary node for a file system in a cluster type:

	fsclustadm -v showprimary mount_point
                  ^^^^
 
To set the primary node for a file system in a cluster type:

	fsclustadm -v setprimary mount_point
                  ^^^

To determine if the current node is the master CVM node type:

	vxdctl -c mode

To configure and unconfigure the cluster, as well as get a quick overview, type:

	cfscluster (status)

To add/delete/modify or view a shared disk group, type:

	cfsdgadm (display)

To add file systems, snapshots, display and modify resources with the shared filesystem, type:

	cfsmntadm (display)

To mount and unmount the cfs filesytems, type:

	cfsmount [-o cluster] 
		
		man mount_vxfs

	cfsumount

To manually import a CFS DG - note '-s' option.

	vxdg -t -s import $DG

To make them NON shared (force an import, clear config flag)

    vxdg -Cf import $DG


================================================================================
Growing a Storage Foundation Cluster File System
There is a master node for CVM as well as a primary for SFCFS. When growing a
file system, you grow the volume from the CVM master, and then grow the file
system from any SFCFS node. The CVM master and the SFCFS node can be two
different nodes.
To determine the primary file system in a cluster, enter: 
   fsclustadm -v showprimary mount_point

To determine if the current node is the master CVM node, enter: 
	vxdctl -c mode

To actually increase the size of the file system, run the following two
commands. On the master CVM node, enter: 
	vxassist -g shared_disk_group growto volume_name newlength

On any SFCFS node, enter: 
	fsadm -F vxfs -b newsize -r device_name mount_point

================================================================================
vxdctl list
vxdctl protocolversion
vxdctl protocolrange
vxdctl support
vxdctl upgrade
vxdg -T 140 upgrade $DG
vxupgrade -n 7 $MTPT

================================================================================
Adding or removing coordinator disks

Before adding coordinator disks, verify the disks support SCSI-3 persistent
reservations.

1 Log in as root on any cluster node.

2 Import the coordinator disk group. The file /etc/vxfendg includes the name
of the disk group containing the coordinator disks. Type: 

	vxdg -tfC import `cat /etc/vxfendg`
	
	where:
	-t specifies that the disk group is imported only until the system reboots.
	-f specifies that the import is to be done forcibly, which is necessary if
		one or more disks is inaccessible.
	-C specifies any import blocks are removed.

3 To add disks to, or remove them from, the disk group, use the VxVM disk
administrator utility, vxdiskadm.

4 After disks are added or removed, deport the disk group: 
	vxdg deport `cat /etc/vxfendg`

5 Reboot each system in the cluster to make the coordinator disks accessible.

================================================================================
To create a snapshot on a cluster file system
1 Create a VxFS file system on a shared VxVM volume:
	mkfs -F vxfs /dev/vx/rdsk/cfsdg/vol1

	version 7 layout

	104857600sectors,52428800blocksofsize1024,logsize16384 blocks
	unlimited inodes, largefiles not supported
	52428800 data blocks, 52399152 free data blocks
	1600 allocation units of 32768 blocks, 32768 data blocks

2 Mount the file system on all nodes (following previous examples, on
system01 and system02):
	cfsmntadm add cfsdg vol1 /mnt1 all=cluster
	cfsmount /mnt1

	The cfsmntadm command adds an entry to the cluster manager
	configuration, then the cfsmount command mounts the file system on all
	nodes.

3 Add the snapshot on a previously created volume (snapvolin this example)
to the cluster manager configuration:

	cfsmntadm add snapshot cfsdg snapvol /mnt1 /mnt1snap \
			system01=ro

	Note: The snapshot of a cluster file system is accessible only on the node
	where it is created; the snapshot file system itself cannot be cluster
	mounted.

4 Mount the snapshot: 
	cfsmount /mnt1snap

5 A snapped file system cannot be unmounted until all of its snapshots are
unmounted. Unmount the snapshot before trying to unmount the snapped cluster
file system:
	cfsumount /mnt1snap

================================================================================
Enable detailed logging for CvmVolDg:
 
) Edit /opt/VRTSvcs/bin/CVMVolDg/cvmvoldg.lib:
 
From this:
# Uncomment the following to start debugging:
#DEBUG="DEBUG"
 
To This:
# Uncomment the following to start debugging:
DEBUG="DEBUG"     
 
) Enable vxconfigd debug log:
vxconfigd -k -x log -x 9 -x timestamp -x
logfile=/var/VRTSvcs/log/vxconfigd.debug.log > /dev/null 2>&1
 
) Run the cable pull test.
 
) Copy /var/adm/syslog.log /var/adm/syslog.log.cablepull.
 
) Edit /opt/VRTSvcs/bin/CVMVolDg/cvmvoldg.lib:
 
From this:
# Uncomment the following to start debugging:
DEBUG="DEBUG"
 
To This:
# Uncomment the following to start debugging:
# DEBUG="DEBUG"     
 
) Disable vxconfigd debug log:
vxconfigd -k -x syslog > /dev/null 2>&1
 
) Run the vxexplorer. 

================================================================================
REMOVE CFS

hastop -all -force
vi /etc/*vcs/conf/config/main.cf # change/remove CVM/CFS 

vxdisk -o alldgs list| grep mydg |awk '{print "/dev/vx/rdmp/"$1}' > /tmp/mydg 
vxfenadm -r all -f /tmp/HA_dpsdg01
vxfenadm -x -k APGR0001 -f /tmp/HA_dpsdg01
vxfenadm -r all -f /tmp/HA_dpsdg01
vxfenadm -a -k A00 -f /tmp/HA_dpsdg01
vxfenadm -c -k A00 -f /tmp/HA_dpsdg01
vxfenadm -r all -f /tmp/HA_dpsdg01
vxfenadm -g all -f /tmp/HA_dpsdg01

/opt/VRTS/bin/vxclustadm stopnode
/opt/VRTS/bin/fsclustadm cfsdeinit

vi /etc/vxfenmode # vxfen_mode=disabled

/sbin/rc2.d/*odm* stop
/sbin/rc2.d/K*vxfen* stop

reboot

================================================================================