OCR / VOTING DISK MAINTENANCE OPERATIONS: MIGRATION FROM OCFS2 TO ASM

Introduction:

Starting with Oracle 11gR2, Oracle recommendation is to use Oracle ASM to store OCR and Voting Disks. With appropriate redundancy level of the ASM Disk Group being used, Oracle can create required number of Voting Disks as part of installation.

Be sure to:

• Have the root password.
• A valid spfile of ASM

One RAC with two nodes on Node_a and Node_b.
OS : Linux

Impact:

The service will be stopped; no connection to the database is possible during this operation.

I. Backups : Make sure there is a recent copy of the OCR/Voting disk before making any changes

1.1  OCR

 

As the root user:

root@Node_a # ocrconfig -manualbackup

/logiciels/oracle/grid/cdata/node-cluster/backup_20130409_145249.ocr
 

1.2  Voting

root@Node_a # dd if=/oracle/ocfs2/storage/vdsk of=vdsk_bkp

41025+0 records in

41025+0 records out

21004800 bytes (21 MB) copied, 0.526757 seconds, 39.9 MB/s

/oracle/ocfs2/storage

root@Node_a # ll

total 44384

-rw-r----- 1 root    dba  272756736 Apr 10 14:51 ocr

-rw-r----- 1 oragrid dba   21004800 Feb 12 09:32 vdsk

-rw-r--r-- 1 root    root  21004800 Apr 12 15:21 vdsk_bkp

II.  Preparing the ASM Disk Group: +CRS & +FRA

The approach is:

1. Create Two ASM Diskgroup :+CRS ,+FRA     
2. Migrate ocr, voting disks and archive logs to ASM in the new ASM disks : +CRS & +FRA

2.1 Check the status of OCR/Voting

oragrid@uvbacko890a:/home/dba/oragrid> ocrcheck

         Version                  :          3

         Total space (kbytes)     :     262120

         Used space (kbytes)      :       3044

         Available space (kbytes) :     259076

         ID                       :  647776079

         Device/File Name         : /oracle/ocfs2/storage/ocr

                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check bypassed due to non-privileged user

root@Node_a # crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------

 1. ONLINE   11d68d19685d4f20bf026eeb463d23aa (/oracle/ocfs2/storage/vdsk)

Located 1 voting disk(s).

 

2.2  Format new SAN disks

root@uvbacko890a # fdisk /dev/sdm

Device contains neither a valid DOS partition table, nor Sun, SGI or OSFdisklabel building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous

content won't be recoverable.

 

The number of cylinders for this disk is set to 1305.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:

1) software that runs at boot time (e.g., old versions of LILO)

2) booting and partitioning software from other OSs

   (e.g., DOS FDISK, OS/2 FDISK)

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n

Command action

   e   extended

   p   primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-1305, default 1)

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-1305, default 1305):

Using default value 1305

Command (m for help): p

 

Disk /dev/sdm: 10.7 GB, 10737418240 bytes

255 heads, 63 sectors/track, 1305 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System

/dev/sdm1               1        1305    10482381   83  Linux

 

Command (m for help): w

The partition table has been altered!

 

Calling ioctl() to re-read partition table.

Syncing disks.

Idem for /dev/sdn

Idem for /dev/sdo

Idem for /dev/sds

 

 

On the first node:

root@Node_a# /etc/init.d/oracleasm createdisk DISK8 /dev/sdm1

Marking disk "DISK8" as an ASM disk:                       [  OK  ]

root@Node_a # /etc/init.d/oracleasm createdisk DISK9 /dev/sdn1

Marking disk "DISK9" as an ASM disk:                       [  OK  ]

root@Node_a # /etc/init.d/oracleasm createdisk DISK10 /dev/sdo1

Marking disk "DISK10" as an ASM disk:                       [  OK  ]

root@Node_a # /etc/init.d/oracleasm createdisk DISK10 /dev/sds1

 

 

Marking disk "DISK11" as an ASM disk:                       [  OK  ]

 

 

On the seconde node:

/etc/init.d/oracleasm scandisks

 

Check:

On the first:

root@Node_a # /etc/init.d/oracleasm listdisks

DISK1

DISK2

DISK3

DISK4

DISK5

DISK6

DISK7

DISK8

DISK9

DISK10

DISK11

 

 On the second:

root@Node_ba # /etc/init.d/oracleasm listdisks

DISK1

DISK2

DISK3

DISK4

DISK5

DISK6

DISK7

DISK8

DISK9

DISK10

DISK11

 

2.3 Create the ASM disk +CRS

Locate the asmca binary (for example /logiciels/oracle/grid/bin/)

cd /logiciels/oracle/grid/bin

            export DISPLAY=XXXXXXX.212:0.0

./asmca

(Normaly the disk DISK10 should have the same size as DISK8 and DISK9)

 

 

Idem for +FRA

All Disks

 

 

2.4 Check: On each node, as oragrid user

 

 

SQL> select state,name,type from v$asm_diskgroup;

 

STATE       NAME                           TYPE

----------- ------------------------------ ------

MOUNTED     DATA_ASM                       EXTERN

MOUNTED     CRS                            NORMAL

MOUNTED     FRA                            EXTERN

2.5 Check that the new disk group is mount :

On each node:

SQL> select state,name,type from v$asm_diskgroup;

STATE       NAME                           TYPE

----------- ------------------------------ ------

MOUNTED     DATA_ASM                       EXTERN

MOUNTED     CRS                            NORMAL

MOUNTED     FRA                            EXTERN

2.5 Case when a disk group is not mount

If for example on the second node, the status of CRS is not MOUNT, like:

SQL> select state,name,type from v$asm_diskgroup;

STATE       NAME                           TYPE

----------- ------------------------------ ------

MOUNTED     DATA_ASM                       EXTERN

MOUNTED     FRA                            EXTERN

DISMOUNTED  CRS

Then check the status of the resource and start it:

oragrid@Node_b:/oracle/ocfs2/storage> crsctl stat res -t

NAME           TARGET  STATE        SERVER      STATE_DETAILS

Local Resources

ora.CRS.dg

               ONLINE  ONLINE       Node_a

               OFFLINE OFFLINE      Node_b

ora.DATA_ASM.dg

               ……

To start ora.CRS.dg on Node_b:

oragrid@Node_b:/oracle/ocfs2/storage> crsctl start resource ora.CRS.dg -n Node_b

III. OCR Disk

3.1 Add an OCRMIRROR device when only OCR device is defined:

On one node

root# ocrconfig -add +CRS

3.2 Remove the old non-ASM shared OCR

root# ocrconfig -delete /oracle/ocfs2/storage/ocr

3.3 Check the status of OCR

root@Node_a:/home/dba/oragrid> ocrcheck

Status of Oracle Cluster Registry is as follows:

         Version                  :          3

         Total space (kbytes)     :     262120

         Used space (kbytes)      :       3028

         Available space (kbytes) :     259092

         ID                       :  647776079

         Device/File Name         :       +CRS

                                    Device/File integrity check succeeded

         Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

IV. Voting Disk

4.1 Check the status of the Voting disk

 

root@Node_a # crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------

 1. ONLINE   11d68d19685d4f20bf026eeb463d23aa (/oracle/ocfs2/storage/vdsk) []

Located 1 voting disk(s).

4.2 Migrate Voting disk to ASM

root@Node_a #  crsctl replace votedisk +CRS

/logiciels/oracle/grid/log/Node_a/client

root@Node_a # crsctl replace votedisk +CRS

Successful addition of voting disk 22a1974019a04feabfddb6f6ff819926.

Successful addition of voting disk 35d5a0f35db94f07bf5774a36cae4435.

Successful addition of voting disk a09b4c45c86f4fbdbf30f0cdc0ebe446.

Successful deletion of voting disk 475d55cb7dda4f00bf32bf7b3da8cbfc.

Successfully replaced voting disk group with +CRS.

CRS-4266: Voting file(s) successfully replaced

4.3 Check the status of Voting disk

One one node, as oragrid user:

root@Node_a # crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------

1. ONLINE   22a1974019a04feabfddb6f6ff819926 (ORCL:DISK10) [CRS]

      2. ONLINE   35d5a0f35db94f07bf5774a36cae4435 (ORCL:DISK8)  [CRS]

      3. ONLINE   a09b4c45c86f4fbdbf30f0cdc0ebe446 (ORCL:DISK9)  [CRS]

 

CHANGE ASM DISKGROUP REDUNDANCY FROM NORMAL TO EXTERNAL

1.1      Verify the existing

When the disk groups are created with some redundancy say External, Normal or High then its redundancy cannot be changed. Need to change redundancy can arise if:

- DBA's want to switch from Normal/High Redundancy to External Redundancy due to disk space constraints or due to plans of using External methods of maintaining redundancy (like RAID 10 , etc) .

- Switch to ASM based Redundancy i.e converting from External redundancy to Normal/High Redundancy

• This note is for RAC Grid infrastructure 11.2.0.2 

• 2 nodes : node1_name & node2_name

• Linux 64nits

oragrid:

SQL>  select state,name,type from v$asm_diskgroup;

STATE       NAME                           TYPE

----------- ------------------------------ ------

MOUNTED     DATA_ASM                       NORMAL

SQL> col gnum format 99

SQL> col filnum format 999999999999

SQL> col path format a60

SQL> SELECT gnum, filnum, concat('+'||gname,sys_connect_by_path(aname, '/')) path

  2  FROM (SELECT g.name gname, a.parent_index pindex, a.name aname,

  3  a.reference_index rindex, a.group_number gnum,a.file_number filnum

  4  FROM v$asm_alias a,v$asm_diskgroup g

  5  WHERE a.group_number = g.group_number)

  6  START WITH (mod(pindex, power(2, 24))) = 0

  7  CONNECT BY PRIOR rindex = pindex;

GNUM        FILNUM PATH

---- ------------- ------------------------------------------------------------

   1    4294967295 +DATA_ASM/scan-cluster

   1    4294967295 +DATA_ASM/scan-cluster/ASMPARAMETERFILE

   1           253 +DATA_ASM/scan-cluster/ASMPARAMETERFILE/REGISTRY.253.8073

                   78527

   1    4294967295 +DATA_ASM/scan-cluster/OCRFILE

   1           255 +DATA_ASM/scan-cluster/OCRFILE/REGISTRY.255.807379617

   1    4294967295 +DATA_ASM/MY_DB

   1    4294967295 +DATA_ASM/MY_DB/DATAFILE

   1           271 +DATA_ASM/MY_DB/DATAFILE/INDX.271.809516379

   1           270 +DATA_ASM/MY_DB/DATAFILE/DATAX.270.807802575

GNUM        FILNUM PATH

---- ------------- ------------------------------------------------------------

   1           268 +DATA_ASM/MY_DB/DATAFILE/DATAX.268.807796427

   1           264 +DATA_ASM/MY_DB/DATAFILE/UNDOTBS2.264.807705403

   1           259 +DATA_ASM/MY_DB/DATAFILE/USERS.259.807705229

   1           258 +DATA_ASM/MY_DB/DATAFILE/UNDOTBS1.258.807705227

   1           257 +DATA_ASM/MY_DB/DATAFILE/SYSAUX.257.807705225

   1           256 +DATA_ASM/MY_DB/DATAFILE/SYSTEM.256.807705225

   1           269 +DATA_ASM/MY_DB/DATAFILE/DATAX.269.807802377

   1    4294967295 +DATA_ASM/MY_DB/CONTROLFILE

   1           260 +DATA_ASM/MY_DB/CONTROLFILE/Current.260.807705377

   1    4294967295 +DATA_ASM/MY_DB/ONLINELOG

   1           265 +DATA_ASM/MY_DB/ONLINELOG/group_3.265.807705465

GNUM        FILNUM PATH

---- ------------- ------------------------------------------------------------

   1           266 +DATA_ASM/MY_DB/ONLINELOG/group_4.266.807705465

   1           261 +DATA_ASM/MY_DB/ONLINELOG/group_1.261.807705379

   1           262 +DATA_ASM/MY_DB/ONLINELOG/group_2.262.807705381

   1    4294967295 +DATA_ASM/MY_DB/TEMPFILE

   1           263 +DATA_ASM/MY_DB/TEMPFILE/TEMP.263.807705385

   1    4294967295 +DATA_ASM/MY_DB/PARAMETERFILE

   1           267 +DATA_ASM/MY_DB/PARAMETERFILE/spfile.267.807705467

   1           267 +DATA_ASM/MY_DB/spfileMY_DB.ora

28 rows selected.

As oracle:

SQL> select name from v$controlfile

union

select name from v$datafile

union

select name from v$tempfile

union

select member from v$logfile

union

select filename from v$block_change_tracking ;

NAME

--------------------------------------------------------------------------------

+DATA_ASM/MY_DB/controlfile/current.260.807705377

+DATA_ASM/MY_DB/datafile/datax.268.807796427

+DATA_ASM/MY_DB/datafile/datax.269.807802377

+DATA_ASM/MY_DB/datafile/datax.270.807802575

+DATA_ASM/MY_DB/datafile/indx.271.809516379

+DATA_ASM/MY_DB/datafile/sysaux.257.807705225

+DATA_ASM/MY_DB/datafile/system.256.807705225

+DATA_ASM/MY_DB/datafile/undotbs1.258.807705227

+DATA_ASM/MY_DB/datafile/undotbs2.264.807705403

+DATA_ASM/MY_DB/datafile/users.259.807705229

+DATA_ASM/MY_DB/onlinelog/group_1.261.807705379

NAME

--------------------------------------------------------------------------------

+DATA_ASM/MY_DB/onlinelog/group_2.262.807705381

+DATA_ASM/MY_DB/onlinelog/group_3.265.807705465

+DATA_ASM/MY_DB/onlinelog/group_4.266.807705465

+DATA_ASM/MY_DB/tempfile/temp.263.807705385

SQL> show parameter pfile;

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

spfile                               string      +DATA_ASM/MY_DB/spfileMY_DB.ora

SQL> select A.group# as "Group", A.thread#, B.member, round(A.bytes/1024/1024,0) as "Size (Mo)"

from v$log a, v$logfile b

where a.group# = b.group#

order by A.group#, B.member;  2    3    4

Group THREAD#

----- -------

MEMBER

--------------------------------------------------------------------------------

 Size (Mo)

----------

    1       1

+DATA_ASM/MY_DB/onlinelog/group_1.261.807705379

        50

    2       1

+DATA_ASM/MY_DB/onlinelog/group_2.262.807705381

        50

Group THREAD#

----- -------

MEMBER

--------------------------------------------------------------------------------

 Size (Mo)

----------

    3       2

+DATA_ASM/MY_DB/onlinelog/group_3.265.807705465

        50

    4       2

+DATA_ASM/MY_DB/onlinelog/group_4.266.807705465

Group THREAD#

----- -------

MEMBER

--------------------------------------------------------------------------------

 Size (Mo)

----------

        50

1.2      Drop the existing disk group after database backup and create a new disk group with External redundancy

1. Shutdown (immediate) the database and then start up mount.  Take a valid RMAN backup of existing database as:

Method1:

On each node, and as oracle user:

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

Startup mount:

SQL> startup mount;

Method2:

You can also, use the following method, as oracle user:

srvctl stop database -d MY_DB -o immediate

Startup mount:

srvctl start database -d MY_DB -o mount

On the first node:

mkdir –p /livraison/backup/

rman target /

RMAN> run {

    allocate channel chn1 device type DISK format '/livraison/backup/%U';

    backup full database spfile plus archivelog delete all input;

}

2. Make copy of spfile to accessible location:

On the first node, and as oracle user:

SQL> create pfile='/livraison/backup/initMY_DB.ora' from spfile;
SQL> alter database backup controlfile to '/livraison/backup/control.ctl';

3. Shutdown the database

srvctl stop database -d MY_DB -o immediate

4. Connect to ASM Instance and Drop the existing Diskgroups

4.1 Stop asm on the other nodes (except the first one).

 As oargrid user:   

 Method1:              

SQL> shutdown immediate;

ASM diskgroups volume disabled

ASM diskgroups dismounted

ASM instance shutdown

 Method2:

             crsctl stop resource ora.asm -n node2_name -f

4.2 On the first node, and as oragrid user:

sqlplus '/as sysasm'

SQL> drop diskgroup DATA_ASM including contents;

Diskgroup dropped.

If you have such errors:

ERROR at line 1:

ORA-15039: diskgroup not dropped

ORA-15027: active use of diskgroup "DATA_ASM" precludes its dismount

Then, ensure that asm are stopped on each node (except the current one) and do:

   SQL> alter diskgroup DATA_ASM dismount force;

   SQL> drop diskgroup DATA_ASM force including contents;

5. Shutdown ASM Instance on the first node.

On the first node, and as oragrid user:

sqlplus '/as sysasm'

SQL> shutdown immediate;

6. Startup the ASM instance in nomount state and Create the new ASM diskgroup

On each node, as oragrid user:

Two methods to do:

Method 1:

sqlplus '/as sysasm'

SQL> startup nomount;

ORA-00099: warning: no parameter file specified for ASM instance

ASM instance started

Total System Global Area  283930624 bytes

Fixed Size                  2225792 bytes

Variable Size             256539008 bytes

ASM Cache                  25165824 bytes

Method 2:

       srvctl start asm -o nomount

Check:

       crsctl stat res -t

The new diskgroups name should be same as of previous diskgroup, it will facilitate the RMAN restore process.

Locate the asmca binary (for example /logiciels/oracle/grid/bin/)

cd /logiciels/oracle/grid/bin/

export DISPLAY=XXXX

./asmca

7. Connect to the RDBMS instance and startup in no mount state using pfile

On the first node,  as oragrid user:

srvctl start instance -d MY_DB -n node1_name -o nomount;

On the first node, and as oracle user:

cd /livraison/backup

SQL> startup nomount pfile='initMY_DB.ora';

sql> create spfile='+DATA_ASM/MY_DB/spfileMY_DB.ora' from pfile='/livraison/backup/initMY_DB.ora';

Check:

    ASMCMD> ls -l DATA_ASM/MY_DB/

    Type           Redund  Striped  Time             Sys  Name

                                                     Y    PARAMETERFILE/

                                                     N    spfileMY_DB.ora => +DATA_ASM/MY_DB/PARAMETERFILE/spfile.256.811956759

    ASMCMD> ls -l DATA_ASM/MY_DB/PARAMETERFILE/

    Type           Redund  Striped  Time             Sys  Name

    PARAMETERFILE  UNPROT  COARSE   APR 05 15:00:00  Y    spfile.256.811956759

At this moment, shutdown and restart the database on no mount.

SQL> Shutdown immediate;

SQL> Startup nomount;

8. Now restore the controlfile and backup's using RMAN

On the first node:

rman target/

RMAN> restore controlfile from '/livraison/backup/control.ctl';

Starting restore at 04-APR-13

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=16 instance=MY_DB1 device type=DISK

channel ORA_DISK_1: copied control file copy

output file name=+DATA_ASM/MY_DB/controlfile/current.256.811867053

Finished restore at 04-APR-13

RMAN> alter database mount;
RMAN> restore database;

Starting restore at 04-APR-13

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=16 instance=MY_DB1 device type=DISK

channel ORA_DISK_1: starting datafile backup set restore

channel ORA_DISK_1: specifying datafile(s) to restore from backup set

channel ORA_DISK_1: restoring datafile 00001 to +DATA_ASM/MY_DB/datafile/system.256.807705225

channel ORA_DISK_1: restoring datafile 00002 to +DATA_ASM/MY_DB/datafile/sysaux.257.807705225

channel ORA_DISK_1: restoring datafile 00003 to +DATA_ASM/MY_DB/datafile/undotbs1.258.807705227

channel ORA_DISK_1: restoring datafile 00004 to +DATA_ASM/MY_DB/datafile/users.259.807705229

channel ORA_DISK_1: restoring datafile 00005 to +DATA_ASM/MY_DB/datafile/undotbs2.264.807705403

channel ORA_DISK_1: restoring datafile 00006 to +DATA_ASM/MY_DB/datafile/datax.271.811862519

channel ORA_DISK_1: restoring datafile 00007 to +DATA_ASM/MY_DB/datafile/indx.270.811862579

channel ORA_DISK_1: reading from backup piece /livraison/backup/04o68323_1_1

channel ORA_DISK_1: piece handle=/livraison/backup/04o68323_1_1 tag=TAG20130404T133147

channel ORA_DISK_1: restored backup piece 1

channel ORA_DISK_1: restore complete, elapsed time: 00:01:15

Finished restore at 04-APR-13

RMAN> alter database open resetlogs;

9. Restart the other nodes

On the first node, and as oracle user:

srvctl start instance -d MY_DB -n node2_name;

1.3      Control

As oragrid:

SQL>  select state,name,type from v$asm_diskgroup;

STATE       NAME                           TYPE

----------- ------------------------------ ------

MOUNTED     DATA_ASM                       EXTERN

/home/dba/oragrid> crsctl status resource ora.DATA_ASM.dg

NAME=ora.DATA_ASM.dg

TYPE=ora.diskgroup.type

TARGET=ONLINE               , ONLINE

STATE=ONLINE on node1_name, ONLINE on node2_name