oracle grid ocr磁盘组丢失恢复

版本:11.2.0.4.0
Reference : OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) (Doc ID 428681.1)

CRS-4256 CRS-4602 While Replacing Voting Disk (文档 ID 1475588.1)
How to Restore ASM Based OCR After Complete Loss of the CRS Diskgroup on Linux/Unix Systems (文档 ID 1062983.1)

环境检查:

确认ocr,votedisk 磁盘位置。

[grid@prod01 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   309717e185584f38bf0380d7e53e95ee (/dev/asm-diskb) [DATA]
 2. ONLINE   791a04adb2534f28bf4954cfa12289a5 (/dev/asm-diskc) [DATA]
 3. ONLINE   af884b0a019d4f77bf3c0269106a3c95 (/dev/asm-diskd) [DATA]
Located 3 voting disk(s).
[grid@prod01 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
     Version                  :          3
     Total space (kbytes)     :     262120
     Used space (kbytes)      :       2828
     Available space (kbytes) :     259292
     ID                       :  671775796
     Device/File Name         :      +DATA
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

     Cluster registry integrity check succeeded

     Logical corruption check bypassed due to non-privileged user

模拟破坏磁盘

[root@prod01 gpnpd]# dd if=/dev/zero of=/dev/asm-diskb bs=100M count=10
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 1.72446 s, 608 MB/s
[root@prod01 gpnpd]# dd if=/dev/zero of=/dev/asm-diskc bs=100M count=10
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 1.75642 s, 597 MB/s
[root@prod01 gpnpd]# dd if=/dev/zero of=/dev/asm-diskd bs=100M count=10
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 1.91177 s, 548 MB/s
[root@prod01 gpnpd]# 

检查集群状态

[root@prod02 ~]# /u01/app/11.2.0/grid/bin/crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.

重建磁盘组,

在一个节点操作,其它节点处于关闭状态。
/u01/app/11.2.0/grid/bin/crsctl start crs -nocrs -excl
建立data磁盘组,修改 asm_diskstring参数。

[root@prod01 ~]# /u01/app/11.2.0/grid/bin/crsctl start crs -nocrs -excl
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'prod01'
CRS-2676: Start of 'ora.mdnsd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'prod01'
CRS-2676: Start of 'ora.gpnpd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'prod01'
CRS-2672: Attempting to start 'ora.gipcd' on 'prod01'
CRS-2676: Start of 'ora.cssdmonitor' on 'prod01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'prod01'
CRS-2672: Attempting to start 'ora.diskmon' on 'prod01'
CRS-2676: Start of 'ora.diskmon' on 'prod01' succeeded
CRS-2676: Start of 'ora.cssd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'prod01'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'prod01'
CRS-2672: Attempting to start 'ora.ctssd' on 'prod01'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'prod01'
CRS-2676: Start of 'ora.drivers.acfs' on 'prod01' succeeded
CRS-2676: Start of 'ora.ctssd' on 'prod01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'prod01'
CRS-2676: Start of 'ora.asm' on 'prod01' succeeded

[grid@prod01 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Wed Nov 21 13:59:13 2018

Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> create diskgroup DATA normal redundancy
DISK  '/dev/asm-diskb','/dev/asm-diskc','/dev/asm-diskd'
attribute 'compatible.asm'='11.2';  2    3  
Diskgroup created.

SQL>  alter system set asm_diskstring='/dev/*';

SQL> exit

Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[grid@prod01 ~]$ 

恢复ocr,votedisk

ocrconfig -showbackup

root:

/u01/app/11.2.0/grid/bin/ocrconfig -restore /u01/app/11.2.0/grid/cdata/prod-cluster/backup_20181121_134213.ocr

grid

[grid@prod01 ~]$ ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy

prod02     2018/11/21 09:00:45     /u01/app/11.2.0/grid/cdata/prod-cluster/backup00.ocr

prod02     2018/11/21 05:00:45     /u01/app/11.2.0/grid/cdata/prod-cluster/backup01.ocr

prod02     2018/11/21 01:00:44     /u01/app/11.2.0/grid/cdata/prod-cluster/backup02.ocr

prod02     2018/11/20 01:00:42     /u01/app/11.2.0/grid/cdata/prod-cluster/day.ocr

prod02     2018/11/12 21:00:25     /u01/app/11.2.0/grid/cdata/prod-cluster/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available

#root 恢复ocr
[root@prod01 ~]# /u01/app/11.2.0/grid/bin/ocrconfig -restore /u01/app/11.2.0/grid/cdata/prod-cluster/backup_20181121_134213.ocr 

#root 恢复votedisk
[root@prod01 bin]# ./crsctl replace votedisk +DATA
Successful addition of voting disk 96e6e83cf3364f9abf13c9310830d798.
Successful addition of voting disk 3d317981258b4fcfbf64f119f41a7f28.
Successful addition of voting disk 3aa046d1930c4f39bf9bcf3297554d85.
Successfully replaced voting disk group with +DATA.
CRS-4266: Voting file(s) successfully replaced

验证集群状态

关闭集群# crsctl stop crs -f
启动所有节点集群:# crsctl start crs

[root@prod01 bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
ora.DG1.dg
               ONLINE  OFFLINE      prod01                                       
               ONLINE  OFFLINE      prod02                                       
ora.LISTENER.lsnr
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
ora.asm
               ONLINE  ONLINE       prod01                   Started             
               ONLINE  ONLINE       prod02                   Started             
ora.gsd
               OFFLINE OFFLINE      prod01                                       
               OFFLINE OFFLINE      prod02                                       
ora.net1.network
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
ora.ons
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
ora.registry.acfs
               ONLINE  ONLINE       prod01                                       
               ONLINE  ONLINE       prod02                                       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       prod01                                       
ora.cvu
      1        OFFLINE OFFLINE                                                   
ora.oc4j
      1        OFFLINE OFFLINE                                                   
ora.prod01.vip
      1        ONLINE  ONLINE       prod01                                       
ora.prod02.vip
      1        ONLINE  ONLINE       prod02                                       
ora.scan1.vip
      1        ONLINE  ONLINE       prod01                                       
[root@prod01 bin]# ./crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   96e6e83cf3364f9abf13c9310830d798 (/dev/asm-diskb) [DATA]
 2. ONLINE   3d317981258b4fcfbf64f119f41a7f28 (/dev/asm-diskc) [DATA]
 3. ONLINE   3aa046d1930c4f39bf9bcf3297554d85 (/dev/asm-diskd) [DATA]
Located 3 voting disk(s).
[root@prod01 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
     Version                  :          3
     Total space (kbytes)     :     262120
     Used space (kbytes)      :       2816
     Available space (kbytes) :     259304
     ID                       :  671775796
     Device/File Name         :      +DATA
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

     Cluster registry integrity check succeeded

     Logical corruption check succeeded

[root@prod01 bin]#

asm spfile恢复

create spfile='+DATA' from memory.

[grid@prod01 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Wed Nov 21 14:24:06 2018

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> show parameter spfile;

NAME                     TYPE     VALUE
------------------------------------ ----------- ------------------------------
spfile                     string
SQL> show parameter asm_disk             

NAME                     TYPE     VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups                 string
asm_diskstring                 string
SQL> alter system set asm_diskstring='/dev/*';

System altered.

SQL> show parameter asm_disk  

NAME                     TYPE     VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups                 string
asm_diskstring                 string     /dev/*
SQL> create spfile='+data' from memory;

File created.

SQL>

CRS-4602错误故障:

[root@prod01 bin]# ./crsctl replace votedisk +DATA
CRS-4602: Failed 27 to add voting file 1d4482d87eea4f84bfb02d760f922b85.
CRS-4602: Failed 27 to add voting file 6dc3d69ccf204f28bf54faa59949aa32.
CRS-4602: Failed 27 to add voting file 5c44d63828734fbebf1cf9c70052cab6.
Failed to replace voting disk group with +DATA.
CRS-4000: Command Replace failed, or completed with errors.
[root@prod01 bin]#

由于asm spfile文件丢失导致asm_diskstring参数为空,导致该问题:

[grid@prod01 prod01]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Wed Nov 21 14:13:18 2018

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter system set asm_diskstring='/dev/*';

System altered.

SQL>

上一篇:mysql语句insert后取到返回的主键id


下一篇:HTML标签 -base标签|学习笔记