这个迁移测试是基于某银行测试环境的一个需求，将现有ASM磁盘迁移到更小的ASM磁盘组上，从而将之前的ASM磁盘给其他项目组用，这个过程中OCR和VOTINGdisk盘的迁移最为关键，我们在实施前进行了测试。下面是详细的测试过程。
SQL> create pfile='/home/grid/asmspfile.ora' from spfile;
File created.
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[grid@rac2 ~]$ ls
asmspfile.ora Desktop Documents Downloads Music Pictures Public Templates Videos
[grid@rac2 ~]$ more asmspfile.ora
+ASM2.asm_diskgroups='ASMVG1'#Manual Mount
+ASM1.asm_diskgroups='ASMVG1'#Manual Mount
*.asm_diskstring='/dev/asm*'
*.asm_power_limit=1
*.diagnostic_dest='/oracle/grid'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
SQL> l
1* select name,path,state from v$asm_disk
SQL> /
NAME PATH STATE
------------------------------ -------------------- --------
ASMVG1_0000 /dev/asm_disk2 NORMAL
ASMOCR_0000 /dev/asm_disk1 NORMAL
ASMOCRNEW_0000 /dev/asm_disk3 NORMAL
SQL> l
1* select group_number,state,name,type,total_mb,free_mb from v$asm_diskgroup
SQL> /
GROUP_NUMBER STATE NAME TYPE TOTAL_MB FREE_MB
------------ ----------- ------------------------------ ------ ---------- ----------
1 MOUNTED ASMOCR EXTERN 2048 1652
2 MOUNTED ASMVG1 EXTERN 20480 18414
3 MOUNTED ASMOCRNEW EXTERN 1024 929
重启集群，看新磁盘状态是否正常。
这里正常
[root@rac2 rules.d]# ocrconfig -add +ASMOCRNEW
[root@rac2 rules.d]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2948
Available space (kbytes) : 259172
ID : 531513161
Device/File Name : +ASMOCR
Device/File integrity check succeeded
Device/File Name : +ASMOCRNEW
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
看集群日志
2019-07-05 00:40:44.938:
[crsd(9016)]CRS-1007:The OCR/OCR mirror location was replaced by +ASMOCRNEW.
看orc配置文件
[root@rac2 rac2]# more /etc/oracle/ocr.loc
#Device/file getting replaced by device +ASMOCRNEW
ocrconfig_loc=+ASMOCR
ocrmirrorconfig_loc=+ASMOCRNEW
迁移votedisk
[root@rac2 rac2]# su - grid
/oracle/grid
/oracle/asm
+ASM2
[grid@rac2 ~]$ crsctl query css votedisk;
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 450d509964764f3dbfe9d8001c63bf9a (/dev/asm_disk1) [ASMOCR]
Located 1 voting disk(s).
[grid@rac2 ~]$ crsctl replace votedisk +ASMOCRNEW
Successful addition of voting disk 247abfc7d6674fbabfd4fa2cefd46fea.
Successful deletion of voting disk 450d509964764f3dbfe9d8001c63bf9a.
Successfully replaced voting disk group with +ASMOCRNEW.
CRS-4266: Voting file(s) successfully replaced
[grid@rac2 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 247abfc7d6674fbabfd4fa2cefd46fea (/dev/asm_disk3) [ASMOCRNEW]
Located 1 voting disk(s).
现在votedisk迁移到新磁盘组
两个节点都查下磁盘组和磁盘信息，两个节点状态一样
SQL> l
1* select group_number,name,path from v$asm_disk
SQL> /
GROUP_NUMBER NAME PATH
------------ ------------------------------ ------------------------------
1 ASMOCRNEW_0000 /dev/asm_disk3
3 ASMVG1_0000 /dev/asm_disk2
2 ASMOCR_0000 /dev/asm_disk1
SQL> select group_number,name,state,type from v$asm_diskgroup;
GROUP_NUMBER NAME STATE TYPE
------------ ------------------------------ ----------- ------
1 ASMOCRNEW MOUNTED EXTERN
2 ASMOCR MOUNTED EXTERN
3 ASMVG1 MOUNTED EXTERN
重建ASM 的spfile
SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +ASMOCR/rac-cluster/asmparamet
erfile/registry.253.1005566595
SQL> create pfile='/home/grid/asmspfile1.ora' from spfile;
File created.
SQL> create spfile ='+ASMOCRNEW' from pfile='/home/grid/asmspfile1.ora'
2 ;
File created.
此时ASM 的spfile 创建到新的磁盘组
查看两个节点的gpnp配置
节点1：
ASM-Profile id="asm" DiscoveryString="/dev/asm*" SPFile="+ASMOCRNEW/rac-cluster/asmparameterfile/registry.253.1012783945"/
节点2：
ASM-Profile id="asm" DiscoveryString="/dev/asm*" SPFile="+ASMOCRNEW/rac-cluster/asmparameterfile/registry.253.1012783945"/>
重启集群，观察集群日志
[cssd(8034)]CRS-1605:CSSD voting file is online: /dev/asm_disk3; details in /oracle/asm/log/rac1/cssd/ocssd.log.
该记录说明找到了新的votedisk
此时集群状态正常
ASM实例查看参数文件
[grid@rac1 ~]$ sqlplus / as sysasm
SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +ASMOCRNEW/rac-cluster/asmpara
meterfile/registry.253.1012783
945
此时可以明确，我们测试库迁移的环境报错votedisk没有发现，说明根本无法访问买游戏账号地图ASM磁盘文件，这是要分析的重点
替换votdisk 后，ASM参数文件的信息
[grid@rac2 ~]$ more asmspfile1.ora
+ASM1.asm_diskgroups='ASMVG1','ASMOCRNEW'#Manual Mount
+ASM2.asm_diskgroups='ASMVG1','ASMOCRNEW'#Manual Mount
*.asm_diskstring='/dev/asm*'
*.asm_power_limit=1
*.diagnostic_dest='/oracle/grid'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
asm的正常状态
SQL> show parameter disk;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups string ASMVG1, ASMOCRNEW
asm_diskstring string /dev/asm*
SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +ASMOCRNEW/rac-cluster/asmparameterfile/registry.253.1012783945
一个正常的迁移ocr和votedisk后的ASM参数文件的内容
[grid@rac2 peer]$ more /home/grid/asmpfilenew.ora
+ASM1.asm_diskgroups='ASMVG1','ASMOCRNEW'#Manual Mount
+ASM2.asm_diskgroups='ASMVG1','ASMOCRNEW'#Manual Mount
*.asm_diskstring='/dev/asm*'
*.asm_power_limit=1
*.diagnostic_dest='/oracle/grid'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
做OCR手工备份，然后破坏掉，看状态，尝试恢复。
/oracle/asm/cdata/rac-cluster/backup_20190705_064714.ocr
dd if=/dev/zero of=/dev/sde bs=1M count=1000
重启集群，此时集群日志报错
[cssd(9493)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /oracle/asm/log/rac2/cssd/ocssd.log
2019-07-05 06:52:02.397: [ CSSD][3734849280]clssnmReadDiscoveryProfile: voting file discovery string(/dev/asm*) <<<<<gpnp profile
2019-07-05 06:52:02.397: [ CSSD][3734849280]clssnmvDDiscThread: using discovery string /dev/asm* for initial discovery
2019-07-05 06:52:02.397: [ SKGFD][3734849280]Discovery with str:/dev/asm*:
2019-07-05 06:52:02.397: [ SKGFD][3734849280]UFS discovery with :/dev/asm*:
2019-07-05 06:52:02.397: [ SKGFD][3734849280]Execute glob on the string /dev/asm*
2019-07-05 06:52:02.397: [ SKGFD][3734849280]running stat on disk:/dev/asm_disk3
2019-07-05 06:52:02.398: [ SKGFD][3734849280]running stat on disk:/dev/asm_disk2
2019-07-05 06:52:02.399: [ SKGFD][3734849280]running stat on disk:/dev/asm_disk1
2019-07-05 06:52:02.399: [ SKGFD][3734849280]Fetching UFS disk :/dev/asm_disk1:
2019-07-05 06:52:02.399: [ SKGFD][3734849280]Fetching UFS disk :/dev/asm_disk2:
2019-07-05 06:52:02.399: [ SKGFD][3734849280]Fetching UFS disk :/dev/asm_disk3:
2019-07-05 06:52:02.399: [ SKGFD][3734849280]OSS discovery with :/dev/asm*:
2019-07-05 06:52:02.399: [ SKGFD][3734849280]Handle 0x7fe4c8140c30 from lib :UFS:: for disk :/dev/asm_disk1:
2019-07-05 06:52:02.400: [ SKGFD][3734849280]Handle 0x7fe4c8139460 from lib :UFS:: for disk :/dev/asm_disk2:
2019-07-05 06:52:02.400: [ SKGFD][3734849280]Handle 0x7fe4c813ee00 from lib :UFS:: for disk :/dev/asm_disk3:
2019-07-05 06:52:02.400: [ SKGFD][3734849280]Lib :UFS:: closing handle 0x7fe4c8140c30 for disk :/dev/asm_disk1: <<<<<< disk oerr closing
2019-07-05 06:52:02.400: [ SKGFD][3734849280]Lib :UFS:: closing handle 0x7fe4c8139460 for disk :/dev/asm_disk2:
2019-07-05 06:52:02.400: [ SKGFD][3734849280]Lib :UFS:: closing handle 0x7fe4c813ee00 for disk :/dev/asm_disk3:
2019-07-05 06:52:02.400: [ CSSD][3734849280]clssnmvDiskVerify: Successful discovery of 0 disks <<<<<<<< no disks found
2019-07-05 06:52:02.400: [ CSSD][3734849280]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2019-07-05 06:52:02.400: [ CSSD][3734849280]clssnmvFindInitialConfigs: No voting files found
启动集群
crsctl start crs -excl -nocrs
创建磁盘组
create diskgroup asmocrnew external redundancy disk '/dev/asm_disk3' attribute 'compatible.asm'='11.2';
SQL> create diskgroup asmocrnew external redundancy disk '/dev/asm_disk3' attribute 'compatible.asm'='11.2';
Diskgroup created.
恢复OCR
[root@rac2 rac2]# ocrconfig -restore /oracle/asm/cdata/rac-cluster/backup_20190705_064714.ocr
[root@rac2 rac2]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2944
Available space (kbytes) : 259176
ID : 531513161
Device/File Name : +ASMOCR
Device/File integrity check succeeded
Device/File Name : +ASMOCRNEW
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
在新创建磁盘创建ASM spfile
SQL> create spfile='+ASMOCRNEW' from pfile='/home/grid/asmpfilenew.ora';
File created.
SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string
SQL> shutdown immediate;
ASM diskgroups dismounted
ASM instance shutdown
SQL> startup pfile='/home/grid/asmpfilenew.ora'
ASM instance started
Total System Global Area 1135747072 bytes
Fixed Size 2260728 bytes
Variable Size 1108320520 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[grid@rac2 ~]$ exit
logout
[root@rac2 rac2]# crsctl replace votedisk +ASMOCRNEW
Successful addition of voting disk 592c9b3c12a24fcebf1394e2b7b9397a.
Successfully replaced voting disk group with +ASMOCRNEW.
CRS-4266: Voting file(s) successfully replaced
votedisk已经回来
[root@rac2 rac2]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 592c9b3c12a24fcebf1394e2b7b9397a (/dev/asm_disk3) [ASMOCRNEW]
Located 1 voting disk(s).
重启集群 <<<<<恢复。