测试环境存储磁盘问题，导致RAC一个节点CRS启动失败
linux 5.6 *64, 2节点RAC 11.2.0.4
如下，按照日志，说明信息。
主机重启，发现CRS进程并未自动启动完成。
a1:/picclife/app/oracle$ crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4535: Cannot communicate with Cluster Ready Services CRS-4530: Communications failure contacting Cluster Synchronization Services daemon CRS-4534: Cannot communicate with Event Manager 查询当前节点信息，发现ASM实例 OFFLINE a1:/picclife/app/grid$ crsctl stat res -t -init -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.asm 1 ONLINE OFFLINE Instance Shutdown ora.cluster_interconnect.haip 1 ONLINE OFFLINE ora.crf 1 ONLINE ONLINE a1 ora.crsd 1 ONLINE OFFLINE ora.cssd 1 ONLINE OFFLINE STARTING ora.cssdmonitor 1 ONLINE ONLINE a1 ora.ctssd 1 ONLINE OFFLINE ora.diskmon 1 OFFLINE OFFLINE ora.evmd 1 ONLINE OFFLINE ora.gipcd 1 ONLINE ONLINE a1 ora.gpnpd 1 ONLINE ONLINE a1 ora.mdnsd 1 ONLINE ONLINE a1
手工启动ASM实例，报错 grid$sqlplus / as sysasm SQL> startup ORA-01078: failure in processing system parameters ORA-29701: unable to connect to Cluster Synchronization Service
查询集群Alert,发现启动资源失败报错，提升表决磁盘信息存在问题，并且指向日志 grid$cd $ORACLE_HOME/log/node_name/ grid$tail -200f a*.log 2020-01-18 10:46:30.622: [ohasd(2344)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'a1'. 2020-01-18 10:46:32.278: [cssd(4941)]CRS-1713:CSSD daemon is started in clustered mode 2020-01-18 10:46:32.329: [cssd(4941)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /picclife/app/11.2.0/grid/log/a1/cssd/ocssd.log 2020-01-18 10:46:33.920: [ohasd(2344)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE 2020-01-18 10:46:33.920: [ohasd(2344)]CRS-2769:Unable to failover resource 'ora.diskmon'. 2020-01-18 10:46:47.335: [cssd(4941)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /picclife/app/11.2.0/grid/log/a1/cssd/ocssd.log 2020-01-18 10:47:02.340: [cssd(4941)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /picclife/app/11.2.0/grid/log/a1/cssd/ocssd.log 根据集群alert日志，指向，查询ocssd.log日志，
clssnmvDiskVerify: Successful discovery of 0 disks 未发现存在一块磁盘！！！ $ tail -200f /picclife/app/11.2.0/grid/log/a1/cssd/ocssd.log 2020-01-18 10:47:47.358: [ CSSD][1087162688]clssnmReadDiscoveryProfile: voting file discovery string(/dev/asm*) 2020-01-18 10:47:47.358: [ CSSD][1087162688]clssnmvDDiscThread: using discovery string /dev/asm* for initial discovery 2020-01-18 10:47:47.358: [ SKGFD][1087162688]Discovery with str:/dev/asm*: 2020-01-18 10:47:47.358: [ SKGFD][1087162688]UFS discovery with :/dev/asm*: 2020-01-18 10:47:47.358: [ SKGFD][1087162688]Execute glob on the string /dev/asm* 2020-01-18 10:47:47.358: [ SKGFD][1087162688]OSS discovery with :/dev/asm*: 2020-01-18 10:47:47.358: [ CSSD][1087162688]clssnmvDiskVerify: Successful discovery of 0 disks 2020-01-18 10:47:47.358: [ CSSD][1087162688]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery 2020-01-18 10:47:47.358: [ CSSD][1087162688]clssnmvFindInitialConfigs: No voting files found 2020-01-18 10:47:47.358: [ CSSD][1087162688](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds 2020-01-18 10:47:47.486: [ CSSD][1100994880]clssscSelect: cookie accept request 0xdee1a0 2020-01-18 10:47:47.486: [ CSSD][1100994880]clssgmAllocProc: (0x122c840) allocated 2020-01-18 10:47:47.486: [ CSSD][1100994880]clssgmClientConnectMsg: properties of cmProc 0x122c840 - 1,2,3,4,5 2020-01-18 10:47:47.486: [ CSSD][1100994880]clssgmClientConnectMsg: Connect from con(0x1416) proc(0x122c840) pid(2686) version 11:2:1:4, properties: 1,2,3,4,5 2020-01-18 10:47:47.486: [ CSSD][1100994880]clssgmClientConnectMsg: msg flags 0x0000 2020-01-18 10:47:47.725: [ CSSD][1100994880]clssscSelect: cookie accept request 0x1222de0 2020-01-18 10:47:47.725: [ CSSD][1100994880]clssscevtypSHRCON: getting client with cmproc 0x1222de0 2020-01-18 10:47:47.725: [ CSSD][1100994880]clssgmRegisterClient: proc(4/0x1222de0), client(73/0x122d7d0) 2020-01-18 10:47:47.725: [ CSSD][1100994880]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are allowed before lease acquisition proc(0x1222de0) client(0x122d7d0) 查询测试环境，默认存储是/dev/asm*磁盘，未发现asm磁盘 [root@a1 dev]# ls -lrt asm* ls: asm*: No such file or directory [root@a1 ~]# fdisk -l 存储问题，未加载ASM磁盘，系统磁盘问题，丢盘了，重新虚拟机加载磁盘，重启主机后，问题解决。