使用Oracle Grid配置Goldengate或其他第三方应用高可用

2022-10-30 08:38:27

1. 概述

Oracle Grid不止能提供自身Oracle Database高可用，还可以为第三方应用提供高可用。

可以为OGG、SharePlex等逻辑复制，Apache等应用提供高可用。

使用Oracle Grid代理第三方应用主要有以下两种方式：

  1. Oracle Grid Infrastructure Agents
  1. Third-Part Script
  1. 官方文档位置：
  2. Clusterware Administration and Deployment Guide
  3. Third-Party Applications Using the Script Agent
  4. Mos文档参考：
  5. Oracle_GoldenGate_Best_Practices_-_Oracle_GoldenGate_high_availability_using_Oracle_Clusterware_v8_6_ID1313703_1_.pdf
  1. 关于第三方应用日志位置
  1. Oracle Grid 11.2如果使用oracle添加资源，则日志位置：
  2. $GRID_HOME/log/{node_name}/agent/crsd/scriptagent_oracle
  4. 12c以后GRID日志也变为标准ADR目录
  5. $GRID_BASE/diag/crs/crs/agent/scriptagent_oracle.trc
  7. # 如果为GRID添加资源，路径或日志名称scriptagent_grid即可。

2. Grid代理第三方脚本

下面测试利用Grid代理第三方脚本形式提供高可用，XAG方式参考官方文档即可。

部署步骤概述：

  1. 配置应用VIP（此VIP不是RAC VIP，仅仅为了应用本身使用），对外提供唯一IP，使切换对应用透明。
  2. 部署goldengate启停第三方脚本。
  3. crsctl加载资源，配置权限。
  4. 测试高可用。

2.1 配置VIP

  1. (1) login as root
  2. # appvipcfg create -network=1 \
  3. -ip=192.168.204.242 \
  4. -vipname=czhvip \
  5. -user=root
  7. (2) 查看配置vip
  8. # crsctl stat res -p |grep -ie .network -ie subnet |grep -ie name -ie subnet
  10. (3) login as root
  11. # crsctl setperm resource czhvip -u user:oracle:r-x
  12. --配置资源使用权限用户,IP资源属主一定必须是root，其他用户无法配置IP，会导致无法启动VIP资源。

2.2 部署OGG

  1. ogg安装部署不在此赘述，可按照以下几种方式：
  2. 1. 使用ACFS作为共享磁盘，OGG软件本身以及dir*相关目录均存放于ACFS文件系统。
  3. ACFS相应版本以及补丁参考下面文档：
  4. ACFS Support On OS Platforms (Certification Matrix). (Doc ID 1369107.1).pdf
  6. 2.使用ACFS存放goldengate的trail文件等，OGG软件本身存放于操作挂载点即可，通过在操作系统相应路径下建立软链接方式指向ACFS中dir*相应目录
  7. $ ln –s /acfs_mount_point/dirdat dirdat
  9. 3.使用例如ocfs2、gpfs等集群文件系统存放

2.2 部署脚本说明

下面脚本仅仅用做示例，实际脚本可以根据不同应用加入相应模块脚本，比如check脚本就需要判断进程状态等等。

Grid 第三方脚本模块说明

  1. 1. Grid 11.2脚本需要包含start/stop/clean/check/abort
  2. --示例脚本
  3. #!/bin/sh
  5. case $1 in
  6. 'start')
  7. echo $(date)' start'>>/tmp/crs.log
  8. exit 0
  9. ;;
  10. 'stop')
  11. echo $(date)' stop'>>/tmp/crs.log
  12. exit 0
  13. ;;
  14. 'clean')
  15. echo $(date)' clean'>>/tmp/crs.log
  16. echo $?'clean' >>/tmp/crs.log
  17. exit 0
  18. ;;
  19. 'check')
  20. echo "CHECK entry point has been called.."
  21. echo $(date)' check'>>/tmp/crs.log
  22. exit 0
  23. ;;
  24. 'abort')
  25. echo $(date)' abort'>>/tmp/crs.log
  26. exit 0
  27. ;;
  28. esac
  30. 2. 模块说明
  31. --主要介绍11gR2引入的两个新模块
  32. --12c以后版本引入了更多模块，这点可以从启动日志中看到。
  33. CLEAN
  34. Clean was introduced with Oracle Clusterware 11g Release 2. It will not be used for Oracle
  35. Clusterware 10g Release 2 or 11g Release 1. Clean is called when there is a need to clean up the
  36. resource. It is a non-graceful operation.
  37. ABORT
  38. Abort was introduced with Oracle Clusterware 11g Release 2. It will not be used for Oracle
  39. Clusterware 10g Release 2 or 11g Release 1. Abort is called if any of the resource components
  40. hang to abort the ongoing action. Abort is not required to be included.
  43. 3.关于脚本中变量说明
  44. 如果start/stop/clean/check/abort对应脚本中启动程序脚本需要依赖环境变量，例如
  45. (1)ogg如果extract配置使用本地ORACLE_SID连接数据库进行捕获，不是使用tnsalias方式连接数据库，则ggsci> start extract时，依赖于环境变量ORACLE_SID,这种情况下，需要在上面脚本中定义好依赖的ORACLE_SID以及ORACLE_HOME变量，因为Grid启动时由于vip属主为root，所以如果vip与ogg资源强依赖时，只能获取到root的用户环境变量，无法获得oracle用户环境变量，会导致资源无法正常启动。
  46. (2)所以环境变量一定要在脚本中完全定义，不要依赖于外部变量，否则将会发生问题后很难排查以及遇到无法启动资源或启动资源无法启动程序中相应进程。

2.3 OGG高可用脚本

下面为OGG连接ASM与版本关系

  1. 如果 Redo Log 存储在 ASM 中，设置 Catpure ASM 连接方式如下：
  2. Oracle 10.2.0.5 或 11.2.0.2 之前版本：
  3. TRANLOGOPTIONS ASMUSER sys@asminst, asmpassword oracle
  4. Oracle 10.2.0.5、11.2.0.2 或之以后版本，GoldenGate 为 11g 或以后版本：
  5. TRANLOGOPTIONS DBLOGREADER
  6. 如果在 AIX 平台数据库的 redo log 使用的是 RAW，则可能需要设置参数：TRANLOGOPTIONS
  7. RAWDEVICEOFFSET，设置此参数：
  8. TRANLOGOPTIONS RAWDEVICEOFFSET 0
  9. 其他平台不需要设置此参数。

下面脚本为未使用ASM或Oracle 10.2.0.5、11.2.0.2 或之以后版本，如果为早期需要调取ASM实例ORACLE_SID，则需要特殊处理

完整示例详细可以参考OracleGoldenGate_Best_Practices-Oracle_GoldenGate_high_availability_using_Oracle_Clusterware_v8_6_ID1313703_1.pdf

  1. #!/bin/sh
  2. # goldengate_action.scr
  4. # 生效oracle用户下环境变量，oracle下环境变量一定要配置相关变量，防止下面启动ogg无法读取相关ORACLE_SID导致启动extract失败
  5. . ~oracle/.bash_profile
  7. # 判断调用脚本是否有选项，如果第一个选项为空，则报错，提示使用选项
  8. [ -z "$1" ]&& echo "ERROR!! Usage $0 <start|stop|abort|clean>"&& exit 99
  10. # 指定goldengate安装目录
  11. GGS_HOME=<set the path here>
  13. #specify delay after start before checking for successful start
  14. start_delay_secs=5
  16. #Include the Oracle GoldenGate home in the library path to start GGSCI,AIX variable is LIBPATH
  17. export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${GGS_HOME}
  19. #set the oracle home to the database to ensure Oracle GoldenGate will get
  20. #the right environment settings to be able to connect to the database
  21. export ORACLE_HOME=<set the ORACLE_HOME path here>
  22. export CRS_HOME=<set the CRS_HOME path here>
  24. #Set NLS_LANG otherwise it will default to US7ASCII
  25. export NLS_LANG=American_America.US7ASCII
  26. logfile=/tmp/crs_gg_start.log
  27. \rm ${logfile}
  29. # define function log.
  30. function log ()
  31. {
  32. DATETIME=`date +%d/%m/%y-%H:%M:%S`
  33. echo $DATETIME "goldengate_action.scr>>" $1
  34. echo $DATETIME "goldengate_action.scr>>" $1 >> $logfile
  35. }
  37. # define function check_process to check goldengate MGR process is runing or not.
  38. #check_process validates that a manager process is running at the PID
  39. #that Oracle GoldenGate specifies.
  40. check_process ()
  41. {
  42. if ( [ -f "${GGS_HOME}/dirpcs/MGR.pcm" ] )
  43. then
  44. pid=`cut -f8 "${GGS_HOME}/dirpcs/MGR.pcm"`
  45. if [ ${pid} = `ps -e |grep ${pid} |grep mgr |awk '{ print $1 }'` ]
  46. then
  47. #manager process is running on the PID . exit success
  48. echo "manager process is running on the PID . exit success">> /tmp/check.out
  49. exit 0
  50. else
  51. #manager process is not running on the PID
  52. echo "manager process is not running on the PID" >> /tmp/check.out
  53. exit 1
  54. fi
  55. else
  56. #manager is not running because there is no PID file
  57. echo "manager is not running because there is no PID file" >> /tmp/check.out
  58. exit 1
  59. fi
  60. }
  62. # call_ggsci is a generic routine that executes a ggsci command
  63. call_ggsci () {
  64. log "entering call_ggsci"
  65. ggsci_command=$1
  66. #log "about to execute $ggsci_command"
  67. log "id= $USER"
  68. cd ${GGS_HOME}
  69. ggsci_output=`${GGS_HOME}/ggsci << EOF
  70. ${ggsci_command}
  71. exit
  72. EOF`
  73. log "got output of : $ggsci_output"
  74. }
  76. case $1 in
  77. 'start')
  78. #Updated by Sourav B (02/10/2011)
  79. # During failover if the “mgr.pcm” file is not deleted at the node crash
  80. # then Oracle clusterware won’t start the manager on the new node assuming the
  81. # manager process is still running on the failed node. To get around this issue
  82. # we will delete the “mgr.prm” file before starting up the manager on the new
  84. # node. We will also delete the other process files with pc* extension and to
  85. # avoid any file locking issue we will first backup the checkpoint files and then
  86. # delete them from the dirchk directory.After that we will restore the checkpoint
  87. # files from backup to the original location (dirchk directory).
  88. log "removing *.pc* files from dirpcs directory..."
  89. rm -f $GGS_HOME/dirpcs/*.pc*
  90. log "creating tmp directory to backup checkpoint file...."
  91. mkdir $GGS_HOME/dirchk/tmp
  92. log "backing up checkpoint files..."
  93. cp $GGS_HOME/dirchk/*.cp* $GGS_HOME/dirchk/tmp
  94. log "Deleting checkpoint files under dirchk......"
  95. rm -f $GGS_HOME/dirchk/*.cp*
  96. log "Restore checkpoint files from backup to dirchk directory...."
  97. cp $GGS_HOME/dirchk/tmp/*.cp* $GGS_HOME/dirchk
  98. log "Deleting tmp directory...."
  99. rm -r $GGS_HOME/dirchk/tmp
  100. log "starting manager"
  101. call_ggsci 'start manager'
  102. #there is a small delay between issuing the start manager command
  103. #and the process being spawned on the OS . wait before checking
  104. log "sleeping for start_delay_secs"
  105. sleep ${start_delay_secs}
  106. #check whether manager is running and exit accordingly
  107. check_process
  108. ;;
  109. 'stop')
  110. #attempt a clean stop for all non-manager processes
  111. call_ggsci 'stop er *'
  112. #ensure everything is stopped
  113. call_ggsci 'stop er *!'
  114. #stop manager without (y/n) confirmation
  115. call_ggsci 'stop manager!'
  116. #exit success
  117. exit 0
  118. ;;
  119. 'check')
  120. check_process
  121. exit 0
  122. ;;
  123. 'clean')
  124. #attempt a clean stop for all non-manager processes
  125. call_ggsci 'stop er *'
  126. #ensure everything is stopped
  127. call_ggsci 'stop er *!'
  128. #in case there are lingering processes
  129. call_ggsci 'kill er *'
  130. #stop manager without (y/n) confirmation
  131. call_ggsci 'stop manager!'
  132. #exit success
  133. exit 0
  134. ;;
  135. 'abort')
  136. #ensure everything is stopped
  137. call_ggsci 'stop er *!'
  138. #in case there are lingering processes
  139. call_ggsci 'kill er *'
  140. #stop manager without (y/n) confirmation
  141. call_ggsci 'stop manager!'
  142. #exit success
  143. exit 0
  144. ;;
  145. esac

2.4 CRSCTL添加ogg Grid资源

  1. # login as oracle:
  2. $ /u01/app/11.2/grid/bin/crsctl add resource oggapp -type cluster_resource -attr "ACTION_SCRIPT='/acfs_mount_point/ogg.sh',CHECK_INTERVAL=30,START_DEPENDENCIES='hard(czhvip) pullup(czhvip)',STOP_DEPENDENCIES='hard(mvggatevip)'"
  4. --脚本位置可以存放于本地oracle用户有读取执行权限的目录，如果存放于本地，则Grid各个节点都需要备份该文件
  5. --如果ogg安装使用acfs，则START_DEPENDENCIES可以配置与ASM强依赖。

上述步骤即已完成第三方应用使用Grid托管，还是非常方便实用的。

3. 遇到问题解决

3.1 无法启动resource

  1. 1. 无法启动
  2. $ crsctl start res czhapp
  3. CRS-2672: Attempting to start 'czhapp' on 'db-oracle-node1'
  4. CRS-2674: Start of 'czhapp' on 'db-oracle-node1' failed
  5. CRS-2679: Attempting to clean 'czhapp' on 'db-oracle-node1'
  6. CRS-2678: 'czhapp' on 'db-oracle-node1' has experienced an unrecoverable failure
  7. CRS-0267: Human intervention required to resume its availability.
  8. CRS-4000: Command Start failed, or completed with errors.
  10. # 如果配置资源属于Oracle，则日志目录为：
  11. $GRID_HOME/log/{node_name}/agent/crsd/scriptagent_oracle
  12. --关键内容如下
  13. 2021-04-26 11:35:07.342: [czhapp][156428032]{1:39006:13462} [clean] Executing action script: /software/crs.sh[clean]
  14. 2021-04-26 11:35:07.397: [ AGFW][156428032]{1:39006:13462} Command: clean for resource: czhapp 1 1 completed with invalid status: 209
  15. 2021-04-26 11:35:07.397: [czhapp][156428032]{1:39006:13462} [check] Executing action script: /software/crs.sh[check]
  16. 2021-04-26 11:35:07.397: [ AGFW][158529280]{1:39006:13462} Agent sending reply for: RESOURCE_CLEAN[czhapp 1 1] ID 4100:717590
  17. 2021-04-26 11:35:07.454: [ AGFW][156428032]{1:39006:13462} Received unknown resource status code: 209
  18. 2021-04-26 11:35:07.455: [ AGFW][158529280]{1:39006:13462} czhapp 1 1 state changed from: CLEANING to: UNKNOWN
  20. 2. 分析
  21. 可以从日志输出看到，识别到了脚本，但是通过在脚本中指定位置配置输出，发现脚本并未真正执行。
  22. 最终排查原因主要为脚本开头未声明脚本类型导致该问题。
  24. 3. 解决
  25. #!/bin/sh
  27. --写脚本还是要规范，以前写脚本偶尔拉下声明部分，并不影响，这次Oracle Grid代理脚本没有声明部分无法启动还是挺意外的，也说明还是要规范。

3.2 OGG无法启动extract

  1. 1. 现象
  2. OCI相关报错，无法连接数据库
  4. 2.分析
  5. AIX:
  6. ps -ef|grep goldengate
  7. ps eauwww <pid>
  8. 查看进程环境变量发现，变量中无ORACLE_SID。
  9. 由于goldengate extract中配置，未配置使用tnsalias方式连接数据库，所以依赖于启动extract时用户操作系统环境变量ORACLE_SID，但是由于appvipcfg配置
  10. 的vip资源未给oracle足够权限，导致使用oracle用户无法启动vip资源，进而导致使用root启动vip资源之后，环境变量无法取到ORACLE_SID，导致未能启动extract。
  12. 3.解决
  13. --login as root
  14. # crsctl setperm resource oggvip -u user:oracle:rwx
  15. --login as oracle 测试
  16. $ crsctl start resource oggvip
  17. --如果上述命令依然无法使oracle启动资源，则继续修改oggvip权限
  18. --login as root
  19. --将other组权限设置为rwx即可解决
  20. # crsctl getperm resource oggvip
  21. # crsctl setperm resource oggvip -u other::rwx

3.3 appvipcfg无法执行

  1. 1. 现象
  2. # ./appvipcfg create -network=1 \
  3. -ip=192.168.204.245 \
  4. -vipname=czhvip \
  5. -user=root
  7. /bin/ls: cannot access /ade/ade_88979932/perl/lib: No such file or directory
  9. 2. 原因
  10. 由于opatch打补丁导致appvipcfg内容发生改变，appvipcfg本身为$GRID_HOME/bin/下的一个脚本文件，不是一个二进制文件，脚本中定义了ORACLE_HOME与ORA_CRS_HOME，由于打补丁导致该文件两个变量不正确，修改为正确路径即可解决。
  11. $ cat /u01/app/11.2/grid/bin/appvipcfg
  12. #!/bin/sh
  13. #
  14. # This script is used for managing
  15. # user mode vip resource.
  16. #
  18. # Do not change the line below for ORACLE_HOME setting
  19. #ORACLE_HOME=/u01/app/11.2/grid
  20. ORACLE_HOME=/ade/ade_19289128/11.2/grid
  21. export ORACLE_HOME
  23. #ORA_CRS_HOME=/u01/app/11.2/grid
  24. ORA_CRS_HOME=/ade/ade_19289128/11.2/grid
  25. export ORA_CRS_HOME
上一篇：Netty 直接内存（堆外内存）溢出分析


下一篇：OGG 的初始化加载