编前语:
小编成功在阿里云上搭建Oracle RAC断断续续花了1个月的时间，中间的过程简直是痛苦，网上很多的资料都是基于图形化界面的，也有静默模式但是步骤都非常的粗略，可能是写这样的步骤太费时间了。这篇文章可以说是从零开始，所有的步骤都有截图。希望可以帮到大家。如果你是才开始接触RAC请仔细阅读文中的每一个字，特别关注里面的一些提示，可以让你节约很多时间
背景：
公司使用的数据库目前为Oracle单机模式，为了提高数据的可用性。需要搭建Oracle集群。于是开始在网上搜索，搭建RAC的文章很多，但是大部分都是基于虚拟机的。公司的服务器全部运行在阿里云上，于是借鉴下面大神的文章开始搭建，大神的文章过于豪放，细节没有覆盖，所有撰写这篇博文
参考文档:
https://yq.aliyun.com/articles/88303
重点:
为了绕开阿里云对于组播的限制，集群中节点的通信利用N2N
步骤:
资源准备
a. ECS 服务器2台, CentOS 6.8 x86-64
b. 阿里云块存储2块 （截至目前，块存储还处于公测，需要提交申请开通，本人提交申请以后等了3天，让后提工单去催的)
20G用于选举，250G用于数据库以及Grid安装
c. Oracle 以及 Grid安装文件(版本:11g)，下载地址如下
d. N2N软件
e. ASM软件
http://www.oracle.com/technetwork/server-storage/linux/asmlib/rhel6-1940776.html
1.修改主机名
#vi /etc/sysconfig/network节点1# Created by anacondaNETWORKING=yesHOSTNAME=rac1NETWORKING_IPV6=noPEERNTP=noNOZEROCONF=yes节点2# Created by anacondaNETWORKING=yesHOSTNAME=rac2NETWORKING_IPV6=noPEERNTP=noNOZEROCONF=yes
2. 安装Oracle准备，节点规划完全复制参考文档里面的
存储组件和参考文档有点不同，这个块存储以后肯定很贵的，所以生产环境可以根据数据库大小来配置。我是80*3=240G，所以我申请了250G的共享块存储
组件
类型
卷大小
ASM卷组名
ASM 冗余
设备名
OCR/VF
ASM
5G
DG_OCR
NORMAL
/dev/vdb1,/dev/vdb2,/dev/vdb3
数据&归档
ASM
80G
DG_DATA
NORMAL
/dev/vdc1,/dev/vdc2,/dev/vdc3
块存储挂载以后：
Disk /dev/vda: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0008d73a
Device Boot Start End Blocks Id System
/dev/vda1 * 2048 125827071 62912512 83 Linux
Disk /dev/vdb: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xf8683fec
Device Boot Start End Blocks Id System
Disk /dev/vdc: 268.4 GB, 268435456000 bytes, 524288000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xdd02467a
Device Boot Start End Blocks Id System
3.安装N2N软件
#将N2N软件上传到阿里云服务器
#在两个节点都要安装N2Nunzip n2n-master.zip cd n2n-mastermake #如果make 提示openssl出错，安装openssl（命令: yum install -y openssl openssl-devel)make PREFIX=/opt/n2n install 安装完毕后启动supernode服务(N2N支持两个supernode，我这里只在节点1上面开启supernode)nohup /opt/n2n/sbin/supernode -l 65530 & #用其中一个节点来当N2N的supernode,我用的节点1， 节点1/opt/n2n/sbin/edge -d edge0 -a 10.10.10.101 -s 255.255.255.0 -c dtstack -k dtstack -l 172.18.56.21:65530 -E -r #这里的172.18.56.21是我ECS服务器的eth0接口IP，也就是supernode的真实IP,如果你起了两个supernode可以直接在-l参数里面添加第二个节点，如-l 172.18.56.21:65530,172.18.56.22:65530/opt/n2n/sbin/edge -d edge1 -a 192.168.100.101 -s 255.255.255.0 -c dtstack -k dtstack -l 172.18.56.21:65530 -E -r #配置完成以后节点会多两个虚拟网卡[root@rac1 n2n-master]# ifconfigedge0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1400 inet 10.10.10.101 netmask 255.255.255.0 broadcast 10.10.10.255 ether 4e:c7:9c:a7:73:95 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 edge1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1400 inet 192.168.100.101 netmask 255.255.255.0 broadcast 192.168.100.255 ether 46:76:79:d8:1b:eb txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.18.56.21 netmask 255.255.240.0 broadcast 172.18.63.255 ether 00:16:3e:08:48:66 txqueuelen 1000 (Ethernet) RX packets 19861 bytes 24810887 (23.6 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 6122 bytes 2000836 (1.9 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 节点2 /opt/n2n/sbin/edge -d edge0 -a 10.10.10.102 -s 255.255.255.0 -c dtstack -k dtstack -l 172.18.56.21:65530 -E -r/opt/n2n/sbin/edge -d edge1 -a 192.168.100.102 -s 255.255.255.0 -c dtstack -k dtstack -l 172.18.56.21:65530 -E -r
4.修改/etc/hosts
节点110.10.10.101 rac110.10.10.102 rac210.10.10.103 rac1-vip10.10.10.104 rac2-vip192.168.100.101 rac1-priv192.168.100.102 rac2-priv10.10.10.105 scan-ip#172.18.56.21 rac1节点210.10.10.101 rac110.10.10.102 rac210.10.10.103 rac1-vip10.10.10.104 rac2-vip192.168.100.101 rac1-priv192.168.100.102 rac2-priv10.10.10.105 scan-ip#172.10.56.22 rac2
5.修改内核参数
#vi /etc/sysctl.conffs.aio-max-nr = 1048576fs.file-max = 6815744kernel.shmall =7864320 #内存大小/4096分页大小 kernel.shmmax = 52451655680 #内存大小kernel.shmmni = 4096kernel.sem = 250 32000 100 128net.ipv4.ip_local_port_range = 9000 65500net.core.rmem_default = 262144net.core.rmem_max = 4194304net.core.wmem_default = 262144net.core.wmem_max = 1048586vm.nr_hugepages = sga/pagesize #开启hugepage，oracle锁定内存防止换出 #sysctl –p
6.配置NTP
#oracle rac必须进行时钟同步，如果没有时钟同步，按照下面的方式配置，让oracle自身来解决/sbin/service ntpd stopchkconfig ntpd offmv /etc/ntp.conf /etc/ntp.conf.org
7.关闭防火墙，阿里云自动的CentOS是关闭了防火强的，你是用虚拟机，需要运行这一步
chkconfig --list iptableschkconfig iptables offchkconfig --list iptablesservice iptables stopservice network restart关闭selinux/etc/selinux/configdisable
8.修改用户限制
#vi /etc/security/limits.conforacle soft nofile 4096oracle hard nofile 65536oracle soft nproc 2047oracle hard nproc 16384oracle soft stack 10240grid soft nofile 4096grid hard nofile 65536grid soft nproc 2047grid hard nproc 16384grid soft stack 10240* soft memlock 18874368* hard memlock 18874368
9. /etc/pam.d/login
echo "session required pam_limits.so" >>/etc/pam.d/login
10.安装需要的包
一般要安装下面的软件包yum install -y compat-libstdc++-33yum install -y elfutils-libelf-develyum install -y gcc-c++yum install -y kshyum install -y libaio-develyum install -y libstdc++-develyum install -y libXpyum install -y numactl-develyum install -y unixODBCyum install -y unixODBC-develyum install -y compat-libcap1.x86_64
11.格式化共享磁盘，这点和参考文档有点区别，他是通过3块盘进行冗余，我是一块盘上划三个分区。如果是生产hua环境，建议划三块盘，将选举磁盘和数据盘分到3个不同的盘。由于是共享磁盘，这个操作在一个节点上进行就OK
[root@rac1 ~]# fdisk /dev/vdbWelcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them.Be careful before using the write command. Command (m for help): nPartition type: p primary (0 primary, 0 extended, 4 free) e extendedSelect (default p): pPartition number (1-4, default 1): 1First sector (2048-41943039, default 2048): Using default value 2048Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039): +5GPartition 1 of type Linux and of size 5 GiB is set Command (m for help): nPartition type: p primary (1 primary, 0 extended, 3 free) e extendedSelect (default p): pPartition number (2-4, default 2): First sector (10487808-41943039, default 10487808): Using default value 10487808Last sector, +sectors or +size{K,M,G} (10487808-41943039, default 41943039): +5GPartition 2 of type Linux and of size 5 GiB is set Command (m for help): nPartition type: p primary (2 primary, 0 extended, 2 free) e extendedSelect (default p): pPartition number (3,4, default 3): 3First sector (20973568-41943039, default 20973568): Using default value 20973568Last sector, +sectors or +size{K,M,G} (20973568-41943039, default 41943039): +5GPartition 3 of type Linux and of size 5 GiB is set Command (m for help): wThe partition table has been altered! Calling ioctl() to re-read partition table.Syncing disks.
[root@rac1 ~]# fdisk /dev/vdcWelcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them.Be careful before using the write command. Command (m for help): nPartition type: p primary (0 primary, 0 extended, 4 free) e extendedSelect (default p): pPartition number (1-4, default 1): 1First sector (2048-524287999, default 2048): Using default value 2048Last sector, +sectors or +size{K,M,G} (2048-524287999, default 524287999): +80GPartition 1 of type Linux and of size 80 GiB is set Command (m for help): nPartition type: p primary (1 primary, 0 extended, 3 free) e extendedSelect (default p): pPartition number (2-4, default 2): First sector (167774208-524287999, default 167774208): Using default value 167774208Last sector, +sectors or +size{K,M,G} (167774208-524287999, default 524287999): +80GPartition 2 of type Linux and of size 80 GiB is set Command (m for help): nPartition type: p primary (2 primary, 0 extended, 2 free) e extendedSelect (default p): pPartition number (3,4, default 3): First sector (335546368-524287999, default 335546368): Using default value 335546368Last sector, +sectors or +size{K,M,G} (335546368-524287999, default 524287999): +80GPartition 3 of type Linux and of size 80 GiB is set Command (m for help): wThe partition table has been altered! Calling ioctl() to re-read partition table.Syncing disks.[root@iZwz9igcce8m634htwm1xwZ ~]# fdisk -l Disk /dev/vda: 64.4 GB, 64424509440 bytes, 125829120 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk label type: dosDisk identifier: 0x0008d73a Device Boot Start End Blocks Id System/dev/vda1 * 2048 125827071 62912512 83 Linux Disk /dev/vdb: 21.5 GB, 21474836480 bytes, 41943040 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk label type: dosDisk identifier: 0xf8683fec Device Boot Start End Blocks Id System/dev/vdb1 2048 10487807 5242880 83 Linux/dev/vdb2 10487808 20973567 5242880 83 Linux/dev/vdb3 20973568 31459327 5242880 83 Linux Disk /dev/vdc: 268.4 GB, 268435456000 bytes, 524288000 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk label type: dosDisk identifier: 0xdd02467a Device Boot Start End Blocks Id System/dev/vdc1 2048 167774207 83886080 83 Linux/dev/vdc2 167774208 335546367 83886080 83 Linux/dev/vdc3 335546368 503318527 83886080 83 Linux[root@rac1~]#
12.安装ASM，这里安装会升级kernel到el7(用CentOS 7装集群应该会好一点)
yum install -y http://download.oracle.com/otn_software/asmlib/oracleasmlib-2.0.4-1.el6.x86_64.rpm
yum install -y http://oss.oracle.com/projects/oracleasm-support/dist/files/RPMS/rhel6/amd64/2.1.8/oracleasm-support-2.1.8-1.el6.x86_64.rpm
13.创建grid,oracle用户（grid主要用于管理集群，oracle主要用于管理数据库，所以待会在配置ASM的时候要把用户设置为grid，不然后面会出现磁盘权限的问题)
/usr/sbin/groupadd -g 501 oinstall/usr/sbin/groupadd -g 502 asmadmin/usr/sbin/groupadd -g 503 dba/usr/sbin/groupadd -g 504 oper/usr/sbin/groupadd -g 505 asmdba/usr/sbin/groupadd -g 506 asmoper/usr/sbin/useradd -u 501 -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid/usr/sbin/useradd -u 502 -g oinstall -G dba,asmdba,oper oracle #配置环境变量#Grid#vi /home/grid/.bash_profileumask 022export ORACLE_BASE=/u01/app/grid_baseexport ORACLE_HOME=/u01/app/grid_homeexport ORACLE_SID=+ASM1 #第二个节点+ASM2export PATH=/usr/sbin:$PATHexport PATH=$ORACLE_HOME/bin:$PATH #Oracle#vi /home/oracle/.bash_profileumask 022export ORACLE_BASE=/u01/app/oracleexport ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1export ORA_GRID_HOME=/u01/app/grid_homeexport ORACLE_UNQNAME=OAODZ1 #第二个节点orcl2export ORACLE_SID=OAODZ1 #第二个节点orcl2export PATH=/usr/sbin:$PATHexport PATH=$ORACLE_HOME/bin:$PATHexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib:/usr/libexport CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlibexport NLS_LANG=AMERICAN_AMERICA.ZHS16GBKexport NLS_DATE_FORMAT="YYYY-MM-DD HH24:MI:SS"export TNS_ADMIN=$ORACLE_HOME/network/adminexport LANG=en_US.UTF-8