一，简介
二，配置虚拟机
1，创建虚拟机
(1)添加三块儿网卡：
| 主节点 | 二节点 |
| eth0: 公网 192.168.1.20/24 NAT | eth0： 公网 192.168.1.21/24 NAT |
| eth1：私网 192.168.2.20/24 host-only | eth1: 私网 192.168.2.21/24 host-only |
| eth2: DHCP Bridged | eth2： DHCP Bridged |
| 主机名： testdb11a | 主机名： testdb11b |
(2)添加五块儿磁盘：其中一块儿分配20G的空间供系统使用，其余四块儿为ASM磁盘，作为共享
创建4块儿独立磁盘，大小分配1GB，选择SCSI，独立永久
现在分配
共享磁盘放到单独的一个目录中，在复制二节点的时候方便寻找，取名字分别为：asm0.vmdk asm1.vmdk asm2.vmdk asm3.vmdk（一定要有后缀.vmdk）
Finsh~~
下面的三块执行同样的操作！
接下来，四块共享磁盘分别分配到和系统硬盘完全不同的总线下面
总体如下：
2，安装操作系统
安装的时候在这里需要注意：
只选择sda，其余的盘对勾全部去掉，继续~
全部默认，软件只要有图形界面即可，主机名，IP地址，和需要的软件我们在安装完操作系统后按需安装
安装11g的时候如果不适用VNS或者X-manager等远程图形界面工具，需要安装vm-tools，否则在开启图形界面的时候会特别的闪
3，配置 ----主节点
(1)关闭iptables和selinux
查看状态
[root@localhost ~]# /etc/init.d/iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination
RH-Firewall--INPUT all -- 0.0.0.0/ 0.0.0.0/ Chain FORWARD (policy ACCEPT)
num target prot opt source destination
RH-Firewall--INPUT all -- 0.0.0.0/ 0.0.0.0/ Chain OUTPUT (policy ACCEPT)
num target prot opt source destination Chain RH-Firewall--INPUT ( references)
num target prot opt source destination
ACCEPT all -- 0.0.0.0/ 0.0.0.0/
ACCEPT icmp -- 0.0.0.0/ 0.0.0.0/ icmp type
ACCEPT esp -- 0.0.0.0/ 0.0.0.0/
ACCEPT ah -- 0.0.0.0/ 0.0.0.0/
ACCEPT udp -- 0.0.0.0/ 224.0.0.251 udp dpt:
ACCEPT udp -- 0.0.0.0/ 0.0.0.0/ udp dpt:
ACCEPT tcp -- 0.0.0.0/ 0.0.0.0/ tcp dpt:
ACCEPT all -- 0.0.0.0/ 0.0.0.0/ state RELATED,ESTABLISHED
ACCEPT tcp -- 0.0.0.0/ 0.0.0.0/ state NEW tcp dpt:
REJECT all -- 0.0.0.0/ 0.0.0.0/ reject-with icmp-host-prohibited
关闭防火墙
[root@localhost ~]# service iptables stop
Flushing firewall rules: [ OK ]
Setting chains to policy ACCEPT: filter [ OK ]
Unloading iptables modules: [ OK ] [root@localhost ~]# chkconfig iptables off 查看
[root@localhost ~]# chkconfig --list | grep iptables
iptables :off :off :off :off :off :off :off [root@localhost ~]# service iptables status Firewall is stopped.
禁用seLinux
查看selinux状态
[root@localhost ~]# getenforce
Enforcing [root@localhost ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - SELinux is fully disabled.
SELINUX=enforcing
# SELINUXTYPE= type of policy in use. Possible values are:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.
SELINUXTYPE=targeted
修改SELINUX=enforcing为
SELINUX=disabled
注：这里需要重启服务器，这里我们等所有的配置都完成后再重启
(2)网络配置
a.配置IP地址
修改配置文件
第一块网卡
[root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth0 # Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth0
BOOTPROTO=static
IPADDR=192.168.1.20
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
HWADDR=:0C::0B:FD:9E
ONBOOT=yes
第二块儿网卡
[root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth1
# Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth1
BOOTPROTO=static
IPADDR=192.168.2.20
NETMASK=255.255.255.0
GATEWAY=192.168.2.1
HWADDR=00:0C:29:0B:FD:A8
ONBOOT=yes
第三块网卡（保持默认）
[root@localhost ~]# vi /etc/sysconfig/network-scripts/ifcfg-eth2 # Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth2
BOOTPROTO=dhcp
HWADDR=:0C::0B:FD:B2
ONBOOT=no
b.修改主机名
[root@localhost ~]# vi /etc/sysconfig/network NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=testdb11a [root@localhost ~]# hostname testdb11a
c.配置hosts文件
说明：如果不适用DNS，在/etc/hosts文件中必须包含虚假的SCAN IP信息
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
#上面两行不能注释
# Public
192.168.1.20 testdb11a
192.168.1.21 testdb11b
# Private
192.168.2.20 testdb11a-priv
192.168.2.21 testdb11b-priv
# Virtual
192.168.1.22 testdb11a-vip
192.168.1.23 testdb11b-vip
# SCAN
192.168.1.24 testdb11-scan
192.168.1.25 testdb11-scan
192.168.1.26 testdb11-scan
说明：SCAN 地址不应该在hosts文件中定义，而是在DNS中定义和public IP地址在同一个网段内的3个循环地址。在这里安装使用hosts文件避免了SCAN 地址，这种方式只实用于11.2.0.2以前的版本，11。2.0.2以后的版本，必须配置DNS，我们安装的是11.2.0.3.0版本，所以DNS是必须配置的，下面使用dnsmasq配置一个简单的DNS
在上面的/etc/hosts文件中加入3个和public网段相同的没有使用的iP地址，编辑下面的文件(nameserver后面可以是127.0.0.，也可以是本地具体的IP地址)为了方便我们就选其中的一个节点做DNS服务器
[root@testdb11a ~]# more /etc/resolv.conf
search localdomain
nameserver 192.168.1.20
启动dnsmasq服务
[root@testdb11a e22489]# /etc/init.d/dnsmasq start
Starting dnsmasq: [ OK ]
查看
[root@testdb11a e22489]# nslookup testdb11-scan
Server: 192.168.1.20
Address: 192.168.1.20# Name: testdb11-scan
Address: 192.168.1.26
Name: testdb11-scan
Address: 192.168.1.24
Name: testdb11-scan
Address: 192.168.1.25
(3)关于时间同步
节点之间的时间同步对于oracle RAC来说是非常重要的，在oracle11g r2之前需要通过操作系统搭建NTP服务完成节点之间时间的同步。在oracle 11g 开始新增加了一个CTSS(Cluster Time Synchronization Services，集群时间同步服务)，来完成节点间时间的同步，在安装Grid Infrastructure的时候，如果没有发现有NTP服务，则会自动安装ctss服务。这只对于RAC节点间时间的同步，对于其他系统来说是不起作用的。要使用oracle自己的CTSS，除了要停止、禁止自动重启NTP服务外，/etc/ntp.conf配置文件也不允许存在.(When the installer finds that the NTP protocol is not active, the Cluster Time Synchronization Service is automatically installed in active mode and synchronizes the time across the nodes. If NTP is found configured, then the Cluster Time Synchronization Service is started in observer mode, and no active time synchronization is performed by Oracle Clusterware within the cluster. )
[root@localhost ~]# service ntpd stop
Shutting down ntpd: [ OK ]
[root@localhost ~]# chkconfig ntpd off
[root@localhost ~]# chkconfig --list | grep ntpd
ntpd :off :off :off :off :off :off :off
[root@localhost ~]# mv /etc/ntp.conf /etc/ntp.conf.
[root@localhost ~]# rm /var/run/ntpd.pid ###这个文件是维护nptd进程pid的
如果想要使用NTP服务进行时间的同步，必须在"/etc/sysconfig/ntpd"中加入“-x” 标记
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"
然后重启ntpd服务
#service ntpd restart
(4)内核参数及安全设置
修改/etc/sysctl.conf 添加下面内容
fs.aio-max-nr =
fs.file-max =
kernel.shmall =
kernel.shmmax =
kernel.shmmni =
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem =
net.ipv4.ip_local_port_range =
net.core.rmem_default=
net.core.rmem_max=
net.core.wmem_default=
net.core.wmem_max=
生效
sysctl -p
修改/etc/security/limits.conf
加入
oracle soft nproc
oracle hard nproc
oracle soft nofile
oracle hard nofile grid soft nproc
grid hard nproc
grid soft nofile
grid hard nofile
在/etc/pam.d/login 中加入
session required pam_limits.so
(5)创建用户和组，以及所需要的目录
创建用户和组
groupadd -g oinstall
groupadd -g dba
groupadd -g oper
groupadd -g asmadmin
groupadd -g asmoper
groupadd -g asmdba
useradd -g oinstall -G dba,asmdba,oper oracle
useradd -g oinstall -G asmadmin,asmdba,asmoper,oper,dba grid
设置密码
passwd oracle
passwd grid
创建所需目录并给于相应的权限
mkdir -p /u01/app/grid
mkdir -p /u01/app/11.2./grid
chown -R grid:oinstall /u01
mkdir -p /u01/app/oracle
chown oracle:oinstall /u01/app/oracle
chmod -R /u01
(6)安装所需要的rpm包
RHEL 5和Oracle Linux 5所需的包如下
binutils-2.17.50.0.
compat-libstdc++--3.2.
compat-libstdc++--3.2. ( bit)
elfutils-libelf-0.125
elfutils-libelf-devel-0.125
gcc-4.1.
gcc-c++-4.1.
glibc-2.5-
glibc-2.5- ( bit)
glibc-common-2.5
glibc-devel-2.5
glibc-devel-2.5 ( bit)
glibc-headers-2.5
ksh-
libaio-0.3.
libaio-0.3. ( bit)
libaio-devel-0.3.
libaio-devel-0.3. ( bit)
libgcc-4.1.
libgcc-4.1. ( bit)
libstdc++-4.1.
libstdc++-4.1. ( bit)
libstdc++-devel 4.1.
make-3.81
sysstat-7.0.2
unixODBC-2.2.11 (32-bit)or laterunixODBC-devel-2.2.11 (64-bit)or laterunixODBC-2.2.11 (64-bit)or later
检查
rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n' binutils \
compat-libstdc++- \
elfutils-libelf \
elfutils-libelf-devel \
gcc \
gcc-c++ \
glibc \
glibc-common \
glibc-devel \
glibc-headers \
ksh \
libaio \
libaio-devel \
libgcc-4.1. \
libstdc++ \
libstdc++-devel \
make \
sysstat \
unixODBC \
unixODBC-devel
安装显示 not installed的包
使用yum安装，避免依赖关系的检查
配置yum源
[root@testdb11a ~]# mount /dev/cdrom /mnt
mount: block device /dev/cdrom is write-protected, mounting read-only
[root@testdb11a ~]#
[root@testdb11a ~]#
[root@testdb11a ~]# cat /etc/yum.repos.d/dvd.repo
[dvd]
name=install dvd
baseurl=file:///mnt/Server
enabled=
gpgcheck=
[root@testdb11a ~]# yum clean all
Loaded plugins: katello, product-id, security, subscription-manager
Updating certificate-based repositories.
Unable to read consumer identity
Cleaning up Everything
[root@testdb11a ~]#yum list
安装
[root@testdb11a ~]# rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n' binutils compat-libstdc++- elfutils-libelf elfutils-libelf-devel gcc gcc-c++ glibc glibc-common glibc-devel glibc-headers ksh libaio libaio-devel libgcc-4.1. libstdc++ libstdc++-devel make sysstat unixODBC unixODBC-devel
binutils-2.17.50.0.-.el5 (x86_64)
compat-libstdc++--3.2.- (x86_64)
compat-libstdc++--3.2.- (i386)
elfutils-libelf-0.137-.el5 (x86_64)
elfutils-libelf-0.137-.el5 (i386)
elfutils-libelf-devel-0.137-.el5 (i386)
elfutils-libelf-devel-0.137-.el5 (x86_64)
package gcc is not installed
package gcc-c++ is not installed
glibc-2.5- (x86_64)
glibc-2.5- (i686)
glibc-common-2.5- (x86_64)
package glibc-devel is not installed
package glibc-headers is not installed
ksh--.el5 (x86_64)
libaio-0.3.- (x86_64)
libaio-0.3.- (i386)
package libaio-devel is not installed
libgcc-4.1.-.el5 (x86_64)
libgcc-4.1.-.el5 (i386)
libstdc++-4.1.-.el5 (x86_64)
libstdc++-4.1.-.el5 (i386)
package libstdc++-devel is not installed
make-3.81-.el5 (x86_64)
package sysstat is not installed
package unixODBC is not installed
package unixODBC-devel is not installed
[root@testdb11a ~]# yum install -y gcc gcc-c++ glibc-develglibc-headerslibaio-devel libstdc++-devel sysstat unixODBC ---把显示not install的包都跟在后面
Loaded plugins: katello, product-id, security, subscription-manager
Updating certificate-based repositories.
Unable to read consumer identity
Setting up Install Process
No package glibc-develglibc-headerslibaio-devel available.
Resolving Dependencies
--> Running transaction check
---> Package gcc.x86_64 :4.1.-.el5 set to be updated
--> Processing Dependency: glibc-devel >= 2.2.- for package: gcc
---> Package gcc-c++.x86_64 :4.1.-.el5 set to be updated
---> Package libstdc++-devel.i386 :4.1.-.el5 set to be updated
dvd/filelists | 2.8 MB :
---> Package libstdc++-devel.x86_64 :4.1.-.el5 set to be updated
---> Package sysstat.x86_64 :7.0.-.el5 set to be updated
---> Package unixODBC.i386 :2.2.-.el5 set to be updated
--> Processing Dependency: unixODBC-libs = 2.2.-.el5 for package: unixODBC
--> Processing Dependency: libodbcinst.so. for package: unixODBC
--> Processing Dependency: libodbc.so. for package: unixODBC
---> Package unixODBC.x86_64 :2.2.-.el5 set to be updated
--> Running transaction check
---> Package glibc-devel.x86_64 :2.5- set to be updated
--> Processing Dependency: glibc-headers = 2.5- for package: glibc-devel
--> Processing Dependency: glibc-headers for package: glibc-devel
---> Package unixODBC-libs.i386 :2.2.-.el5 set to be updated
---> Package unixODBC-libs.x86_64 :2.2.-.el5 set to be updated
--> Running transaction check
---> Package glibc-headers.x86_64 :2.5- set to be updated
--> Processing Dependency: kernel-headers >= 2.2. for package: glibc-headers
--> Processing Dependency: kernel-headers for package: glibc-headers
--> Running transaction check
---> Package kernel-headers.x86_64 :2.6.-.el5 set to be updated
--> Finished Dependency Resolution
Dependencies Resolved
=====================================================================================================================================================
Package Arch Version Repository Size
=====================================================================================================================================================
Installing:
gcc x86_64 4.1.-.el5 dvd 5.3 M
gcc-c++ x86_64 4.1.-.el5 dvd 3.8 M
libstdc++-devel i386 4.1.-.el5 dvd 2.8 M
libstdc++-devel x86_64 4.1.-.el5 dvd 2.8 M
sysstat x86_64 7.0.-.el5 dvd k
unixODBC i386 2.2.-.el5 dvd k
unixODBC x86_64 2.2.-.el5 dvd k
Installing for dependencies:
glibc-devel x86_64 2.5- dvd 2.4 M
glibc-headers x86_64 2.5- dvd k
kernel-headers x86_64 2.6.-.el5 dvd 1.4 M
unixODBC-libs i386 2.2.-.el5 dvd k
unixODBC-libs x86_64 2.2.-.el5 dvd k
Transaction Summary
=====================================================================================================================================================
Install Package(s)
Upgrade Package(s)
Total download size: M
Downloading Packages:
-----------------------------------------------------------------------------------------------------------------------------------------------------
Total MB/s | MB :
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : unixODBC-libs /
Installing : kernel-headers /
Installing : glibc-headers /
Installing : glibc-devel /
Installing : libstdc++-devel /
Installing : unixODBC /
Installing : sysstat /
Installing : libstdc++-devel /
Installing : unixODBC-libs /
Installing : gcc /
Installing : gcc-c++ /
Installing : unixODBC /
Installed products updated.
Installed:
gcc.x86_64 :4.1.-.el5 gcc-c++.x86_64 :4.1.-.el5 libstdc++-devel.i386 :4.1.-.el5 libstdc++-devel.x86_64 :4.1.-.el5
sysstat.x86_64 :7.0.-.el5 unixODBC.i386 :2.2.-.el5 unixODBC.x86_64 :2.2.-.el5
Dependency Installed:
glibc-devel.x86_64 :2.5- glibc-headers.x86_64 :2.5- kernel-headers.x86_64 :2.6.-.el5 unixODBC-libs.i386 :2.2.-.el5
unixODBC-libs.x86_64 :2.2.-.el5
Complete!
最后检查
[root@testdb11a ~]# rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n' binutils compat-libstdc++- elfutils-libelf elfutils-libelf-devel gcc gcc-c++ glibc glibc-common glibc-devel glibc-headers ksh libaio libaio-devel libgcc-4.1. libstdc++ libstdc++-devel make sysstat unixODBC unixODBC-devel
binutils-2.17.50.0.-.el5 (x86_64)
compat-libstdc++--3.2.- (x86_64)
compat-libstdc++--3.2.- (i386)
elfutils-libelf-0.137-.el5 (x86_64)
elfutils-libelf-0.137-.el5 (i386)
elfutils-libelf-devel-0.137-.el5 (i386)
elfutils-libelf-devel-0.137-.el5 (x86_64)
gcc-4.1.-.el5 (x86_64)
gcc-c++-4.1.-.el5 (x86_64)
glibc-2.5- (x86_64)
glibc-2.5- (i686)
glibc-common-2.5- (x86_64)
glibc-devel-2.5- (x86_64)
glibc-headers-2.5- (x86_64)
ksh--.el5 (x86_64)
libaio-0.3.- (x86_64)
libaio-0.3.- (i386)
libaio-devel
libgcc-4.1.-.el5 (x86_64)
libgcc-4.1.-.el5 (i386)
libstdc++-4.1.-.el5 (x86_64)
libstdc++-4.1.-.el5 (i386)
libstdc++-devel-4.1.-.el5 (i386)
libstdc++-devel-4.1.-.el5 (x86_64)
make-3.81-.el5 (x86_64)
sysstat-7.0.-.el5 (x86_64)
unixODBC-2.2.-.el5 (x86_64)
unixODBC-2.2.-.el5 (i386)
unixODBC-devel-2.2.-.el5 (x86_64)
unixODBC-devel-2.2.-.el5 (i386)
已经全部安装~~~
(7)设置grid和oracle的环境变量
grid用户：
[root@localhost ~]# su - grid
[grid@testdb11a ~]$ vi .bash_profile # .bash_profile # Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi # User specific environment and startup programs PATH=$PATH:$HOME/bin #export PATH
##################################################
# User specific environment and startup programs #
##################################################
export ORACLE_BASE=/u01/app/grid
####export ORACLE_HOME=$ORACLE_BASE/grid/product/11.2.0
export ORACLE_HOME=/u01/app/11.2.0/grid ####这里一定不能使用上面的安装目录，否则在安装grid Infrastructure的时候包INS-32026
export ORACLE_PATH=$ORACLE_BASE/common/oracle/sql:.:$ORACLE_HOME/rdbms/admin
export ORACLE_SID=+ASM1
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin:$ORACLE_HOME/bin
export ORACLE_TERM=xterm
#export TNS_ADMIN=$ORACLE_HOME/network/admin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp
##################################################
# set NLS_LANG to resolve messy code in SQLPLUS #
##################################################
export NLS_LANG=AMERICAN_AMERICA.WE8ISO8859P1
##################################################
# Shell setting. #
##################################################
umask
set -o vi
export PS1="\${ORACLE_SID}@`hostname` \${PWD}$ "
oracle用户：
[root@localhost ~]# su - oracle
dbrac1@testdb11a /home/oracle$ cat .bash_profile
# .bash_profile # Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi # User specific environment and startup programs PATH=$PATH:$HOME/bin #export PATH
##################################################
# User specific environment and startup programs#
##################################################
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.
export ORACLE_PATH=$ORACLE_BASE/common/oracle/sql:.:$ORACLE_HOME/rdbms/admin
export ORACLE_SID=db111
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin:$ORACLE_HOME/bin
export ORACLE_TERM=xterm
#export TNS_ADMIN=$ORACLE_HOME/network/admin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp
##################################################
# set NLS_LANG to resolve messy code in SQLPLUS
##################################################
export NLS_LANG=AMERICAN_AMERICA.WE8ISO8859P1
##################################################
# Shell setting.
##################################################
umask
set -o vi
export PS1="\${ORACLE_SID}@`hostname` \${PWD}$ "
(8)创建共享磁盘
[root@testdb11a dev]# ls sd*
sda sda1 sda2 sdb sdc sdd sde
分区
[root@testdb11a dev]# fdisk /dev/sdb Command (m for help): n
Command action
e extended
p primary partition (-)
p
Partition number (-):
First cylinder (-, default ):
Using default value
Last cylinder or +size or +sizeM or +sizeK (-, default ):
Using default value Command (m for help): w
The partition table has been altered! Calling ioctl() to re-read partition table.
Syncing disks.
后面的sdc，sdd，sde执行相同的操作
[root@testdb11a dev]# ls sd*
sda sda1 sda2 sdb sdb1 sdc sdc1 sdd sdd1 sde sde1
绑定裸设备
编辑文件/etc/sysconfig/rawdevice
[root@testdb11a ~]# vi /etc/sysconfig/rawdevices # raw device bindings
# format: <rawdev> <major> <minor>
# <rawdev> <blockdev>
# example: /dev/raw/raw1 /dev/sda1
# /dev/raw/raw2
/dev/raw/raw1 /dev/sdb1
/dev/raw/raw2 /dev/sdc1
/dev/raw/raw3 /dev/sdd1
/dev/raw/raw4 /dev/sde1
启动rawdevice
[root@testdb11a ~]# /etc/init.d/rawdevices start
Assigning devices:
/dev/raw/raw1 --> /dev/sdb1
/dev/raw/raw1: bound to major , minor
/dev/raw/raw2 --> /dev/sdc1
/dev/raw/raw2: bound to major , minor
/dev/raw/raw3 --> /dev/sdd1
/dev/raw/raw3: bound to major , minor
/dev/raw/raw4 --> /dev/sde1
/dev/raw/raw4: bound to major , minor
done
查看
[root@testdb11a ~]# raw -qa
/dev/raw/raw1: bound to major , minor
/dev/raw/raw2: bound to major , minor
/dev/raw/raw3: bound to major , minor
/dev/raw/raw4: bound to major , minor
修改设备权限
[root@testdb11a raw]# ll
total
crw------- root root , Sep : raw1
crw------- root root , Sep : raw2
crw------- root root , Sep : raw3
crw------- root root , Sep : raw4
[root@testdb11a raw]# chown grid:asmdba /dev/raw/raw[-]
[root@testdb11a raw]# chmod /dev/raw/raw[-]
[root@testdb11a raw]# ll
total
crw-rw---- grid asmdba , Sep : raw1
crw-rw---- grid asmdba , Sep : raw2
crw-rw---- grid asmdba , Sep : raw3
crw-rw---- grid asmdba , Sep : raw4
设置重启自动修改设备权限
编辑/etc/rc.d/rc.local
加入
chown grid:asmdba /dev/raw/raw[-]
chmod /dev/raw/raw[-]
PS：需要安装vm-tools
4，配置副节点
(1)关闭系统
[root@testdb11a ~]# shutdown -h Broadcast message from root (pts/) (Mon Sep :: ): The system is going down for system halt NOW!
(2)克隆主节点
把克隆的节点放到对应的目录下
(3)在主节点和二节点的虚拟机.vmx参数文件中加入
disk.locking="false"
diskLib.dataCacheMaxSize = ""
diskLib.dataCacheMaxReadAheadSize = ""
diskLib.DataCacheMinReadAheadSize = ""
diskLib.dataCachePageSize = ""
diskLib.maxUnsyncedWrites = ""
scsi1:.deviceType = "disk"
scsi1:.deviceType = "disk"
scsi1:.deviceType = "disk"
scsi1:.deviceType = "disk"
disk.EnableUUID ="true"
启动rac1和rac2两个虚拟机
(4)修改二节点中的与一节点中不同的配置
root用户登录系统，修改网络参数
- hostname: testdb11b
- IP Address eth0: 192.168.1.21 (public address)
- Default Gateway eth0: 192.168.1.1 (public address)
- IP Address eth1: 192.168.2.21 (private address)
- Default Gateway eth1: none
修改主机名
[root@testdb11a ~]# vi /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=testdb11b
修改IP地址（ifcfg-eth0和ifcfg-eth1两个文件）
[root@testdb11a ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth0
BOOTPROTO=static
IPADDR=192.168.1.21
NETMASK=255.255.255.0
GATAWAY=192.168.1.1
ONBOOT=yes
HWADDR=:0c::b6::
[root@testdb11a ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
# Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth1
BOOTPROTO=static
IPADDR=192.168.2.21
NETMASK=255.255.255.0
ONBOOT=yes
HWADDR=:0c::b6::5e
修改环境变量
grid用户
.bash_profile 中的 export ORACLE_SID=+ASM2
oracle用户
.bash_profile文件的export ORACLE_SID=db112
在rac2上查看共享磁盘
[root@testdb11a raw]# raw -qa
/dev/raw/raw1: bound to major , minor
/dev/raw/raw2: bound to major , minor
/dev/raw/raw3: bound to major , minor
/dev/raw/raw4: bound to major , minor
[root@testdb11a raw]# cd /dev/raw
[root@testdb11a raw]# ll
total
crw-rw---- grid asmdba , Sep : raw1
crw-rw---- grid asmdba , Sep : raw2
crw-rw---- grid asmdba , Sep : raw3
crw-rw---- grid asmdba , Sep : raw4
重启二节点虚拟机
执行
ping -c testdb11a
ping -c testdb11b
ping -c testdb11a-priv
ping -c testdb11b-priv
保证两台虚拟机之间网络互通
在安装之前，软件解压包的目录下有一个节点间校验系统要求的工具，可以使用下面的命令进行校验预安装要求，但是使用这个工具必须配置ssh互信，所有这里就不做校验
./runcluvfy.sh stage -pre crsinst -n testdb11a,testdb11b -verbose
三，安装grid Infrastructure
(1)通过ftp或者其他方式上传软件，并解压
创建一个放置oracle软件的目录
mkdir -p /oracle/media
chown -R oracle:oinstall /oracle/media
chmod -R /oracle/media
[root@testdb11a media]# ll
total
-rw-r--r-- oracle oinstall Nov p10404530_112030_Linux-x86-64_1of7.zip
-rw-r--r-- oracle oinstall Nov p10404530_112030_Linux-x86-64_2of7.zip ----上面两个为数据库软件
-rw-r--r-- grid oinstall Sep : p10404530_112030_Linux-x86-64_3of7.zip ----grid 软件
[root@testdb11a media]#
解压
[root@testdb11a app]# cd /oracle/media/
[root@testdb11a media]# ll
total
drwxr-xr-x oracle oinstall Sep database
drwxr-xr-x grid oinstall Sep grid
[root@testdb11a media]#
在/oracle/media/grid/rpm这个目录下有个cvuqdisk-1.0.9-1.rpm包，在两个节点上安装
scp cvuqdisk-1.0.-.rpm testdb11b:/home/grid
[root@testdb11b grid]# rpm -ivh cvuqdisk-1.0.-.rpm
Preparing... ########################################### [%]
Using default group oinstall to install package
:cvuqdisk ########################################### [%]
(2)启动图形界面
执行./runInstaller
跳过软件更新
选择“install and Configure Grid Infrastructure for a Cluster ”
选择“Typical Installation”
输入正确的SCAN名，点击Add把另一个节点加入
填入二节点的正确的Hostname和vip name
点击“SSH Connectivity”，输入grid的密码，点击“Setup”配置互信（备注：下面的这幅截图SCAN Name写错了，应该是testdb11-scan，和hosts文件中的相同）
出现上面的提示，则互信配置成功，下面点击“Test”进行测试
点击“Identify network interfaces..”检查网络配置是否正确，确认无误后，点击“OK”
输入grid的安装目录，选择“Oracle Automatic Storage Management ”,输入ASM的密码，下一步
使用简单密码，继续
选择“External”，选择所有的磁盘，如果磁盘不显示，点击“Change Discovery Path”，输入“/dev/raw/raw*”
默认
下面的这几个警告可以忽略，勾选"Ignore All"，继续（说明，下图中第三个警告不能跳过）
安装
提示需要执行两个脚本
执行结果如下
节点一，节点二
在这里二节点执行root.sh的时候报错
CRS-: IP Address: 192.168.1.24 is already in use in the network
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/testdb11b/agent/crsd/orarootagent_root/orarootagent_root.log".
出现上面的错误的原因是：我没有配置dns服务，只在hosts文件中加入了相应的scan-ip，希望能骗过oracle，但是这种做法在11.2.0.2开始这种做法已经不可取，所以只能配置dns去解决。下面的安装软件过程中检测出来的错误，在11.2.0.2之前是不检查/etc/resolv.conf这个文件的(两个节点都需要配置)
重新执行脚本，
显示成功后，点击OK继续安装
查看日志
INFO: Checking Single Client Access Name (SCAN)...
INFO: ERROR:
INFO: PRVF- : Scan Listener "LISTENER_SCAN1" not running
INFO: Checking name resolution setup for "testdb11-scan"...
INFO: ERROR:
INFO: PRVF- : Name resolution setup check for "testdb11-scan" (IP address: 192.168.1.25) failed
INFO: ERROR:
INFO: PRVF- : Name resolution setup check for "testdb11-scan" (IP address: 192.168.1.26) failed
INFO: ERROR:
INFO: PRVF- : Found inconsistent name resolution entries for SCAN name "testdb11-scan"
INFO: Verification of SCAN VIP and Listener setup failed
可以放心跳过
四，安装数据库软件
进入oracle用户，database目录，执行
./runInstaller
点击下一步
点击“SSH Connectivity”配置oracle用户的节点间互信。然后点击“Setup”，并“Test”
选择语言，然后下一步
选择企业版
选择安装路径
选择管理组
先决条件检查，选择忽略
我分配的内存为1GB，内存绝对不能小于1GB推荐最小1.5g。卡在56%半天才过去
执行脚本
执行完成后，点击OK，database soft安装完成
五，使用DBCA进行建库
出现上面的结果，表明顺利安装完成！
验证：
查看两个节点的服务状态：
[root@testdb11a db11]# /u01/app/11.2./grid/bin/crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA.dg ora....up.type / / ONLINE ONLINE testdb11a
ora....ER.lsnr ora....er.type / / ONLINE ONLINE testdb11a
ora....N1.lsnr ora....er.type / / ONLINE ONLINE testdb11a
ora.asm ora.asm.type / / ONLINE ONLINE testdb11a
ora.cvu ora.cvu.type / / ONLINE ONLINE testdb11a
ora.db11.db ora....se.type / / ONLINE ONLINE testdb11a
ora.gsd ora.gsd.type / / OFFLINE OFFLINE
ora....network ora....rk.type / / ONLINE ONLINE testdb11a
ora.oc4j ora.oc4j.type / / ONLINE ONLINE testdb11a
ora.ons ora.ons.type / / ONLINE ONLINE testdb11a
ora....ry.acfs ora....fs.type / / ONLINE ONLINE testdb11a
ora.scan1.vip ora....ip.type / / ONLINE ONLINE testdb11a
ora....SM1.asm application / / ONLINE ONLINE testdb11a
ora....1A.lsnr application / / ONLINE ONLINE testdb11a
ora....11a.gsd application / / OFFLINE OFFLINE
ora....11a.ons application / / ONLINE ONLINE testdb11a
ora....11a.vip ora....t1.type / / ONLINE ONLINE testdb11a
ora....SM2.asm application / / ONLINE ONLINE testdb11b
ora....1B.lsnr application / / ONLINE ONLINE testdb11b
ora....11b.gsd application / / OFFLINE OFFLINE
ora....11b.ons application / / ONLINE ONLINE testdb11b
ora....11b.vip ora....t1.type / / ONLINE ONLINE testdb11b
说明：gsd是在9i rac中用于管理客户端的服务，11g默认的关闭的。
查看数据库状态
db112@testdb11b /home/oracle$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.3. Production on Sun Sep :: Copyright (c) , , Oracle. All rights reserved. Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3. - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options SQL> select instance_name,status from gv$instance; INSTANCE_NAME STATUS
---------------- ------------
db111 OPEN
db112 OPEN
SQL> select * from v$active_instances;
INST_NUMBER INST_NAME
----------- ------------------------------------------------------------
1 testdb11a:db111
2 testdb11b:db112
db111@testdb11a /home/oracle$ srvctl config database -d db11
Database unique name: db11
Database name: db11
Oracle home: /u01/app/oracle/product/11.2.
Oracle user: oracle
Spfile: +DATA/db11/spfiledb11.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: db11
Database instances: db111,db112
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Database is administrator managed
db111@testdb11a /home/oracle$ srvctl status database -d db11
Instance db111 is running on node testdb11a
Instance db112 is running on node testdb11b
六，遇到的问题
问题1：二节点执行root.sh脚本报错
问题2：只能看到一个节点
解决：oracle 11g RAC安装节点二执行结果错误CRS-5005: IP Address: 192.168.1.24 is already in use in the network