Exadata是Oracle软硬结合的一体机，通过自己的机子不可能完全模拟出来，这里搭建的虚拟机只是一个简单的、可以供我们学习Exadata的环境。
最简单可以模拟的Exadata虚拟机需要两个节点，CELL节点和DB节点。
首先说一下CELL节点安装介质：
OEL5.6-x86-64
cell-11.2.3.3.1_LINUX.X64
这两个在https://edelivery.oracle.com/ 可以下载的到
下面是具体安装步骤：
关闭UEK，否则启动cellsrv会报错：
[root@CELL ~]# vi /etc/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
# initrd /initrd-version.img
#boot=/dev/sda
default=1 原本为0 改为1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
。。。。略。。。
重启系统：
[root@CELL ~]# reboot
hosts文件配置如下：
[root@CELL ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
192.168.90.151 CELL
网卡配置如下：
[root@CELL ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0# Intel Corporation 82545EM Gigabit Ethernet Controller (Copper)
DEVICE=eth0
BOOTPROTO=static
BROADCAST=192.168.90.255
HWADDR=00:0C:29:3D:9A:03
IPADDR=192.168.90.151
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.0
NETWORK=192.168.90.0
ONBOOT=yes
使用oracle-validated安装一些rpm包
[root@CELL ~]# vi /etc/yum.repos.d/public-yum-el5.repo
[oel5]
name = Enterprise Linux 5.6 DVD
baseurl=file:///mnt/Server/
gpgcheck=0
enabled=1
[root@CELL ~]# mount /dev/cdrom /mnt
mount: block device /dev/cdrom is write-protected, mounting read-only
[root@CELL ~]# yum install oracle-validated
Loaded plugins: rhnplugin, security
This system is not registered with ULN.
ULN support will be disabled.
oel5 | 1.1 kB 00:00
oel5/primary | 1.4 MB 00:00
oel5 3247/3247
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package oracle-validated.x86_64 0:1.1.0-7.el5 set to be updated
oel5/filelists | 3.1 MB 00:00
--> Processing Dependency: /usr/lib/libaio.so for package: oracle-validated
--> Processing Dependency: unixODBC-devel for package: oracle-validated
--> Processing Dependency: sysstat for package: oracle-validated
--> Processing Dependency: /usr/lib64/libodbccr.so for package: oracle-validated
--> Processing Dependency: libaio-devel for package: oracle-validated
--> Processing Dependency: compat-gcc-34-c++ for package: oracle-validated
--> Processing Dependency: /usr/lib/libodbc.so.1 for package: oracle-validated
--> Processing Dependency: compat-gcc-34 for package: oracle-validated
--> Processing Dependency: /usr/lib64/libaio.so for package: oracle-validated
--> Processing Dependency: /usr/lib/libodbccr.so for package: oracle-validated
--> Processing Dependency: libXp.so.6 for package: oracle-validated
--> Processing Dependency: libdb-4.2.so()(64bit) for package: oracle-validated
--> Processing Dependency: libodbc.so.1()(64bit) for package: oracle-validated
--> Running transaction check
---> Package compat-db.x86_64 0:4.2.52-5.1 set to be updated
---> Package compat-gcc-34.x86_64 0:3.4.6-4.1 set to be updated
---> Package compat-gcc-34-c++.x86_64 0:3.4.6-4.1 set to be updated
---> Package libXp.i386 0:1.0.0-8.1.el5 set to be updated
---> Package libaio-devel.i386 0:0.3.106-5 set to be updated
---> Package libaio-devel.x86_64 0:0.3.106-5 set to be updated
---> Package sysstat.x86_64 0:7.0.2-3.el5_5.1 set to be updated
---> Package unixODBC.i386 0:2.2.11-7.1 set to be updated
---> Package unixODBC.x86_64 0:2.2.11-7.1 set to be updated
---> Package unixODBC-devel.i386 0:2.2.11-7.1 set to be updated
---> Package unixODBC-devel.x86_64 0:2.2.11-7.1 set to be updated
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
oracle-validated x86_64 1.1.0-7.el5 oel5 23 k
Installing for dependencies:
compat-db x86_64 4.2.52-5.1 oel5 1.6 M
compat-gcc-34 x86_64 3.4.6-4.1 oel5 4.3 M
compat-gcc-34-c++ x86_64 3.4.6-4.1 oel5 13 M
libXp i386 1.0.0-8.1.el5 oel5 22 k
libaio-devel i386 0.3.106-5 oel5 12 k
libaio-devel x86_64 0.3.106-5 oel5 11 k
sysstat x86_64 7.0.2-3.el5_5.1 oel5 175 k
unixODBC i386 2.2.11-7.1 oel5 830 k
unixODBC x86_64 2.2.11-7.1 oel5 835 k
unixODBC-devel i386 2.2.11-7.1 oel5 743 k
unixODBC-devel x86_64 2.2.11-7.1 oel5 799 k
Transaction Summary
================================================================================
Install 12 Package(s)
Upgrade 0 Package(s)
Total download size: 22 M
Is this ok [y/N]: y
....略....
建立目录：
[root@CELL ~]# mkdir -p /var/log/oracle
[root@CELL ~]# chmod -R 775 /var/log/oracle
上传介质后解压：
[root@CELL soft]# tar -pxvf cellImageMaker_11.2.3.3.1_LINUX.X64_140708-1.x86_64.tar
。。。略。。。
解压后得到dl180文件夹
[root@CELL soft]# cd dl180/boot/cellbits/
[root@CELL cellbits]# unzip cell.bin
Archive: cell.bin
warning [cell.bin]: 20038 extra bytes at beginning or within zipfile
(attempting to process anyway)
inflating: cell-11.2.3.3.1_LINUX.X64_140708-1.x86_64.rpm
解压得到cell-11.2.3.3.1_LINUX.X64_140708-1.x86_64.rpm，进行安装
[root@CELL cellbits]# rpm -ivh cell-11.2.3.3.1_LINUX.X64_140708-1.x86_64.rpm
error: Failed dependencies:
jdk >= 1.5.0_12 is needed by cell-11.2.3.3.1_LINUX.X64_140708-1.x86_64
安装包报错，提示需要JDK，JDK我们可以从同目录下cellrpms.tbz中找到
[root@CELL cellbits]# tar -tf cellrpms.tbz
cellrpmlist
exadata-firmware-cell-11.2.3.3.1.140708-1.x86_64.rpm
exadata-validations-cell-11.2.3.3.1.140708-1.x86_64.rpm
extradepsrpmlist
image_functions
imagerpmlist
install.sh
jdk-1.7.0_55-fcs.x86_64.rpm
python-numeric-23.7-2.2.2.el5_6.1.x86_64.rpm
scsi-target-utils-1.0.14-2.el5.x86_64.rpm
selinux-policy-2.4.6-346.el5.noarch.rpm
selinux-policy-targeted-2.4.6-346.el5.noarch.rpm
将jdk解压出来并安装
[root@CELL cellbits]# tar -pxvf cellrpms.tbz jdk-1.7.0_55-fcs.x86_64.rpm
jdk-1.7.0_55-fcs.x86_64.rpm
[root@CELL cellbits]# rpm -ivh jdk-1.7.0_55-fcs.x86_64.rpm
warning: jdk-1.7.0_55-fcs.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing... ########################################### [100%]
1:jdk ########################################### [100%]
Unpacking JAR files...
rt.jar...
jsse.jar...
charsets.jar...
tools.jar...
localedata.jar...
jfxrt.jar...
安装cell
[root@CELL cellbits]# rpm -ivh cell-11.2.3.3.1_LINUX.X64_140708-1.x86_64.rpm
Preparing... ########################################### [100%]
Pre Installation steps in progress ...
1:cell ########################################### [100%]
Post Installation steps in progress ...
Set cellusers group for /opt/oracle/cell11.2.3.3.1_LINUX.X64_140708/cellsrv/deploy/log directory
Set 775 permissions for /opt/oracle/cell11.2.3.3.1_LINUX.X64_140708/cellsrv/deploy/log directory
/
/
Installation SUCCESSFUL.
Starting RS and MS... as user celladmin
Done. Please Login as user celladmin and create cell to startup CELLSRV to complete cell configuration.
If this is a manual installation, please stop and restart ExaWatcher to pick up newly installed binaries.
You can run "/opt/oracle.ExaWatcher/ExaWatcher.sh --stop" and then "/opt/oracle.ExaWatcher/ExaWatcher.sh --fromconf" to stop and restart ExaWatcher.
Logout and then re-login to use the new cell environment.
[root@CELL ~]# mkdir -p /opt/oracle/cell/disks/raw
[root@CELL ~]# cd /opt/oracle/cell/disks/raw
[root@CELL raw]# vi dd.sh
[root@CELL raw]# cat dd.sh
dd if=/dev/zero of=disk01 bs=1M count=1000
dd if=/dev/zero of=disk02 bs=1M count=1000
dd if=/dev/zero of=disk03 bs=1M count=1000
dd if=/dev/zero of=disk04 bs=1M count=1000
dd if=/dev/zero of=disk05 bs=1M count=1000
dd if=/dev/zero of=disk06 bs=1M count=1000
dd if=/dev/zero of=disk07 bs=1M count=1000
dd if=/dev/zero of=disk08 bs=1M count=1000
dd if=/dev/zero of=disk09 bs=1M count=1000
dd if=/dev/zero of=disk10 bs=1M count=1000
dd if=/dev/zero of=disk11 bs=1M count=1000
dd if=/dev/zero of=disk12 bs=1M count=1000
dd if=/dev/zero of=FLASH01 bs=1M count=1000
dd if=/dev/zero of=FLASH02 bs=1M count=1000
dd if=/dev/zero of=FLASH03 bs=1M count=1000
dd if=/dev/zero of=FLASH04 bs=1M count=1000
执行脚本：
[root@CELL raw]# sh dd.sh
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 36.0089 seconds, 29.1 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 45.0823 seconds, 23.3 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 38.5387 seconds, 27.2 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 66.3054 seconds, 15.8 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 37.9832 seconds, 27.6 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 33.8476 seconds, 31.0 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 57.1281 seconds, 18.4 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 34.5081 seconds, 30.4 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 52.1244 seconds, 20.1 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 58.0057 seconds, 18.1 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 42.3233 seconds, 24.8 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 50.184 seconds, 20.9 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 41.2316 seconds, 25.4 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 38.1167 seconds, 27.5 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 39.3045 seconds, 26.7 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 34.2013 seconds, 30.7 MB/s
进行赋权：
[root@CELL raw]# chmod 660 *
[root@CELL raw]# ls -ltr
total 16400068
-rw-rw---- 1 root root 692 Sep 28 12:29 dd.sh
-rw-rw---- 1 root root 1048576000 Sep 28 12:30 disk01
-rw-rw---- 1 root root 1048576000 Sep 28 12:31 disk02
-rw-rw---- 1 root root 1048576000 Sep 28 12:32 disk03
-rw-rw---- 1 root root 1048576000 Sep 28 12:33 disk04
-rw-rw---- 1 root root 1048576000 Sep 28 12:34 disk05
-rw-rw---- 1 root root 1048576000 Sep 28 12:34 disk06
-rw-rw---- 1 root root 1048576000 Sep 28 12:35 disk07
-rw-rw---- 1 root root 1048576000 Sep 28 12:36 disk08
-rw-rw---- 1 root root 1048576000 Sep 28 12:37 disk09
-rw-rw---- 1 root root 1048576000 Sep 28 12:38 disk10
-rw-rw---- 1 root root 1048576000 Sep 28 12:38 disk11
-rw-rw---- 1 root root 1048576000 Sep 28 12:39 disk12
-rw-rw---- 1 root root 1048576000 Sep 28 12:40 FLASH01
-rw-rw---- 1 root root 1048576000 Sep 28 12:41 FLASH02
-rw-rw---- 1 root root 1048576000 Sep 28 12:41 FLASH03
-rw-rw---- 1 root root 1048576000 Sep 28 12:42 FLASH04
删掉dd.sh
[root@CELL raw]# rm dd.sh
rm: remove regular file `dd.sh‘? yes
切换到celladmin下重启cell服务
[celladmin@CELL ~]$ cellcli -e alter cell restart services all
Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services... running
Starting CELLSRV services...
The STARTUP of CELLSRV services was not successful.
CELL-01553: Incorrect IP in cellinit.ora. IP is invalid or has incorrect/missing netmask. Please refer to RS incident log.
Starting MS services...
The STARTUP of MS services was successful.
报错说没有ip配置
[celladmin@CELL ~]$ cellcli -e create cell cell1 interconnect1=eth0
Cell cell1 successfully created
Starting CELLSRV services...
The STARTUP of CELLSRV services was not successful.
CELL-02018: Internal error detected: Not enough open file descriptors.
ip配置好后依然报错
alert日志位置：
/opt/oracle/cell11.2.3.3.1_LINUX.X64_140708/log/diag/asm/cell/CELL/trace/alert.log
从中可以看到以下内容：
Number of open file descriptors per process is 1024
The minimum number is 16384. Please change ulimit
CELLSRV boot failed to start due to the error 11 (Not enough open file descriptors)
接下来我们调整限制：
[root@CELL ~]# vi /etc/sysctl.conf
最后一行添加：
fs.file-max = 65536
生效一下：
[root@CELL ~]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
fs.file-max = 65536
继续修改/etc/security/limits.conf文件：
[root@CELL ~]# vi /etc/security/limits.conf
最后添加两行：
* soft nofile 65536
* hard nofile 65536
修改完后用celladmin用户重新登录，查看限制：
[celladmin@CELL ~]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16384
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16384
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
重启cell服务：
[celladmin@CELL ~]$ cellcli -e alter cell restart services all
Stopping the RS, CELLSRV, and MS services...
The SHUTDOWN of services was successful.
Starting the RS, CELLSRV, and MS services...
Getting the state of RS services... running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
The STARTUP of MS services was successful.
[celladmin@CELL ~]$ cellcli
CellCLI: Release 11.2.3.3.1 - Production on Mon Sep 29 10:04:46 CST 2014
Copyright (c) 2007, 2013, Oracle. All rights reserved.
Cell Efficiency Ratio: 1
CellCLI> create celldisk all
CellDisk CD_disk01_cell1 successfully created
CellDisk CD_disk02_cell1 successfully created
CellDisk CD_disk03_cell1 successfully created
CellDisk CD_disk04_cell1 successfully created
CellDisk CD_disk05_cell1 successfully created
CellDisk CD_disk06_cell1 successfully created
CellDisk CD_disk07_cell1 successfully created
CellDisk CD_disk08_cell1 successfully created
CellDisk CD_disk09_cell1 successfully created
CellDisk CD_disk10_cell1 successfully created
CellDisk CD_disk11_cell1 successfully created
CellDisk CD_disk12_cell1 successfully created
CellDisk FD_00_cell1 successfully created
CellDisk FD_01_cell1 successfully created
CellDisk FD_02_cell1 successfully created
CellDisk FD_03_cell1 successfully created
CellCLI> create flashcache all size=2G
Flash cache cell1_FLASHCACHE successfully created
CellCLI> create flashlog all
Flash log cell1_FLASHLOG successfully created
CellCLI> list flashcache detail
name: cell1_FLASHCACHE
cellDisk: FD_00_cell1,FD_02_cell1,FD_03_cell1,FD_01_cell1
creationTime: 2014-09-29T10:27:46+08:00
degradedCelldisks:
effectiveCacheSize: 2G
id: 5a6f81d4-c9a1-4494-adf0-cdea2a3243ea
size: 2G
status: normal
CellCLI> list flashlog detail
name: cell1_FLASHLOG
cellDisk: FD_03_cell1,FD_00_cell1,FD_02_cell1,FD_01_cell1
creationTime: 2014-09-29T10:28:14+08:00
degradedCelldisks:
effectiveSize: 512M
efficiency: 100.0
id: 4acc8873-d311-4e90-959b-5e8850e9e9ad
size: 512M
status: normal
CellCLI> list cell
cell1 online
CellCLI> list celldisk
CD_disk01_cell1 normal
CD_disk02_cell1 normal
CD_disk03_cell1 normal
CD_disk04_cell1 normal
CD_disk05_cell1 normal
CD_disk06_cell1 normal
CD_disk07_cell1 normal
CD_disk08_cell1 normal
CD_disk09_cell1 normal
CD_disk10_cell1 normal
CD_disk11_cell1 normal
CD_disk12_cell1 normal
FD_00_cell1 normal
FD_01_cell1 normal
FD_02_cell1 normal
FD_03_cell1 normal
CellCLI> create griddisk all harddisk prefix=data, size=300M
GridDisk data_CD_disk01_cell1 successfully created
GridDisk data_CD_disk02_cell1 successfully created
GridDisk data_CD_disk03_cell1 successfully created
GridDisk data_CD_disk04_cell1 successfully created
GridDisk data_CD_disk05_cell1 successfully created
GridDisk data_CD_disk06_cell1 successfully created
GridDisk data_CD_disk07_cell1 successfully created
GridDisk data_CD_disk08_cell1 successfully created
GridDisk data_CD_disk09_cell1 successfully created
GridDisk data_CD_disk10_cell1 successfully created
GridDisk data_CD_disk11_cell1 successfully created
GridDisk data_CD_disk12_cell1 successfully created
CellCLI> create griddisk all harddisk prefix=reco, size=160M
GridDisk reco_CD_disk01_cell1 successfully created
GridDisk reco_CD_disk02_cell1 successfully created
GridDisk reco_CD_disk03_cell1 successfully created
GridDisk reco_CD_disk04_cell1 successfully created
GridDisk reco_CD_disk05_cell1 successfully created
GridDisk reco_CD_disk06_cell1 successfully created
GridDisk reco_CD_disk07_cell1 successfully created
GridDisk reco_CD_disk08_cell1 successfully created
GridDisk reco_CD_disk09_cell1 successfully created
GridDisk reco_CD_disk10_cell1 successfully created
GridDisk reco_CD_disk11_cell1 successfully created
GridDisk reco_CD_disk12_cell1 successfully created
CellCLI> list griddisk
data_CD_disk01_cell1 active
data_CD_disk02_cell1 active
data_CD_disk03_cell1 active
data_CD_disk04_cell1 active
data_CD_disk05_cell1 active
data_CD_disk06_cell1 active
data_CD_disk07_cell1 active
data_CD_disk08_cell1 active
data_CD_disk09_cell1 active
data_CD_disk10_cell1 active
data_CD_disk11_cell1 active
data_CD_disk12_cell1 active
reco_CD_disk01_cell1 active
reco_CD_disk02_cell1 active
reco_CD_disk03_cell1 active
reco_CD_disk04_cell1 active
reco_CD_disk05_cell1 active
reco_CD_disk06_cell1 active
reco_CD_disk07_cell1 active
reco_CD_disk08_cell1 active
reco_CD_disk09_cell1 active
reco_CD_disk10_cell1 active
reco_CD_disk11_cell1 active
reco_CD_disk12_cell1 active
CellCLI> list celldisk FD_00_cell1 detail
name: FD_00_cell1
comment:
creationTime: 2014-09-29T10:26:28+08:00
deviceName: /opt/oracle/cell11.2.3.3.1_LINUX.X64_140708/disks/raw/FLASH01
devicePartition: /opt/oracle/cell11.2.3.3.1_LINUX.X64_140708/disks/raw/FLASH01
diskType: FlashDisk
errorCount: 0
freeSpace: 336M
freeSpaceMap: offset=688M,size=336M
id: 09b329a7-2099-4b5a-8208-933546f0e23c
interleaving: none
lun: /opt/oracle/cell11.2.3.3.1_LINUX.X64_140708/disks/raw/FLASH01
physicalDisk: /opt/oracle/cell11.2.3.3.1_LINUX.X64_140708/disks/raw/FLASH01
raidLevel: "RAID 0"
size: 1G
status: normal
CellCLI> list celldisk CD_disk01_cell1 detail
name: CD_disk01_cell1
comment:
creationTime: 2014-09-29T10:05:19+08:00
deviceName: /opt/oracle/cell11.2.3.3.1_LINUX.X64_140708/disks/raw/disk01
devicePartition: /opt/oracle/cell11.2.3.3.1_LINUX.X64_140708/disks/raw/disk01
diskType: HardDisk
errorCount: 0
freeSpace: 528M
freeSpaceMap: offset=496M,size=528M
id: 43697fde-2a47-4f84-a496-de3b516ecaca
interleaving: none
lun: /opt/oracle/cell11.2.3.3.1_LINUX.X64_140708/disks/raw/disk01
physicalDisk: /opt/oracle/cell11.2.3.3.1_LINUX.X64_140708/disks/raw/disk01
raidLevel: "RAID 0"
size: 1G
status: normal
CellCLI> list griddisk data_CD_disk01_cell1 detail
name: data_CD_disk01_cell1
asmDiskgroupName:
asmDiskName:
asmFailGroupName:
availableTo:
cachingPolicy: default
cellDisk: CD_disk01_cell1
comment:
creationTime: 2014-09-29T10:30:56+08:00
diskType: HardDisk
errorCount: 0
id: d085144e-f71a-42b2-9db7-0b8f2bee95b4
offset: 48M
size: 288M
status: active
至此CELL节点ok