实验环境：
vm1-ning:172.16.3.1/16
vm2-hong:172.16.3.2/16
drbd操作步骤：
1、创建mysql的drbd资源磁盘(两节点都需要创建)
[root@hong ~]# fdisk /dev/sda
n
p
+5G
w
[root@hong ~]# kpartx -af /dev/sda
[root@hong ~]# partx -a /dev/sda
2、drbd设置（全局配置文件这里和web高可用中的一样没有改变，不再次说明。）
[root@hong ~]# cd /etc/drbd.d
[root@hong drbd.d]# ls
global_common.conf global_common.conf.bak global_common.conf.rpmsave web.res
[root@hong drbd.d]# cp web.res mydata.res
[root@hong drbd.d]# vim mydata.res 定义资源名称
[root@hong drbd.d]# cat mydata.res
resource mydata { 定义资源名称也是唯一的，不能相同
on ning {
device /dev/drbd1; 指定drdb设备名称为drbd1
disk /dev/sda4; 定义的分区磁盘
address 172.16.3.1:7790; 两个资源之间不能使用相同的端口号
meta-disk internal;
}
on hong {
device /dev/drbd1;
disk /dev/sda4;
address 172.16.3.2:7790;
meta-disk internal;
}
}
[root@hong drbd.d]# scp mydata.res 172.16.3.1:/etc/drbd.d/ 双方节点上都需要配置这样复制过去即可
root@172.16.3.1‘s password:
mydata.res
100% 278 0.3KB/s 00:00
3、初始化drbd在两节点上都需要初始化，并启动drbd
[root@hong drbd.d]# drbdadm create-md mydata
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
[root@ning ~]# drbdadm create-md mydata
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
[root@hong drbd.d]# service drbd start
[root@ning ~]# service drbd start
4、提升一个节点为主节点进行分区数据同步和格式化分区
[root@hong drbd.d]# drbd-overview （查看当前状态）
0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
1:mydata/0 Connected Secondary/Secondary Inconsistent/Inconsistent C r----- （分区还没有进行同步，数据不一致状态）
[root@hong drbd.d]# drbdadm primary --force mydata （提升hong节点为主节点，进行数据同步，并进行格式化）
[root@hong drbd.d]# drbd-overview
0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
1:mydata/0 SyncSource Primary/Secondary UpToDate/Inconsistent C r---n-
[>....................] sync‘ed: 4.3% (4912/5128)M （正在同步块按照位在对齐）
[root@hong drbd.d]# drbd-overview
0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
1:mydata/0 Connected Primary/Secondary UpToDate/UpToDate C r----- （分区同步完成）
[root@hong ~]# mke2fs -t ext4 /dev/drbd1 （对分区进行格式化）
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe blocks
328656 inodes, 1313263 blocks
65663 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1346371584
41 block groups
32768 blocks per group, 32768 fragments per group
8016 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 39 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
mysql配置：
1、创建mysql用户（注意两节点的mysql的ID号要一致）
hong节点：
[root@hong drbd.d]# groupadd -r -g 306 mysql
[root@hong drbd.d]# useradd -r -g 306 -u 306 mysql
[root@hong drbd.d]# id mysql
uid=306(mysql) gid=306(mysql) groups=306(mysql)
[root@hong ~]# mkdir /mydata 创建数据的存储目录
ning节点：
[root@ning ~]# groupadd -r -g 306 mysql
[root@ning ~]# useradd -r -g 306 -u 306 mysql
[root@ning ~]# id mysql
uid=306(mysql) gid=306(mysql) groups=306(mysql)
[root@ning ~]# mkdir /mydata 创建数据的存储目录（磁盘的挂载点）
2、安装mysql
（1）在主节点hong上挂载/dev/drbd1到数据的存储目录/mydata
[root@hong ~]# mount /dev/drbd1 /mydata/
（2）在挂载目录/mydata下创建数据存储目录为data，并设置属主组
[root@hong ~]# cd /mydata/
[root@hong mydata]# mkdir data
[root@hong mydata]# chown -R mysql.mysql data
[root@hong mydata]# ll
total 20
drwxr-xr-x 2 mysql mysql 4096 Aug 16 17:21 data
drwx------ 2 root root 16384 Aug 16 17:16 lost+found
验证下是否两节点都是有data目录的
[root@hong ~]# umount /mydata
[root@hong ~]# drbdadm secondary mydata 在hong上降级drbd为从节点
[root@hong ~]# drbd-overview
0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
1:mydata/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
[root@ning ~]# drbdadm primary mydata 在ning节点上提升为主节点
[root@ning ~]# drbd-overview
0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
1:mydata/0 Connected Primary/Secondary UpToDate/UpToDate C r-----
[root@ning ~]# mount /dev/drbd1 /mydata/
[root@ning ~]# cd /mydata/
[root@ning mydata]# ll
total 20
drwxr-xr-x 2 mysql mysql 4096 Aug 16 17:21 data （查看是否data目录为mysql属主组）
drwx------ 2 root root 16384 Aug 16 17:16 lost+found
（3）安装mysql软件
包： mariadb-5.5.36-linux-x86_64.tar.gz
ning节点上：
[root@ning ~]# tar xf mariadb-5.5.36-linux-x86_64.tar.gz -C /usr/local
[root@ning ~]# cd /usr/local
[root@ning local]# ln -sv mariadb-5.5.36-linux-x86_64/ mysql
`mysql‘ -> `mariadb-5.5.36-linux-x86_64/‘
[root@ning local]# chown -R root.mysql mysql/*
[root@ning local]# cd mysql
[root@ning mysql]# scripts/mysql_install_db --datadir=/mydata/data/ --user=mysql
[root@ning mysql]# mkdir /etc/mysql
[root@ning mysql]# cp support-files/my-large.cnf /etc/mysql/my.cnf
[root@ning mysql]# vim /etc/mysql/my.cnf
[mysqld]
datadir = /mydata/data
[root@ning mysql]# cp support-files/my-large.cnf /etc/mysql/my.cnf
[root@ning mysql]# vim /etc/mysql/my.cnf
[root@ning mysql]# vim /etc/mysql/my.cnf
[root@ning mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@ning mysql]# chmod +x /etc/rc.d/init.d/mysqld
[root@ning mysql]# chkconfig --add mysqld
[root@ning mysql]# chkconfig mysqld off
[root@ning mysql]# service mysqld start
Starting MySQL.... [ OK ]
[root@ning mysql]# ls /mydata/data
aria_log.00000001 ibdata1 ib_logfile1 mysql-bin.000001 ning.pid test
aria_log_control ib_logfile0 mysql mysql-bin.index performance_schema
[root@ning mysql]# /usr/local/mysql/bin/mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.36-MariaDB-log MariaDB Server
Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.
Type ‘help;‘ or ‘\h‘ for help. Type ‘\c‘ to clear the current input statement.
MariaDB [(none)]> create database mysdb;
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> \q
Bye
到这里mariadb数据库已经安装完成
[root@ning mysql]# service mysqld stop
Shutting down MySQL. [ OK ]
[root@ning mysql]# cd
[root@ning ~]# umount /dev/drbd1 卸载挂载点
[root@ning ~]# drbdadm secondary mydata 降级为从节点
[root@ning ~]# drbd-overview
0:web/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
1:mydata/0 Connected Secondary/Secondary UpToDate/UpToDate C r-----
[root@ning ~]# scp mariadb-5.5.36-linux-x86_64.tar.gz 172.16.3.2:/root 复制包到ning节点，进行安装
root@172.16.3.2‘s password:
mariadb-5.5.36-linux-x86_64.tar.gz 100% 212MB 26.5MB/s 00:08
hong节点：
[root@hong ~]# drbdadm primary mydata 提升hong节点为主节点
[root@hong ~]# mount /dev/drbd1 /mydata 挂载分区
[root@hong ~]# scp -r 172.16.3.1:/etc/mysql /etc/ 并复制ning节点上的mysqld配置文件，（因为配置文件是一样的）
root@172.16.3.1‘s password:
my.cnf 100% 4924 4.8KB/s 00:00
[root@hong ~]# tar xf mariadb-5.5.36-linux-x86_64.tar.gz -C /usr/local
[root@hong ~]# cd /usr/local
[root@hong local]# ln -sv mariadb-5.5.36-linux-x86_64/ mysql
`mysql‘ -> `mariadb-5.5.36-linux-x86_64/‘
[root@hong local]# cd mysql
[root@hong mysql]# chown -R root.mysql ./*
这里说明下，不需要对mariadb做初始化了，因为数据已经有了，在ning节点上做的。在这里只需要提供服务启动脚本即可
[root@hong mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@hong mysql]# chmod +x /etc/rc.d/init.d/mysqld
[root@hong mysql]# chkconfig --add mysqld
[root@hong mysql]# service mysqld start
Starting MySQL.... [ OK ]
[root@hong mysql]# /usr/local/mysql/bin/mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.36-MariaDB-log MariaDB Server
Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.
Type ‘help;‘ or ‘\h‘ for help. Type ‘\c‘ to clear the current input statement.
MariaDB [(none)]> show databases; 查看在ning节点上创建的数据库是存在的
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysdb | 看这里呵呵
| mysql |
| performance_schema |
| test |
+--------------------+
5 rows in set (0.05 sec)
MariaDB [(none)]> grant all on *.* to root@‘172.16.%.%‘ identified by ‘ning‘; 授权远程连接用户
Query OK, 0 rows affected (0.03 sec)
MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)
测试下是否能够连接上数据库：（随便找个节点上测试的（这节点上需要有mysql客户端））
[root@liang ~]# mysql -uroot -h 172.16.3.2 -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.5.36-MariaDB-log MariaDB Server
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type ‘help;‘ or ‘\h‘ for help. Type ‘\c‘ to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysdb |
| mysql | 这里可以看到创建的数据库
| performance_schema |
| test |
+--------------------+
5 rows in set (0.00 sec)
mysql> \q
Bye
（4）测试完成后，在两节点上关闭mysqld和drbd开始配置corosync
[root@hong mysql]# service mysqld stop
Shutting down MySQL. [ OK ]
[root@hong mysql]# umount /mydata
[root@hong mysql]# drbdadm secondary mydata
[root@hong mysql]# service drbd stop
Stopping all DRBD resources: .
[root@ning ~]# service drbd stop
Stopping all DRBD resources: .
配置corosync集群设置
1、定义drbd
[root@ning ~]# crm
crm(live)# configure
定义drbd资源
crm(live)configure# primitive mydrbd ocf:linbit:drbd params drbd_resource=mydata op monitor role=Master interval=10s timeout=20s op monitor role=Slave interval=20s timeout=30s op start timeout=240s op stop timeout=100s
注意下定义时间间隔的时候主从不能一样。
crm(live)configure# verify
定义drbd克隆资源
crm(live)configure# master ms_mydrbd mydrbd meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=True
crm(live)configure# verify
定义挂载磁盘资源
crm(live)configure# primitive myfs ocf:heartbeat:Filesystem params device=/dev/drbd1 directory=/mydata fstype=ext4 op monitor interval=20s timeout=60s op start timeout=60s op stop timeout=60s
crm(live)configure# verify
定义mysqlIP资源
crm(live)configure# primitive myip ocf:heartbeat:IPaddr params ip=172.16.3.101 op monitor interval=30s timeout=20s
crm(live)configure# verify
定义mysql服务启动资源
crm(live)configure# primitive myserver lsb:mysqld op monitor interval=30s timeout=20s
crm(live)configure# verify
定义规则
crm(live)configure# group myservice myip myfs myserver 定义组名称为myservice资源名称为myip\myfs\myserver
crm(live)configure# colocation myfs_with_ms_mydrbd_master inf: myfs ms_mydrbd:Master 定义myfs挂载磁盘资源一定和ms_mydrbd克隆资源在一起
crm(live)configure# verify
crm(live)configure# order myfs_after_ms_mydrbd_master mandatory: ms_mydrbd:promote myfs:start 定义启动顺序挂载磁盘my_fs一定在克隆资源（成为主节点）ms_mydrbd之后启动
crm(live)configure# colocation myserver_with_myfs inf: myserver myfs 定义mysql服务器启动一定要和挂载磁盘my_fs在一起
crm(live)configure# verify
crm(live)configure# commit
crm(live)# status (查看状态信息)
Last updated: Fri Sep 19 13:16:37 2014
Last change: Fri Sep 19 12:26:20 2014 via cibadmin on hong
Stack: classic openais (with plugin)
Current DC: ning - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
5 Resources configured
Online: [ hong ning ] （两个节点都在线）
Master/Slave Set: ms_mydrbd [mydrbd]
Masters: [ ning ]
Slaves: [ hong ]
Resource Group: myservice
myip (ocf::heartbeat:IPaddr): Started ning
myfs (ocf::heartbeat:Filesystem): Started ning
myserver (lsb:mysqld): Started ning
测试corosync+mariadb+drbd
1、测试是否ning节点下线，资源将被切换到hong节点上去
[root@ning ~]# crm node standby 让ning节点下线
[root@ning ~]# crm status
Last updated: Fri Sep 19 13:24:33 2014
Last change: Fri Sep 19 13:24:25 2014 via crm_attribute on ning
Stack: classic openais (with plugin)
Current DC: ning - partition with quorum
Version: 1.1.10-14.el6-368c726
2 Nodes configured, 2 expected votes
5 Resources configured
Node ning: standby
Online: [ hong ]
Master/Slave Set: ms_mydrbd [mydrbd]
Masters: [ hong ] drbd将自动切换到hong节点上
Stopped: [ ning ]
Resource Group: myservice
myip (ocf::heartbeat:IPaddr): Started hong
myfs (ocf::heartbeat:Filesystem): Started hong
myserver (lsb:mysqld): Started hong
在hong 节点查看下mysqlIP；
[root@hong ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:d7:f7:9c brd ff:ff:ff:ff:ff:ff
inet 172.16.3.2/16 brd 172.16.255.255 scope global eth0
inet 172.16.3.101/16 brd 172.16.255.255 scope global secondary eth0 这里是有mysqlIP
inet6 fe80::20c:29ff:fed7:f79c/64 scope link
valid_lft forever preferred_lft forever
在liang节点上尝试连接下mariadb
[root@liang ~]# mysql -uroot -h 172.16.3.101 -p 这里连接的mysqlIP的地址
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.5.36-MariaDB-log MariaDB Server
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type ‘help;‘ or ‘\h‘ for help. Type ‘\c‘ to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysdb |这里有我们在上面测试用到的mysdb
| mysql |
| performance_schema |
| test |
+--------------------+
5 rows in set (0.10 sec)
2、测试mysqld手动当掉，是否会自动启动？
# service mysqld stop
# ss -tnl 观察3306端口将会看到30s后会自动上线