在上一章我们学习了mfs文件系统，那么我们在这一章将使用pacemaker,pcs,corosync,iscsi,fence来实现高可用，集群管理等

实验环境：我们在上一章的基础上增加一个高可用节点server4作为master

1.server4安装master，编辑域名解析，开启服务

2.server1和server2配置高可用yum源

3.server1,server4上安装相关的组件

[root@server1 3.0.103]# yum install -y pacemaker corosync pcs
[root@server4 ~]# yum install -y pacemaker corosync pcs

4.在server1和server4上作免密登陆

[root@server1 3.0.103]# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
f0:2c:47:08:5b:39:01:6f:09:48:8a:47:9d:43:f5:9f root@server1
The key's randomart image is:
+--[ RSA 2048]----+
| .o+=++o         |
|.o. +*o+         |
|o . ..*.o        |
| .   . = . .     |
|      . S E      |
|       o         |
|                 |
|                 |
|                 |
+-----------------+
[root@server1 3.0.103]# ssh-copy-id server1
[root@server1 3.0.103]# ssh-copy-id server4

5.开启pcs（资源管理工具），并设置为开机自启动

[root@server1 3.0.103]# systemctl start pcsd.service 
[root@server1 3.0.103]# systemctl enable pcsd.service
[root@server4 ~]# systemctl start pcsd.service 
[root@server4 ~]# systemctl enable pcsd.service

6.用户hacluster(集群管理用户)设置密码

[root@server1 3.0.103]# id hacluster
[root@server4 ~]# passwd hacluster

7.创建集群，并启动


8.查看集群状态

9.我们在上面查看集群状态时可以看到一个警告warning，那么我们来解决他

[root@server1 3.0.103]# crm_verify -L -V		检测
   error: unpack_resources:	Resource start-up disabled since no STONITH resources have been defined
   error: unpack_resources:	Either configure some or disable STONITH with the stonith-enabled option
   error: unpack_resources:	NOTE: Clusters with shared data need STONITH to ensure data integrity
Errors found during check: config not valid
[root@server1 3.0.103]# pcs property set stonith-enabled=false
[root@server1 3.0.103]# crm_verify -L -V
[root@server1 3.0.103]# 

10.为集群创建资源

[root@server1 3.0.103]# pcs resource create vip ocf:heartbeat:IPaddr2 ip=172.25.66.100 cidr_netmask=32 op monitor interval=30s  添加一个vip作高可用
[root@server1 3.0.103]# pcs status

在server4上查看监控crm_mon

11.高可用测试
首先查看server1上的虚拟ip

然后我们在集群中停掉server1

看server4的监控

可以发现vip在server4上

在重新开启server1，vip不会漂移

数据共享

1.一开始，先卸载之前的回收站机制，关闭server1，2，3的mfs

[root@foundation66 addons]# umount /mnt/mfsmeta/
[root@server1 3.0.103]# systemctl stop moosefs-master.service
[root@server2 mfs]# systemctl stop moosefs-chunkserver.service 
[root@server3 ~]# systemctl stop moosefs-chunkserver.service 
[root@server4 ~]# systemctl stop moosefs-master.service

2.所有节点添加解析（物理机，server1-4）


3.给server2添加一块磁盘

4.安装targetcli，将server2的新加磁盘共享出去
server2:

[root@server2 mfs]# yum install -y targetcli
[root@server2 mfs]# systemctl start target
[root@server2 mfs]# targetcli 
Warning: Could not load preferences file /root/.targetcli/prefs.bin.
targetcli shell version 2.1.fb41
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/> ls
o- / .................................................................................................... [...]
  o- backstores ......................................................................................... [...]
  | o- block ............................................................................. [Storage Objects: 0]
  | o- fileio ............................................................................ [Storage Objects: 0]
  | o- pscsi ............................................................................. [Storage Objects: 0]
  | o- ramdisk ........................................................................... [Storage Objects: 0]
  o- iscsi ....................................................................................... [Targets: 0]
  o- loopback .................................................................................... [Targets: 0]
/> cd backstores/block 
/backstores/block> create my_disk1 /dev/vda
Created block storage object my_disk1 using /dev/vda.
/backstores/block> cd ../../iscsi 
/iscsi> create iqn.2019-05.com.example:server2
Created target iqn.2019-05.com.example:server2.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
/iscsi> cd iqn.2019-05.com.example:server2/tpg1/luns 
/iscsi/iqn.20...er2/tpg1/luns> create /backstores/block/my_disk1
Created LUN 0.
/iscsi/iqn.20...er2/tpg1/luns> cd ../acls 
/iscsi/iqn.20...er2/tpg1/acls> create iqn.2019-05.com.example:client
Created Node ACL for iqn.2019-05.com.example:client
Created mapped LUN 0.
/iscsi/iqn.20...er2/tpg1/acls> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json
[root@server2 mfs]# 

server1:

[root@server1 3.0.103]# yum install -y iscsi-*	安装iscsi的客户端
[root@server1 3.0.103]# vim /etc/iscsi/initiatorname.iscsi
	InitiatorName=iqn.2019-05.com.example:client
[root@server1 3.0.103]# iscsiadm -m discovery -t st -p 172.25.66.2	查找共享设备
172.25.66.2:3260,1 iqn.2019-05.com.example:server2
[root@server1 3.0.103]# iscsiadm -m node -l	登陆共享设备
Logging in to [iface: default, target: iqn.2019-05.com.example:server2, portal: 172.25.66.2,3260] (multiple)
Login to [iface: default, target: iqn.2019-05.com.example:server2, portal: 172.25.66.2,3260] successful.
查看共享磁盘
[root@server1 3.0.103]# fdisk -l
Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

创建分区并且格式化
[root@server1 3.0.103]# fdisk /dev/sdb
[root@server1 3.0.103]# mkfs.xfs /dev/sdb1

挂载
[root@server1 3.0.103]# mount /dev/sdb1 /mnt/
[root@server1 3.0.103]# cd /var/lib/mfs/
[root@server1 mfs]# cp -p * /mnt/
[root@server1 mfs]# cd /mnt/
[root@server1 mnt]# ls
changelog.10.mfs  changelog.3.mfs  metadata.crc  metadata.mfs.back.1  stats.mfs
changelog.1.mfs   changelog.9.mfs  metadata.mfs  metadata.mfs.empty
[root@server1 mnt]# cd /var/lib/mfs/
[root@server1 mfs]# ls
changelog.10.mfs  changelog.3.mfs  metadata.crc  metadata.mfs.back.1  stats.mfs
changelog.1.mfs   changelog.9.mfs  metadata.mfs  metadata.mfs.empty
[root@server1 mfs]# chown mfs.mfs /mnt/
[root@server1 mfs]# cd
[root@server1 ~]# umount /mnt/
[root@server1 ~]# mount /dev/sdb1 /var/lib/mfs/
[root@server1 ~]# systemctl start moosefs-master
[root@server1 ~]# systemctl stop moosefs-master

server4:

[root@server4 ~]# yum install -y iscsi-*
[root@server1 ~]# scp /etc/iscsi/initiatorname.iscsi server4:/etc/iscsi/
[root@server4 ~]# iscsiadm -m discovery -t st -p 172.25.66.2
172.25.66.2:3260,1 iqn.2019-05.com.example:server2
[root@server4 ~]# iscsiadm -m node -l
Logging in to [iface: default, target: iqn.2019-05.com.example:server2, portal: 172.25.66.2,3260] (multiple)
Login to [iface: default, target: iqn.2019-05.com.example:server2, portal: 172.25.66.2,3260] successful.
[root@server4 ~]# mount /dev/sdb1 /var/lib/mfs
[root@server4 ~]# systemctl start moosefs-master

创建集群的存储资源

[root@server1 ~]# pcs resource create mfsdata ocf:heartbeat:Filesystem device=/dev/sdb1 directory=/var/lib/mfs fstype=xfs op monitor interval=30s

创建集群的mfs启动

[root@server1 ~]# pcs resource create mfsd systemd:moosefs-master op monitor interval=1min

为了让这些集群资源可以统一工作在一个节点上，我们为其创建一个组，组的第一个成员的工作节点就是组的工作节点

[root@server1 ~]# pcs resource group add mfsgroup vip mfsdata mfsd

我们可以发现他们全部工作在server4上

我们来进行测试，关掉server4

[root@server1 ~]# pcs cluster stop server4

我们可以发现server4挂掉后server1上线
重新启动server4

[root@server1 ~]# pcs cluster start server4

添加fence

我们在之前配置rhcs套件时讲过fence，当遇到问题时直接闭掉坏的节点，相当于直接关掉电源
1.在server1，server4安装fence-virt

[root@server1 ~]# yum install -y fence-virt
[root@server4 ~]# yum install -y fence-virt
[root@server1 ~]# pcs stonith list
fence_virt - Fence agent for virtual machines
fence_xvm - Fence agent for virtual machines

2.客户端安装fence-virtd,并且配置fence

[root@foundation66 ~]# yum install -y fence-virtd
[root@foundation66 cluster]# fence_virtd -c
	Interface [virbr0]: br0       ##注意br0需要修改，其余回车即可
随即截取一个密钥，注意密钥名是不能随意的
[root@foundation66 cluster]# dd if=/dev/urandom of=fence_xvm.key bs=128 count=1
1+0 records in
1+0 records out
128 bytes (128 B) copied, 0.000241822 s, 529 kB/s
[root@foundation66 cluster]# ls
fence_xvm.key

[root@server1 ~]# mkdir /etc/cluster
[root@server4 ~]# mkdir /etc/cluster

[root@foundation66 cluster]# scp fence_xvm.key server1:/etc/cluster/
[root@foundation66 cluster]# scp fence_xvm.key server4:/etc/cluster/

[root@foundation66 cluster]# systemctl start fence_virtd.service
如果无法开启fence，那么可能是你缺少了几个模块
[root@foundation66 cluster]# yum install -y fence-virtd-libvirt.x86_64 fence-virtd-multicast.x86_64  fence-virtd-serial.x86_64

3.server1集群添加fence

	[root@server1 cluster]# pcs stonith create vmfence fence_xvm pcmk_host_map="ljz1:server1,ljz4:server4" op monitor interval=1min
	注意这里的映射关系是虚拟机的名字（qcow2文件名）和主机名
[root@server1 cluster]# pcs status
Cluster name: mycluster
Stack: corosync
Current DC: server1 (version 1.1.15-11.el7-e174ec8) - partition with quorum
Last updated: Tue May 21 04:35:16 2019		Last change: Tue May 21 04:35:05 2019 by root via cibadmin on server1

2 nodes and 4 resources configured

Online: [ server1 server4 ]

Full list of resources:

 Resource Group: mfsgroup
     vip	(ocf::heartbeat:IPaddr2):	Started server1
     mfsdata	(ocf::heartbeat:Filesystem):	Started server1
     mfsd	(systemd:moosefs-master):	Started server1
 vmfence	(stonith:fence_xvm):	Stopped

[root@server1 cluster]# pcs property set stonith-enabled=true
[root@server1 cluster]# crm_verify -L -V

测试：
使server1崩溃

[root@server1 cluster]# echo c > /proc/sysrq-trigger