一、实验架构图以及要求
实验架构图
实验要求：
要求能通过keepalived的vip访问静态或者动态页面，mysql的master节点能够实现故障自动切换
二、实验步骤
1、环境准备
| 服务器类型 | IP地址 | 需安装 |
| ha01 | 192.168.229.60 | ipvsadm、keepalived |
| ha02 | 192.168.229.50 | ipvsadm、keepalived |
| master01 | 192.168.229.90 | MySQL、NFS、MHA node |
| slave01 | 192.168.229.80 | MySQL、Nginx、Tomcat、MHA manager、MHA node |
| slave02 | 192.168.229.70 | MySQL、Nginx、Tomcat、MHA node |
| VIP | 192.168.229.188 | |
2、所有服务器关闭防火墙
systemctl stop firewalld.service setenforce 0
3、搭建LVS-DR
3.1 配置负载调度器ha01（192.168.229.60）
modprobe ip_vs cat /proc/net/ip_vs yum -y install ipvsadm
3.2 配置虚拟 IP 地址（VIP：192.168.229.188）
root@cm ~]# cd /etc/sysconfig/network-scripts/
[root@cm network-scripts]# ls
ifcfg-ens33 ifdown-ib ifdown-ppp ifdown-tunnel ifup-ib ifup-plusb ifup-Team network-functions
ifcfg-lo ifdown-ippp ifdown-routes ifup ifup-ippp ifup-post ifup-TeamPort network-functions-ipv6
ifdown ifdown-ipv6 ifdown-sit ifup-aliases ifup-ipv6 ifup-ppp ifup-tunnel
ifdown-bnep ifdown-isdn ifdown-Team ifup-bnep ifup-isdn ifup-routes ifup-wireless
ifdown-eth ifdown-post ifdown-TeamPort ifup-eth ifup-plip ifup-sit init.ipv6-global
[root@cm network-scripts]# cp ifcfg-ens33 ifcfg-ens33:0
[root@cm network-scripts]# vim ifcfg-ens33:0
[root@cm network-scripts]# ifup ens33:0
[root@cm network-scripts]# ifup ens33:0
[root@cm network-scripts]# ifconfig ens33:0
ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.229.188 netmask 255.255.255.255 broadcast 192.168.229.188
ether 00:0c:29:4a:77:8d txqueuelen 1000 (Ethernet)
[root@cm network-scripts]# route add -host 192.168.229.188 dev ens33:0
[root@cm network-scripts]# vim /etc/rc.local
[root@cm network-scripts]# cat /etc/rc.local
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.
touch /var/lock/subsys/local
/usr/sbin/route add -host 192.168.229.188 dev ens33:0
3.3 调整内核的ARP响应参数以阻止更新VIP的MAC地址，避免发生冲突
[root@cm network-scripts]# vim /etc/sysctl.conf [root@cm network-scripts]# cat /etc/sysctl.conf # sysctl settings are defined through files in # /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/. # # Vendors settings live in /usr/lib/sysctl.d/. # To override a whole file, create a new file with the same in # /etc/sysctl.d/ and put new settings there. To override # only specific settings, add a file with a lexically later # name in /etc/sysctl.d/ and put new settings there. # # For more information, see sysctl.conf(5) and sysctl.d(5). net.ipv4.ip_forward = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.ens33.send_redirects = 0 [root@cm network-scripts]# sysctl -p net.ipv4.ip_forward = 0 net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.ens33.send_redirects = 0
3.4 配置负载分配策略
[root@cm network-scripts]# ipvsadm-save > /etc/sysconfig/ipvsadm [root@cm network-scripts]# systemctl start ipvsadm.service [root@cm network-scripts]# ipvsadm -C [root@cm network-scripts]# ipvsadm -A -t 192.168.229.188:80 -s rr [root@cm network-scripts]# ipvsadm -a -t 192.168.229.188:80 -r 192.168.229.70:80 -g [root@cm network-scripts]# ipvsadm -a -t 192.168.229.188:80 -r 192.168.229.80:80 -g [root@cm network-scripts]# ipvsadm IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.229.188:http rr -> 192.168.229.70:http Route 1 0 0 -> 192.168.229.80:http Route 1 0 0
3.5 配置节点服务器（192.168.229.80、192.168.229.70）
3.5.1 配置虚拟 IP 地址（VIP：192.168.229.188）
[root@192 ~]# cd /etc/sysconfig/network-scripts/
[root@192 network-scripts]# cp ifcfg-ens33 ifcfg-lo:0
[root@192 network-scripts]# vim ifcfg-lo:0
[root@192 network-scripts]# ifup lo:0
[root@192 network-scripts]# cat ifcfg-lo:0
DEVICE=lo:0
ONBOOT=yes
IPADDR=192.168.229.188
NETMASK=255.255.255.255
[root@192 network-scripts]# ifup lo:0
[root@192 network-scripts]# ifconfig lo:0
lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 192.168.229.188 netmask 255.255.255.255
loop txqueuelen 1 (Local Loopback)
[root@192 network-scripts]# route add -host 192.168.229.188 dev lo:0
[root@192 network-scripts]# vim /etc/rc.local
[root@192 network-scripts]# cat /etc/rc.local
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.
touch /var/lock/subsys/local
route add -host 192.168.229.188 dev lo:0
3.5.2 调整内核的ARP响应参数以阻止更新VIP的MAC地址，避免发生冲突
[root@192 network-scripts]# vim /etc/sysctl.conf [root@192 network-scripts]# sysctl -p net.ipv4.conf.lo.arp_ignore = 1 net.ipv4.conf.lo.arp_announce = 2 net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2
3.6 部署共享存储（NFS服务器：192.168.229.90）
3.7 在节点服务器安装web服务（Nginx）并挂载共享目录
Nginx安装详见博客Nginx网站服务——编译安装、基于授权和客户端访问控制
如果节点服务器已经安装了httpd服务，记得关闭，再安装nginx，以防端口冲突
[root@www ~]# netstat -natp | grep 80 tcp 0 0 192.168.229.70:80 0.0.0.0:* LISTEN 48836/httpd [root@www ~]# systemctl stop httpd.service [root@www ~]# netstat -natp | grep 80
节点服务器挂载共享目录 （slave1 slave2分别挂载/opt/ly /opt/weq）
[root@192 ~]# rpm -q rpcbind nfs-utils rpcbind-0.2.0-42.el7.x86_64 nfs-utils-1.3.0-0.48.el7.x86_64 [root@192 ~]# systemctl start rpcbind [root@192 ~]# systemctl start nfs [root@192 ~]# systemctl enable rpcbind [root@192 ~]# systemctl enable nfs Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service. [root@192 ~]# showmount -e 192.168.229.90 Export list for 192.168.229.90: /opt/weq 192.168.229.0/24 /opt/ly 192.168.229.0/24 [root@192 ~]# mount.nfs 192.168.229.90:/opt/ly /usr/local/nginx/html/ [root@192 ~]# cd /usr/local/nginx/html/ [root@192 html]# ls index.html [root@192 html]# cat index.html <h1>This is node server1</h1>
使用（192.168.229.90）访问浏览器，测试LVS—DR模式
这并不是最终目的，我们需要在LVS-DR的基础上设置nginx tomcat的动静分离，访问静态页面交给nginx处理，动态页面交给tomcat处理
4、搭建Nginx+Tomcat的动静分离
4.1 安装nginx+tomcat
节点服务器已经安装了Nginx，再安装tomcat作为后端服务器（192.168.229.80 192.168.229.70）
tomcat的安装详见博客Tomcat部署及优化，也可使用shell脚本一键部署详见博客shell脚本一键部署——tomcat安装部署（含可复制代码）
4.2 动静分离Tomcat server配置（192.168.229.80 192.168.229.70）
4.2.1 配置Tomcat的动态网页显示内容
slave1 192.168.2229.80
slave2 192.168.2229.70
4.2.2 Tomcat实例主配置删除前面的 HOST 配置，添加新的HOST配置
两个节点服务器配置相同
4.3 Nginx server 配置（192.168.229.80 192.168.229.70）两者配置相同
[root@cm html]# mkdir /usr/local/nginx/html/img
[root@cm html]# cd /usr/local/nginx/html/img/
[root@cm img]# rz -E
rz waiting to receive.
[root@cm img]# ls
meme.jpg
[root@cm img]# cd /usr/local/nginx/conf/
[root@cm conf]# ls
fastcgi.conf fastcgi_params.default mime.types nginx.conf.default uwsgi_params
fastcgi.conf.default koi-utf mime.types.default scgi_params uwsgi_params.default
fastcgi_params koi-win nginx.conf scgi_params.default win-utf
[root@cm conf]# cp nginx.conf{,.bak}
[root@cm conf]# vim nginx.conf
upstream tomcat_server {
35 server 192.168.229.80:8080 weight=1;
36 server 192.168.229.70:8080 weight=1;
37 }
38
39 server {
40 listen 80;
41 server_name www.ly.com;
42
43 charset utf-8;
44
45 #access_log logs/host.access.log main;
46 location ~ .*\.jsp$ {
47 proxy_pass http://tomcat_server;
48 proxy_set_header HOST $host;
49 proxy_set_header X-Real-IP $remote_addr;
50 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
51 }
52
53 location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|css)$ {
54 root /usr/local/nginx/html/img;
55 expires 10d;
56 }
57
58 location / {
59 root html;
60 index index.html index.htm;
61 }
4.4、测试效果
使用浏览器（192.168.229.90）测试动静分离以及LVS-DR效果（前面已测试），不断刷新浏览器测试：
浏览器（192.168.229.90）访问 http://192.168.229.188/index.jsp
5、搭建keepalived
5.1 配置负载调度器(两者配置相同)
ha01（192.168.229.60）ha02（192.168.229.50）
5.2 配置keeplived（主（ha01）、备(ha02)DR 服务器上都要设置）
主DR 服务器： 192.168.229.60
[root@cm ~]# cd /etc/keepalived/
[root@cm keepalived]# ls
keepalived.conf
[root@cm keepalived]# cp keepalived.conf{,.bak}
[root@cm keepalived]# vim keepalived.conf
[root@cm keepalived]# systemctl start keepalived
[root@cm keepalived]# ip addr show dev ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:4a:77:8d brd ff:ff:ff:ff:ff:ff
inet 192.168.229.60/24 brd 192.168.229.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.229.188/32 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::25a6:a3f:9572:ce67/64 scope link
valid_lft forever preferred_lft forever
[root@cm keepalived]# scp keepalived.conf 192.168.229.50:/etc/keepalived/
The authenticity of host '192.168.229.50 (192.168.229.50)' can't be established.
ECDSA key fingerprint is SHA256:zvdcI+0BjCl1doLfrmClVqmsd6bgzvlrhn1Q/9wq5K8.
ECDSA key fingerprint is MD5:36:dc:ed:72:35:2c:97:c5:22:0f:59:49:da:87:0d:82.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.229.50' (ECDSA) to the list of known hosts.
root@192.168.229.50's password:
keepalived.conf 100% 1177 2.3MB/s 00:00
重启服务
[root@cm keepalived]# systemctl start keepalived
备DR 服务器： 192.168.229.50
重启服务
[root@192 keepalived]# systemctl restart keepalived.service
5.3 调整内核 proc 响应参数，关闭linux内核的重定向参数响应
主DR在LVS-DR模式中已经调整了响应参数，这里只需要修改备DR服务器即可
[root@192 keepalived]# vim /etc/sysctl.conf [root@192 keepalived]# sysctl -p net.ipv4.conf.all.send_redirects = 0 net.ipv4.conf.default.send_redirects = 0 net.ipv4.conf.ens33.send_redirects = 0
5.4 测试
在浏览器（192.168.229.90）访问测试
关闭主DR（ha01）的keepalived服务
查看备DR（ha01）的IP
在浏览器（192.168.229.90）访问测试
6、搭建MHA
6.1 搭建主从复制（一主两从）
master01(192.168.229.90)、slave01(192.168.229.80)、slave02(192.168.229.70)安装mysql，详见博客LAMP源码编译安装（二）——编译安装mysqld 服务、PHP解析环境以及安装论坛（附超详细步骤，快来学习吧！）
Master01、Slave01、Slave02 节点上安装 mysql5.7
修改 Master01、Slave01、Slave02 节点的主机名
hostnamectl set-hostname Mysql1 hostnamectl set-hostname Mysql2 hostnamectl set-hostname Mysql3
在所有服务器上/etc/hosts添加IP与主机名的解析并进行ping测试
[root@mysql1 ~]# vim /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.229.90 Mysql1 192.168.229.80 Mysql2 192.168.229.70 Mysql3
修改 Master01、Slave01、Slave02 节点的 Mysql主配置文件/etc/my.cnf
##Master01 节点##
vim /etc/my.cnf [mysqld]
#添加内容 server-id = 1 log_bin = master-bin log-slave-updates = true systemctl restart mysqld
##Slave01、Slave02 节点##
vim /etc/my.cnf server-id = 2 #三台服务器的 server-id 不能一样 log_bin = master-bin relay-log = relay-log-bin relay-log-index = slave-relay-bin.index systemctl restart mysqld
在 Master01、Slave01、Slave02 节点上都创建两个软链接
[root@mysql1 ~]# systemctl restart mysqld.service
[root@mysql1 ~]# ln -s /usr/local/mysql/bin/{mysql,mysqlbinlog} /usr/sbin/
[root@mysql1 ~]# ll /usr/sbin/mysql*
lrwxrwxrwx 1 root root 26 9月 22 14:01 /usr/sbin/mysql -> /usr/local/mysql/bin/mysql
lrwxrwxrwx 1 root root 32 9月 22 14:01 /usr/sbin/mysqlbinlog -> /usr/local/mysql/bin/mysqlbinlog
所有数据库节点进行 mysql 授权（一主两从）
mysql -uroot -pabc123 grant replication slave on *.* to 'myslave'@'192.168.229.%' identified by '12345'; #从数据库同步使用 grant all privileges on *.* to 'mha'@'192.168.229.%' identified by 'manager'; #manager 使用 grant all privileges on *.* to 'mha'@'Mysql1' identified by 'manager'; #防止从库通过主机名连接不上主库 grant all privileges on *.* to 'mha'@'Mysql2' identified by 'manager'; grant all privileges on *.* to 'mha'@'Mysql3' identified by 'manager'; flush privileges;
在 Master01 节点查看二进制文件和同步点
mysql> show master status; +-------------------+----------+--------------+------------------+-------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +-------------------+----------+--------------+------------------+-------------------+ | master-bin.000001 | 1747 | | | | +-------------------+----------+--------------+------------------+-------------------+ 1 row in set (0.00 sec)
在 Slave01、Slave02 节点执行同步操作，并查看数据同步结果
mysql> change master to master_host='192.168.229.90',master_user='myslave',master_password='12345',master_log_file='master-bin.000001',master_log_pos=1747;
Query OK, 0 rows affected, 2 warnings (0.04 sec)
mysql> start slave;
Query OK, 0 rows affected (0.00 sec)
mysql> show slave status\G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.229.90
Master_User: myslave
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: master-bin.000001
Read_Master_Log_Pos: 1747
Relay_Log_File: relay-log-bin.000002
Relay_Log_Pos: 321
Relay_Master_Log_File: master-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
两个从库必须设置为只读模式：
set global read_only=1;
插入数据测试数据库同步
##在 Master 主库插入条数据，测试是否同步##
mysql> create database bbc;
Query OK, 1 row affected (0.01 sec)
mysql> use bbc;
Database changed
mysql> create table abc(id int,name char(5),age int);
Query OK, 0 rows affected (0.05 sec)
mysql> insert into abc values(1,'lili',20);
Query OK, 1 row affected (0.01 sec)
mysql> select * from abc
-> ;
+------+------+------+
| id | name | age |
+------+------+------+
| 1 | lili | 20 |
+------+------+------+
1 row in set (0.00 sec)
##在slave01 02 查看数据是否同步##
6.2、MHA搭建
在 Master01、Slave01、Slave02 所有服务器上都安装 MHA 依赖的环境，首先安装 epel 源
yum install epel-release --nogpgcheck -y yum install -y perl-DBD-MySQL \ perl-Config-Tiny \ perl-Log-Dispatch \ perl-Parallel-ForkManager \ perl-ExtUtils-CBuilder \ perl-ExtUtils-MakeMaker \ perl-CPAN
安装 MHA 软件包，先在 Master01、Slave01、Slave02所有服务器上必须先安装 node 组件
cd /opt tar zxvf mha4mysql-node-0.57.tar.gz cd mha4mysql-node-0.57 perl Makefile.PL make && make install
在 slave01节点上安装 manager 组件
cd /opt tar zxvf mha4mysql-manager-0.57.tar.gz cd mha4mysql-manager-0.57 perl Makefile.PL make && make install
在Master01、Slave01、Slave02所有服务器上配置无密码认证
在 mysql1 上配置到数据库节点 mysql2 和 mysql3 的无密码认证
ssh-keygen -t rsa ssh-copy-id 192.168.229.80 ssh-copy-id 192.168.229.70
在 mysql2 上配置到数据库节点 mysql1 和 mysql3 的无密码认证
ssh-keygen -t rsa ssh-copy-id 192.168.229.90 ssh-copy-id 192.168.229.80
在 mysql3 上配置到数据库节点 mysql1 和 mysql2 的无密码认证
ssh-keygen -t rsa ssh-copy-id 192.168.229.90 ssh-copy-id 192.168.229.80
在 manager 节点(slave1:192.168.229.80)上配置 MHA
在 manager 节点上复制相关脚本到/usr/local/bin 目录
[root@mysql2 samples]# cp -rp /opt/mha4mysql-manager-0.57/samples/scripts/ /usr/local/bin/ [root@mysql2 samples]# ll /usr/local/bin/ 总用量 84 -r-xr-xr-x 1 root root 16381 9月 22 15:45 apply_diff_relay_logs -r-xr-xr-x 1 root root 4807 9月 22 15:45 filter_mysqlbinlog -r-xr-xr-x 1 root root 1995 9月 22 15:50 masterha_check_repl -r-xr-xr-x 1 root root 1779 9月 22 15:50 masterha_check_ssh -r-xr-xr-x 1 root root 1865 9月 22 15:50 masterha_check_status -r-xr-xr-x 1 root root 3201 9月 22 15:50 masterha_conf_host -r-xr-xr-x 1 root root 2517 9月 22 15:50 masterha_manager -r-xr-xr-x 1 root root 2165 9月 22 15:50 masterha_master_monitor -r-xr-xr-x 1 root root 2373 9月 22 15:50 masterha_master_switch -r-xr-xr-x 1 root root 5171 9月 22 15:50 masterha_secondary_check -r-xr-xr-x 1 root root 1739 9月 22 15:50 masterha_stop -r-xr-xr-x 1 root root 8261 9月 22 15:45 purge_relay_logs -r-xr-xr-x 1 root root 7525 9月 22 15:45 save_binary_logs drwxr-xr-x 2 nginx nginx 103 5月 31 2015 scripts [root@mysql2 scripts]# cd /usr/local/bin/scripts/ [root@mysql2 scripts]# ls master_ip_failover master_ip_online_change power_manager send_report
复制上述的自动切换时 VIP 管理的脚本到 /usr/local/bin 目录，这里使用master_ip_failover脚本来管理 VIP 和故障切换
具体的脚本内容详见博客：MySQL —MHA高可用配置及故障切换
创建 MHA 软件目录并拷贝配置文件，这里使用app1.cnf配置文件来管理 mysql 节点服务器
mkdir /etc/masterha cp /opt/mha4mysql-manager-0.57/samples/conf/app1.cnf /etc/masterha vim /etc/masterha/app1.cnf #删除原有内容，直接复制并修改节点服务器的IP地址 [server default] manager_log=/var/log/masterha/app1/manager.log manager_workdir=/var/log/masterha/app1 master_binlog_dir=/usr/local/mysql/data master_ip_failover_script=/usr/local/bin/master_ip_failover master_ip_online_change_script=/usr/local/bin/master_ip_online_change password=manager ping_interval=1 remote_workdir=/tmp repl_password=12345 repl_user=myslave secondary_check_script=/usr/local/bin/masterha_secondary_check -s 192.168.229.80 -s 192.168.229.70 shutdown_script="" ssh_user=root user=mha [server1] hostname=192.168.229.90 port=3306 [server2] candidate_master=1 check_repl_delay=0 hostname=192.168.229.80 port=3306 [server3] hostname=192.168.229.70 port=3306
第一次配置需要在 Master 节点上手动开启虚拟IP
/sbin/ifconfig ens33:1 192.168.229.188/24
在 manager 节点上测试 ssh 无密码认证，如果正常最后会输出 successfully
[root@mysql2 .ssh]# masterha_check_ssh -conf=/etc/masterha/app1.cnf
在 manager 节点上测试 mysql 主从连接情况，最后出现 MySQL Replication Health is OK 字样说明正常
masterha_check_repl -conf=/etc/masterha/app1.cnf
在 manager （slave01）节点上启动 MHA 并查看MHA状态以及日志
[root@mysql2 masterha]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 & [1] 25892 [root@mysql2 masterha]# masterha_check_status --conf=/etc/masterha/app1.cnf app1 (pid:25892) is running(0:PING_OK), master:192.168.229.90 [root@mysql2 masterha]# cat /var/log/masterha/app1/manager.log | grep "current master" Thu Sep 23 00:38:20 2021 - [info] Checking SSH publickey authentication settings on the current master.. 192.168.229.90(192.168.229.90:3306) (current master)
6.3 故障切换测试
在 Master01 节点 Mysql1 上停止mysql服务
在 manager 节点（slave01）上监控观察日志记录
tail -f /var/log/masterha/app1/manager.log
证明了MHA搭建成功，能够实现master故障切换，实现了高可用
到此所有实验部分全部完成