基于MHA的MySQL高可用

2023-08-18 14:21:52

基于 MHA 的 MySQL 高可用方案

一、MHA 简介

MHA（Master High Availability）目前在 MySQL 高可用方面是一个相对成熟的解决方案，它由日本 DeNA 公司的 youshimaton 员工（现就职于 Facebook 公司）开发，是一套优秀的作为 MySQL 高可用性环境下故障切换和主从角色提升的高可用软件。在 MySQL 故障切换过程中，MHA 能做到在 0~30 秒之内自动完成数据库的主从故障切换操作，并且在进行故障切换的过程中，MHA 能在最大程度上保证数据的一致性，以达到真正意义上的高可用。

MHA 由两部分组成：MHA Manage（r 管理节点）和 MHA Node（数据节点）。MHA Manager

可以单独部署在一*立的机器上管理多个 master-slave 集群，也可以部署在一台 slave 节点上。MHA Node 运行在每台MySQL 服务器及Manager 服务器上，MHA Manager 会定时探测集群中的 master 节点，当 master 出现故障时，它可以自动将拥有最新数据的 slave 提升为新的 master，然后将所有其他的 slave 重新指向新提升的 master。整个故障转移过程对应用程序层面完全透明。

在 MHA 自动故障切换过程中，MHA 会试图从宕机的主服务器上保存二进制日志，最大程度的保证数据不丢失，但这种操作是有概率性的。例如，如果主服务器硬件故障或无法通过 ssh 访问，MHA 没法保存二进制日志，只进行故障转移从而丢失了最新的数据。使用 MySQL

5.5 的半同步复制，可以降低数据丢失的风险。MHA 可以与半同步复制结合起来。如果只有一个 slave 已经收到了最新的二进制日志，MHA 可以将最新的二进制日志应用于其他所有的slave 服务器上，因此可以保证所有节点的数据一致性。

目前 MHA 主要支持一主多从的架构，要搭建 MHA,要求一个复制集群中必须最少有三台数据库服务器，一主二从，即一台充当 master，一台充当备用 master，另外一台充当从库，因为至少需要三台服务器，出于机器成本的考虑，淘宝也在该基础上进行了改造，目前淘宝 TMHA 已经支持一主一从。另外对于想快速搭建的可以参考：MHA 快速搭建

我们自己使用其实也可以使用 1 主 1 从，但是 master 主机宕机后无法切换，以及无法补全

binlog。master 的 mysqld 进程crash 后，还是可以切换成功，以及补全 binlog 的。官方介绍：https://code.google.com/p/mysql-master-ha/

1.1 、工作流程

（1）从宕机崩溃的 master 上尝试保存二进制日志事件（binlog events）;

（2）识别含有最新更新的 slave 服务器；

（3）应用差异的中继日志（relay log）到其他的 slave；

（4）应用从 master 保存的二进制日志事件（binlog events）；

（5）提升一个 slave 为新的 master 服务器；

（6）将其他的 slave 连接指向新的 master 进行主从复制；

1.2 、MHA 工具介绍

MHA 软件由两部分组成，Manager 工具包和 Node 工具包，具体的说明如下：

masterha_check_ssh 检查 MHA 的 SSH 配置状况Manager 工具包主要包括以下几个工具：

masterha_check_repl 检查 MySQL 复制状况
masterha_manger 启动 MHA
masterha_check_status 检测当前MHA 运行状态
masterha_master_monitor 检测 master 是否宕机
masterha_master_switch 控制故障转移（自动或者手动）
masterha_conf_host 添加或删除配置的 server 信息

Node 工具包（这些工具通常由 MHA Manager 的脚本触发，无需人为操作）主要包括以下几个工具：

➢ save_binary_logs 保存和复制 master 的二进制日志 ➢ apply_diff_relay_logs 识别差异的中继日志事件并将其差异的事件应用于其他的 slave ➢ filter_mysqlbinlog 去除不必要的 ROLLBACK 事件（MHA 已不再使用这个工具） ➢ purge_relay_logs 清除中继日志（不会阻塞 SQL 线程）

注意：为了尽可能的减少主库硬件损坏宕机造成的数据丢失，因此在配置 MHA 的同时建议配置成 MySQL 5.5 的半同步复制。关于半同步复制原理各位自己进行查阅。（不是必须）

1.1 、MHA 环境说明

角色 IP 地址主机名 ServerID 数据库类型 Primary Master 192.168.200.111 server01 1 写入 Secondary Master 192.168.200.112 server02 2 写入 Slave1 192.168.200.113 server03 3 读 Slave2 192.168.200.114 server04 4 读 Manager 192.168.200.115 server05 - 监控复制组 LVS Matser 192.168.200.116 localhost - LVS Backup 192.168.200.117 localhost - 二、前期环境部署 2.1、配置所有主机名称 master1 主机: hostname server01 bash master2 主机: hostname server02 bash slave1 主机: hostname server03 bash slave2 主机: hostname server04 bash manager 主机: hostname server05 bash 2.2、配置所有主机名映射 [root@server05 ~]# vim /etc/hosts 192.168.200.111 server01 192.168.200.112 server02 192.168.200.113 server03 192.168.200.114 server04 192.168.200.115 server05 scp /etc/hosts 192.168.200.111:/etc/ scp /etc/hosts 192.168.200.112:/etc/ scp /etc/hosts 192.168.200.113:/etc/ scp /etc/hosts 192.168.200.114:/etc/ 2.3、所有主机关闭防火墙和安全机制 systemctl stop iptables systemctl stop firewalld setenforce 0 2.4、下载 mha-manager 和 mha-node http://downloads.mariadb.com/MHA/ 三、安装 MHA node 3.1、所有主机安装 MHA node 及相关 perl 依赖包 安装 epel 源: rpm -ivh epel-release-latest-7.noarch.rpm yum install -y perl-DBD-MySQL.x86_64 perl-DBI.x86_64 perl-CPAN perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker rpm -q perl-DBD-MySQL.x86_64 perl-DBI.x86_64 perl-CPAN perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker perl-DBD-MySQL-4.023-6.el7.x86_64 perl-DBI-1.627-4.el7.x86_64 perl-CPAN-1.9800-292.el7.noarch perl-ExtUtils-CBuilder-0.28.2.6-292.el7.noarch perl-ExtUtils-MakeMaker-6.68-3.el7.noarch 注意：安装后建议检查一下所需软件包是否全部安装 3.2、所有主机上安装 MHA Node tar xf mha4mysql-node-0.56.tar.gz cd mha4mysql-node-0.56/ perl Makefile.PL make && make install 3.3、MHA Node 安装完后会在 /usr/local/bin 生成以下脚本 ls -l /usr/local/bin/ 总用量 40 -r-xr-xr-x 1 root root 16346 12 月 29 15:07 apply_diff_relay_logs -r-xr-xr-x 1 root root 4807 12 月 29 15:07 filter_mysqlbinlog -r-xr-xr-x 1 root root 7401 12 月 29 15:07 purge_relay_logs -r-xr-xr-x 1 root root 7395 12 月 29 15:07 save_binary_logs 四、安装 MHA Manger 注意：安装 MHA Manger 之前也需要安装 MHA Node 4.1、首先安装 MHA Manger 依赖的 perl 模块（我这里使用 yum 安装） yum install -y perl perl-Log-Dispatch perl-Parallel-ForkManager perl-DBD-MySQL perl-DBI perl-Time-HiRes yum -y install perl-Config-Tiny-2.14-7.el7.noarch.rpm rpm -q perl cpan perl-Log-Dispatch perl-Parallel-ForkManager perl-DBD-MySQL perl-DBI perl-Time-HiRes perl-Config-Tiny perl-5.16.3-292.el7.x86_64 perl-Log-Dispatch-2.41-1.el7.1.noarch perl-Parallel-ForkManager-1.18-2.el7.noarch perl-DBD-MySQL-4.023-6.el7.x86_64 perl-DBI-1.627-4.el7.x86_64 perl-Time-HiRes-1.9725-3.el7.x86_64 perl-Config-Tiny-2.14-7.el7.noarch 注意：之前时候 perl-Config-Tiny.noarch 没有安装成功,后来用 cpan(cpan install Config::Tiny ) 4.2、安装 MHA Manger 软件包 tar xf mha4mysql-manager-0.56.tar.gz cd mha4mysql-manager-0.56/ perl Makefile.PL make && make install 4.3、安装完成后会有以下脚本文件 ls -l /usr/local/bin/ 总用量 76 -r-xr-xr-x 1 root root 16346 12 月 29 15:08 apply_diff_relay_logs -r-xr-xr-x 1 root root 4807 12 月 29 15:08 filter_mysqlbinlog -r-xr-xr-x 1 root root 1995 12 月 29 15:37 masterha_check_repl -r-xr-xr-x 1 root root 1779 12 月 29 15:37 masterha_check_ssh -r-xr-xr-x 1 root root 1865 12 月 29 15:37 masterha_check_status -r-xr-xr-x 1 root root 3201 12 月 29 15:37 masterha_conf_host -r-xr-xr-x 1 root root 2517 12 月 29 15:37 masterha_manager -r-xr-xr-x 1 root root 2165 12 月 29 15:37 masterha_master_monitor -r-xr-xr-x 1 root root 2373 12 月 29 15:37 masterha_master_switch -r-xr-xr-x 1 root root 3879 12 月 29 15:37 masterha_secondary_check -r-xr-xr-x 1 root root 1739 12 月 29 15:37 masterha_stop -r-xr-xr-x 1 root root 7401 12 月 29 15:08 purge_relay_logs -r-xr-xr-x 1 root root 7395 12 月 29 15:08 save_binary_logs 五、配置 SSH 密钥对验证服务器之间需要实现密钥对验证。关于配置密钥对验证可看下面步骤。但是有一点需要注意：不能禁止 password 登陆，否则会出现错误 1. 服务器先生成一个密钥对 2. 把自己的公钥传给对方 5.1、Server05(192.168.200.115)上: ssh-keygen -t rsa ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111 ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112 ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113 ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114 注意：Server05 需要连接每个主机测试，因为第一次连接的时候需要输入 yes，影响后期故 障切换时，对于每个主机的 SSH 控制。 ssh server01 ssh server02 ssh server03 ssh server04 5.2、Primary Master(192.168.200.111): ssh-keygen -t rsa ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112 ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113 ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114 5.3、Secondary Master(192.168.200.112): ssh-keygen -t rsa ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111 ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113 ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114 5.4、slave1(192.168.200.113): ssh-keygen -t rsa ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111 ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112 ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.114 5.5、slave2(192.168.200.114): ssh-keygen -t rsa ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.111 ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.112 ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.200.113 六、安装 mysql 111-114 主机上的操作： yum -y install mariadb mariadb-server mariadb-devel systemctl start mariadb netstat -lnpt | grep :3306 设置数据库初始密码（后续操作中使用） mysqladmin -u root password 123456 七、搭建主从复制环境注意：binlog-do-db 和 replicate-ignore-db 设置必须相同。 MHA 在启动时候会检测过滤规则，如果过滤规则不同，MHA 将不启动监控和故障转移功能。 7.1、修改 mysql 主机的配置文件 Primary Master(192.168.200.111): vim /etc/my.cnf [mysqld] server-id = 1 log-bin=master-bin log-slave-updates=true relay_log_purge=0 systemctl restart mariadb Secondary Master(192.168.200.112): vim /etc/my.cnf [mysqld] server-id=2 log-bin=master-bin log-slave-updates=true relay_log_purge=0 systemctl restart mariadb slave1(192.168.200.113): vim /etc/my.cnf [mysqld] server-id=3 log-bin=mysql-bin relay-log=slave-relay-bin log-slave-updates=true relay_log_purge=0 systemctl restart mariadb slave2(192.168.200.114): vim /etc/my.cnf [mysqld] server-id=4 log-bin=mysql-bin relay-log=slave-relay-bin log-slave-updates=true relay_log_purge=0 systemctl restart mariadb 7.2、在 Primary Master(192.168.200.111)上对旧数据进行备份新机器不用作 mysqldump --master-data=2 --single-transaction -R --triggers -A > all.sql 参数解释： --master-data=2 备份时刻记录 master 的 Binlog 位置和 Position --single-transaction 获取一致性快照 -R 备份存储过程和函数 -triggres 备份触发器 -A 备份所有的库。 7.3、mysql 服务器创建复制授权用户 grant replication slave on *.* to 'repl'@'192.168.200.%' identified by '123456'; flush privileges; 7.4、查看主库备份时的 binlog 名称和位置 MariaDB [(none)]> show master status; +-------------------+----------+--------------+------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | +-------------------+----------+--------------+------------------+ | master-bin.000001 | 474 | | | +-------------------+----------+--------------+------------------+ 1 row in set (0.00 sec) 7.5、把数据备份复制其他主机 scp all.sql 192.168.200.112:/tmp/ scp all.sql 192.168.200.113:/tmp/ scp all.sql 192.168.200.114:/tmp/ 7.6、导入备份数据到 112-114 主机上执行复制相关命令 mysql -uroot -p123456< /tmp/all.sql stop slave; CHANGE MASTER TO MASTER_HOST='192.168.200.111', MASTER_USER='repl', MASTER_PASSWORD='123456', MASTER_LOG_FILE='master-bin.000001', MASTER_LOG_POS=474; start slave; show slave status\G # 检查 IO 和 SQL 线程是否为：yes Slave_IO_Running: Yes Slave_SQL_Running: Yes 7.7、主从同步故障处理 Slave_IO_Running: No Slave_SQL_Running: Yes -----------------------------------忽略部分信息----------------------------------- Last_IO_Errno: 1236 Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file' -----------------------------------忽略部分信息----------------------------------- 处理方式： stop slave; reset slave; set global sql_slave_skip_counter =1 ; start slave; 7.8、三台 slave 服务器设置 read_only 状态从库对外只提供读服务，只所以没有写进 mysql 配置文件，是因为随时 server02 会提升为 master [root@server02 ~]# mysql -uroot -p123456 -e 'set global read_only=1' [root@server03 ~]# mysql -uroot -p123456 -e 'set global read_only=1' [root@server04 ~]# mysql -uroot -p123456 -e 'set global read_only=1' 7.9、创建监控用户（111-114 主机上的操作）： grant all privileges on *.* to 'root'@'192.168.200.%' identified by '123456'; flush privileges; 为自己的主机名授权： grant all privileges on *.* to 'root'@'server04' identified by '123456'; flush privileges; 到这里整个 mysql 主从集群环境已经搭建完毕。八、配置 MHA 环境 8.1、创建 MHA 的工作目录及相关配置文件 Server05(192.168.200.115)：在软件包解压后的目录里面有样例配置文件 mkdir /etc/masterha cp mha4mysql-manager-0.56/samples/conf/app1.cnf /etc/masterha 8.2、修改 app1.cnf 配置文件 /usr/local/bin/master_ip_failover 脚本需要根据自己环境修改 ip 和网卡名称等。 vim /etc/masterha/app1.cnf [server default] #设置 manager 的工作日志 manager_workdir=/var/log/masterha/app1 #设置 manager 的日志,这两条都是默认存在的 manager_log=/var/log/masterha/app1/manager.log #设置 master 默认保存 binlog 的位置,以便 MHA 可以找到 master 日志 master_binlog_dir=/var/lib/mysql #设置自动 failover 时候的切换脚本 master_ip_failover_script= /usr/local/bin/master_ip_failover #设置 mysql 中 root 用户的密码 password=123456 user=root #ping 包的时间间隔 ping_interval=1 #设置远端 mysql 在发生切换时保存 binlog 的具体位置 remote_workdir=/tmp #设置复制用户的密码和用户名 repl_password=123456 repl_user=repl [server1] hostname=server01 port=3306 [server2] hostname=server02 candidate_master=1 port=3306 check_repl_delay=0 [server3] hostname=server03 port=3306 [server4] hostname=server04 port=3306 8.3、配置故障转移脚本 [root@server05 ~]# vim /usr/local/bin/master_ip_failover #!/usr/bin/env perl use strict; use warnings FATAL => 'all'; use Getopt::Long; my ( $command, $ssh_user, $orig_master_host, $orig_master_ip, $orig_master_port, $new_master_host, $new_master_ip, $new_master_port, ); my $vip = '192.168.200.100'; # 写入 VIP my $key = "1"; #非 keepalived 方式切换脚本使用的 my $ssh_start_vip = "/sbin/ifconfig ens32:$key $vip"; my $ssh_stop_vip = "/sbin/ifconfig ens32:$key down"; #那么这里写服务的开关命令 $ssh_user = "root"; GetOptions( 'command=s' => \$command, 'ssh_user=s' => \$ssh_user, 'orig_master_host=s' => \$orig_master_host, 'orig_master_ip=s' => \$orig_master_ip, 'orig_master_port=i' => \$orig_master_port, 'new_master_host=s' => \$new_master_host, 'new_master_ip=s' => \$new_master_ip, 'new_master_port=i' => \$new_master_port, ); exit &main(); sub main { print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n"; if ( $command eq "stop" || $command eq "stopssh" ) { # $orig_master_host, $orig_master_ip, $orig_master_port are passed. # If you manage master ip address at global catalog database, # invalidate orig_master_ip here. my $exit_code = 1; #eval { # print "Disabling the VIP on old master: $orig_master_host \n"; # &stop_vip(); # $exit_code = 0; #}; eval { print "Disabling the VIP on old master: $orig_master_host \n"; #my $ping=`ping -c 1 10.0.0.13 | grep "packet loss" | awk -F',' '{print $3}' | awk '{print $1}'`; #if ( $ping le "90.0%"&& $ping gt "0.0%" ){ #$exit_code = 0; #} #else { &stop_vip(); # updating global catalog, etc $exit_code = 0; #} }; if ($@) { warn "Got Error: $@\n"; exit $exit_code; } exit $exit_code; } elsif ( $command eq "start" ) { # all arguments are passed. # If you manage master ip address at global catalog database, # activate new_master_ip here. # You can also grant write access (create user, set read_only=0, etc) here. my $exit_code = 10; eval { print "Enabling the VIP - $vip on the new master - $new_master_host \n"; &start_vip(); $exit_code = 0; }; if ($@) { warn $@; exit $exit_code; } exit $exit_code; } elsif ( $command eq "status" ) { print "Checking the Status of the script.. OK \n"; `ssh $ssh_user\@$orig_master_ip \" $ssh_start_vip \"`; exit 0; } else { &usage(); exit 1; } } # A simple system call that enable the VIP on the new master sub start_vip() { `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`; } # A simple system call that disable the VIP on the old_master sub stop_vip() { `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`; } sub usage { print "Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port -- new_master_host=host --new_master_ip=ip --new_master_port=port\n"; } [root@server05 ~]# chmod +x /usr/local/bin/master_ip_failover 8.4、设置从库 relay log 的清除方式(112-114)： mysql -uroot -p123456 -e 'set global relay_log_purge=0;' 注意： MHA 在故障切换的过程中，从库的恢复过程依赖于 relay log 的相关信息，所以这里要将 relay log 的自动清除设置为 OFF，采用手动清除 relay log 的方式。在默认情况下，从服务器上的中继日志会在 SQL 线程执行完毕后被自动删除。但是在 MHA 环境中，这些中继日志在恢复其他从服务器时可能会被用到，因此需要禁用中继日志的自动清除功能。定期清除中继日志需要考虑到复制延时的问题。在 ext3 的文件系统下，删除大的文件需要一定的时间，会导致严重的复制延时。为了避免复制延时，需要暂时为中继日志创建硬链接，因为在 linux 系统中通过硬链接删除大文件速度会很快。（在 mysql 数据库中，删除大表时，通常也采用建立硬链接的方式） 8.5、配置从库（112-114）relay_log 清除脚本加入计划任务 MHA 节点中包含了 pure_relay_logs 命令工具，它可以为中继日志创建硬链接，执行 SET GLOBAL relay_log_purge=1,等待几秒钟以便 SQL 线程切换到新的中继日志，再执行 SET GLOBAL relay_log_purge=0。 vim purge_relay_log.sh #!/bin/bash user=root passwd=123456 #注意：数据库要有密码,填自己所设置的密码就可以，前面设置过 port=3306 log_dir='/tmp' work_dir='/tmp' purge='/usr/local/bin/purge_relay_logs' if [ ! -d $log_dir ] then mkdir $log_dir -p fi $purge --user=$user --password=$passwd --disable_relay_log_purge --port=$port --workdir=$work_dir >> $log_dir/purge_relay_logs.log 2>&1 chmod +x purge_relay_log.sh crontab -e 0 4 * * * /bin/bash /root/purge_relay_log.sh pure_relay_logs 脚本参数如下所示： --user mysql 用户名 --password mysql 密码 --port 端口号 --workdir 指定创建 relay log 的硬链接的位置，默认是/var/tmp，由于系统不同分区创建硬链接文件会失败，故需要执行硬链接具体位置，成功执行脚本后，硬链接的中继日志文件被删除 --disable_relay_log_purge 默认情况下，如果 relay_log_purge=1，脚本会什么都不清理，自动退出，通过设定这个参数，当 relay_log_purge=1 的情况下会将 relay_log_purge 设置为 0。清理 relay log 之后，最后将参数设置为 OFF。 8.6、手动清除中继日志 purge_relay_logs --user=root --password=123456 --disable_relay_log_purge --port=3306 --workdir=/tmp 2017-08-31 21:33:52: purge_relay_logs script started. Found relay_log.info: /usr/local/mysql/data/relay-log.info Removing hard linked relay log files slave-relay-bin* under /tmp.. done. Current relay log file: /usr/local/mysql/data/slave-relay-bin.000002 Archiving unused relay log files (up to /usr/local/mysql/data/slave-relay-bin.000001) ... Creating hard link for /usr/local/mysql/data/slave-relay-bin.000001 under /tmp/slave-relay-bin.000001 .. ok. Creating hard links for unused relay log files completed. Executing SET GLOBAL relay_log_purge=1; FLUSH LOGS; sleeping a few seconds so that SQL thread can delete older relay log files (if i t keeps up); SET GLOBAL relay_log_purge=0; .. ok. Removing hard linked relay log files slave-relay-bin* under /tmp.. done. 2017-08-31 21:33:56: All relay log purging operations succeeded. 8.7、检查 MHA ssh 通信状态 [root@server05 ~]# masterha_check_ssh --conf=/etc/masterha/app1.cnf Sat Dec 29 16:03:57 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sat Dec 29 16:03:57 2018 - [info] Reading application default configurations from /etc/masterha/app1.cnf.. Sat Dec 29 16:03:57 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf.. Sat Dec 29 16:03:57 2018 - [info] Starting SSH connection tests.. Sat Dec 29 16:04:02 2018 - [debug] Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from root@server02(192.168.200.112:22) to root@server01(192.168.200.111:22).. Sat Dec 29 16:03:59 2018 - [debug] ok. Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from root@server02(192.168.200.112:22) to root@server03(192.168.200.113:22).. Sat Dec 29 16:04:00 2018 - [debug] ok. Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from root@server02(192.168.200.112:22) to root@server04(192.168.200.114:22).. Sat Dec 29 16:04:02 2018 - [debug] ok. Sat Dec 29 16:04:02 2018 - [debug] Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from root@server03(192.168.200.113:22) to root@server01(192.168.200.111:22).. Sat Dec 29 16:04:00 2018 - [debug] ok. Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from root@server03(192.168.200.113:22) to root@server02(192.168.200.112:22).. Sat Dec 29 16:04:01 2018 - [debug] ok. Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from root@server03(192.168.200.113:22) to root@server04(192.168.200.114:22).. Sat Dec 29 16:04:02 2018 - [debug] ok. Sat Dec 29 16:04:02 2018 - [debug] Sat Dec 29 16:03:57 2018 - [debug] Connecting via SSH from root@server01(192.168.200.111:22) to root@server02(192.168.200.112:22).. Sat Dec 29 16:03:59 2018 - [debug] ok. Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from root@server01(192.168.200.111:22) to root@server03(192.168.200.113:22).. Sat Dec 29 16:04:00 2018 - [debug] ok. Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from root@server01(192.168.200.111:22) to root@server04(192.168.200.114:22).. Sat Dec 29 16:04:01 2018 - [debug] ok. Sat Dec 29 16:04:02 2018 - [debug] Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from root@server04(192.168.200.114:22) to root@server01(192.168.200.111:22).. Sat Dec 29 16:04:00 2018 - [debug] ok. Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from root@server04(192.168.200.114:22) to root@server02(192.168.200.112:22).. Sat Dec 29 16:04:01 2018 - [debug] ok. Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from root@server04(192.168.200.114:22) to root@server03(192.168.200.113:22).. Sat Dec 29 16:04:02 2018 - [debug] ok. Sat Dec 29 16:04:02 2018 - [info] All SSH connection tests passed successfully. 最后会返回 successfully 表示没有问题 8.9、检查整个集群的状态 [root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf Sat Dec 29 16:03:57 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sat Dec 29 16:03:57 2018 - [info] Reading application default configurations from /etc/masterha/app1.cnf.. Sat Dec 29 16:03:57 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf.. Sat Dec 29 16:03:57 2018 - [info] Starting SSH connection tests.. Sat Dec 29 16:04:02 2018 - [debug] Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from root@server02(192.168.200.112:22) to root@server01(192.168.200.111:22).. Sat Dec 29 16:03:59 2018 - [debug] ok. Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from root@server02(192.168.200.112:22) to root@server03(192.168.200.113:22).. Sat Dec 29 16:04:00 2018 - [debug] ok. Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from root@server02(192.168.200.112:22) to root@server04(192.168.200.114:22).. Sat Dec 29 16:04:02 2018 - [debug] ok. Sat Dec 29 16:04:02 2018 - [debug] Sat Dec 29 16:03:58 2018 - [debug] Connecting via SSH from root@server03(192.168.200.113:22) to root@server01(192.168.200.111:22).. Sat Dec 29 16:04:00 2018 - [debug] ok. Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from root@server03(192.168.200.113:22) to root@server02(192.168.200.112:22).. Sat Dec 29 16:04:01 2018 - [debug] ok. Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from root@server03(192.168.200.113:22) to root@server04(192.168.200.114:22).. Sat Dec 29 16:04:02 2018 - [debug] ok. Sat Dec 29 16:04:02 2018 - [debug] Sat Dec 29 16:03:57 2018 - [debug] Connecting via SSH from root@server01(192.168.200.111:22) to root@server02(192.168.200.112:22).. Sat Dec 29 16:03:59 2018 - [debug] ok. Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from root@server01(192.168.200.111:22) to root@server03(192.168.200.113:22).. Sat Dec 29 16:04:00 2018 - [debug] ok. Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from root@server01(192.168.200.111:22) to root@server04(192.168.200.114:22).. Sat Dec 29 16:04:01 2018 - [debug] ok. Sat Dec 29 16:04:02 2018 - [debug] Sat Dec 29 16:03:59 2018 - [debug] Connecting via SSH from root@server04(192.168.200.114:22) to root@server01(192.168.200.111:22).. Sat Dec 29 16:04:00 2018 - [debug] ok. Sat Dec 29 16:04:00 2018 - [debug] Connecting via SSH from root@server04(192.168.200.114:22) to root@server02(192.168.200.112:22).. Sat Dec 29 16:04:01 2018 - [debug] ok. Sat Dec 29 16:04:01 2018 - [debug] Connecting via SSH from root@server04(192.168.200.114:22) to root@server03(192.168.200.113:22).. Sat Dec 29 16:04:02 2018 - [debug] ok. Sat Dec 29 16:04:02 2018 - [info] All SSH connection tests passed successfully. [root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf [root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf Sat Dec 29 16:04:53 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sat Dec 29 16:04:53 2018 - [info] Reading application default configurations from /etc/masterha/app1.cnf.. Sat Dec 29 16:04:53 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf.. Sat Dec 29 16:04:53 2018 - [info] MHA::MasterMonitor version 0.56. Creating directory /var/log/masterha/app1.. done. Sat Dec 29 16:04:55 2018 - [info] Dead Servers: Sat Dec 29 16:04:55 2018 - [info] Alive Servers: Sat Dec 29 16:04:55 2018 - [info] server01(192.168.200.111:3306) Sat Dec 29 16:04:55 2018 - [info] server02(192.168.200.112:3306) Sat Dec 29 16:04:55 2018 - [info] server03(192.168.200.113:3306) Sat Dec 29 16:04:55 2018 - [info] server04(192.168.200.114:3306) Sat Dec 29 16:04:55 2018 - [info] Alive Slaves: Sat Dec 29 16:04:55 2018 - [info] server02(192.168.200.112:3306) Version=5.5.56-MariaDB (oldest major version between slaves) log-bin:enabled Sat Dec 29 16:04:55 2018 - [info] Replicating from 192.168.200.111(192.168.200.111:3306) Sat Dec 29 16:04:55 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sat Dec 29 16:04:55 2018 - [info] server03(192.168.200.113:3306) Version=5.5.56-MariaDB (oldest major version between slaves) log-bin:enabled Sat Dec 29 16:04:55 2018 - [info] Replicating from 192.168.200.111(192.168.200.111:3306) Sat Dec 29 16:04:55 2018 - [info] server04(192.168.200.114:3306) Version=5.5.56-MariaDB (oldest major version between slaves) log-bin:enabled Sat Dec 29 16:04:55 2018 - [info] Replicating from 192.168.200.111(192.168.200.111:3306) Sat Dec 29 16:04:55 2018 - [info] Current Alive Master: server01(192.168.200.111:3306) Sat Dec 29 16:04:55 2018 - [info] Checking slave configurations.. Sat Dec 29 16:04:55 2018 - [info] Checking replication filtering settings.. Sat Dec 29 16:04:55 2018 - [info] binlog_do_db= , binlog_ignore_db= Sat Dec 29 16:04:55 2018 - [info] Replication filtering check ok. Sat Dec 29 16:04:55 2018 - [info] Starting SSH connection tests.. Sat Dec 29 16:05:00 2018 - [info] All SSH connection tests passed successfully. Sat Dec 29 16:05:00 2018 - [info] Checking MHA Node version.. Sat Dec 29 16:05:02 2018 - [info] Version check ok. Sat Dec 29 16:05:02 2018 - [info] Checking SSH publickey authentication settings on the current master.. Sat Dec 29 16:05:03 2018 - [info] HealthCheck: SSH to server01 is reachable. Sat Dec 29 16:05:04 2018 - [info] Master MHA Node version is 0.56. Sat Dec 29 16:05:04 2018 - [info] Checking recovery script configurations on the current master.. Sat Dec 29 16:05:04 2018 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test -- manager_version=0.56 --start_file=master-bin.000001 Sat Dec 29 16:05:04 2018 - [info] Connecting to root@server01(server01).. Creating /tmp if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /var/lib/mysql, up to master-bin.000001 Sat Dec 29 16:05:04 2018 - [info] Master setting check done. Sat Dec 29 16:05:04 2018 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. Sat Dec 29 16:05:04 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server02 --slave_ip=192.168.200.112 --slave_po rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxxSat Dec 29 16:05:04 2018 - [info] Connecting to root@192.168.200.112(server02:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000002 Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sat Dec 29 16:05:05 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server03 --slave_ip=192.168.200.113 --slave_po rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxxSat Dec 29 16:05:05 2018 - [info] Connecting to root@192.168.200.113(server03:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to slave-relay-bin.000002 Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sat Dec 29 16:05:07 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server04 --slave_ip=192.168.200.114 --slave_po rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxxSat Dec 29 16:05:07 2018 - [info] Connecting to root@192.168.200.114(server04:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to slave-relay-bin.000002 Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sat Dec 29 16:05:08 2018 - [info] Slaves settings check done. Sat Dec 29 16:05:09 2018 - [info] server01 (current master) +--server02 +--server03 +--server04 Sat Dec 29 16:05:09 2018 - [info] Checking replication health on server02.. Sat Dec 29 16:05:09 2018 - [info] ok. Sat Dec 29 16:05:09 2018 - [info] Checking replication health on server03.. Sat Dec 29 16:05:09 2018 - [info] ok. Sat Dec 29 16:05:09 2018 - [info] Checking replication health on server04.. Sat Dec 29 16:05:09 2018 - [info] ok. Sat Dec 29 16:05:09 2018 - [info] Checking master_ip_failover_script status: Sat Dec 29 16:05:09 2018 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=server01 --orig_master_ip=192.168.200.111 --orig_m aster_port=3306 IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start=== Checking the Status of the script.. OK bash: /etc/init.d/keepalived: 没有那个文件或目录 Sat Dec 29 16:05:09 2018 - [info] OK. Sat Dec 29 16:05:09 2018 - [warning] shutdown_script is not defined. Sat Dec 29 16:05:09 2018 - [info] Got exit code 0 (Not master dead). MySQL Replication Health is OK. 九、VIP 配置管理 Master vip 配置有两种方式,一种是通过 keepalived 或者 heartbeat 类似的软件的方式管理 VIP 的浮动,另一种为通过命令方式管理。通过命令方式管理 VIP 地址：打开在前面编辑过的文件/etc/masterha/app1.cnf，检查如下行是否正确，再检查集群状态。 [root@server05 ~]# grep -n 'master_ip_failover_script' /etc/masterha/app1.cnf 9:master_ip_failover_script= /usr/local/bin/master_ip_failover Primary Master(192.168.200.111) [root@server01 ~]# ip a | grep ens32 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32 inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1 Server05(192.168.200.115)修改故障转移脚本 [root@server05 ~]# head -13 /usr/local/bin/master_ip_failover #!/usr/bin/env perl use strict; use warnings FATAL => 'all'; use Getopt::Long; my ( $command, $ssh_user, $orig_master_host, $orig_master_ip, $orig_master_port, $new_master_host, $new_master_ip, $new_master_port, ); my $vip = '192.168.200.100'; # 写入 VIP my $key = "1"; #非 keepalived 方式切换脚本使用的 my $ssh_start_vip = "/sbin/ifconfig ens32:$key $vip"; #若是使用 keepalived my $ssh_stop_vip = "/sbin/ifconfig ens32:$key down"; #那么这里写服务的开关命令 /usr/local/bin/master_ip_failover 文件的内容意思是当主库发生故障时，会触发 MHA 切换，MHA manager 会停掉主库上的 ens32:1 接口，触发虚拟 ip 漂移到备选从库，从而完成切换。 Server05(192.168.200.115) 检查 manager 状态 masterha_check_status --conf=/etc/masterha/app1.cnf app1 is stopped(2:NOT_RUNNING). 注意：如果正常会显示"PING_OK"，否则会显示"NOT_RUNNING"，代表 MHA 监控没有开启。 Server05(192.168.200.115) 开启 manager 监控 [root@server05 ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dea d_master_conf --ignore_last_failover< /dev/null >/var/log/masterha/app1/manager.log 2>&1 & [1] 65837 启动参数介绍： --remove_dead_master_conf 该参数代表当发生主从切换后，老的主库的 ip 将会从配置文件中移除。 --manger_log 日志存放位置 --ignore_last_failover 在缺省情况下，如果 MHA 检测到连续发生宕机，且两次宕机间隔不足 8 小时的话，则不会进行 Failover，之所以这样限制是为了避免 ping-pong 效应。该参数代表忽略上次 MHA 触发切换产生的文件，默认情况下，MHA 发生切换后会在日志目录，也就是上面我设置的/data 产生 app1.failover.complete 文件，下次再次切换的时候如果发现该目录下存在该文件将不允许触发切换，除非在第一次切换后收到删除该文件，为了方便，这里设置为--ignore_last_failover。 Server05(192.168.200.115)查看 Server05 监控是否正常： [root@monitor ~]# masterha_check_status --conf=/etc/masterha/app1.cnf app1 (pid:65837) is running(0:PING_OK), master:server01 可以看见已经在监控了 Server05(192.168.200.115)查看启动日志 [root@server05 ~]# cat /var/log/masterha/app1/manager.log Sat Dec 29 16:09:50 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping. Sat Dec 29 16:09:50 2018 - [info] Reading application default configurations from /etc/masterha/app1.cnf.. Sat Dec 29 16:09:50 2018 - [info] Reading server configurations from /etc/masterha/app1.cnf.. Sat Dec 29 16:09:50 2018 - [info] MHA::MasterMonitor version 0.56. Sat Dec 29 16:09:51 2018 - [info] Dead Servers: Sat Dec 29 16:09:51 2018 - [info] Alive Servers: Sat Dec 29 16:09:51 2018 - [info] server01(192.168.200.111:3306) Sat Dec 29 16:09:51 2018 - [info] server02(192.168.200.112:3306) Sat Dec 29 16:09:51 2018 - [info] server03(192.168.200.113:3306) Sat Dec 29 16:09:51 2018 - [info] server04(192.168.200.114:3306) Sat Dec 29 16:09:51 2018 - [info] Alive Slaves: Sat Dec 29 16:09:51 2018 - [info] server02(192.168.200.112:3306) Version=5.5.56-MariaDB (oldest major version between slaves) log-bin:enabled Sat Dec 29 16:09:51 2018 - [info] Replicating from 192.168.200.111(192.168.200.111:3306) Sat Dec 29 16:09:51 2018 - [info] Primary candidate for the new Master (candidate_master is set) Sat Dec 29 16:09:51 2018 - [info] server03(192.168.200.113:3306) Version=5.5.56-MariaDB (oldest major version between slaves) log-bin:enabled Sat Dec 29 16:09:51 2018 - [info] Replicating from 192.168.200.111(192.168.200.111:3306) Sat Dec 29 16:09:51 2018 - [info] server04(192.168.200.114:3306) Version=5.5.56-MariaDB (oldest major version between slaves) log-bin:enabled Sat Dec 29 16:09:51 2018 - [info] Replicating from 192.168.200.111(192.168.200.111:3306) Sat Dec 29 16:09:51 2018 - [info] Current Alive Master: server01(192.168.200.111:3306) Sat Dec 29 16:09:51 2018 - [info] Checking slave configurations.. Sat Dec 29 16:09:51 2018 - [info] Checking replication filtering settings.. Sat Dec 29 16:09:51 2018 - [info] binlog_do_db= , binlog_ignore_db= Sat Dec 29 16:09:51 2018 - [info] Replication filtering check ok. Sat Dec 29 16:09:51 2018 - [info] Starting SSH connection tests.. Sat Dec 29 16:09:58 2018 - [info] All SSH connection tests passed successfully. Sat Dec 29 16:09:58 2018 - [info] Checking MHA Node version.. Sat Dec 29 16:10:01 2018 - [info] Version check ok. Sat Dec 29 16:10:01 2018 - [info] Checking SSH publickey authentication settings on the current master.. Sat Dec 29 16:10:02 2018 - [info] HealthCheck: SSH to server01 is reachable. Sat Dec 29 16:10:03 2018 - [info] Master MHA Node version is 0.56. Sat Dec 29 16:10:03 2018 - [info] Checking recovery script configurations on the current master.. Sat Dec 29 16:10:03 2018 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/tmp/save_binary_logs_test -- manager_version=0.56 --start_file=master-bin.000001 Sat Dec 29 16:10:03 2018 - [info] Connecting to root@server01(server01).. Creating /tmp if not exists.. ok. Checking output directory is accessible or not.. ok. Binlog found at /var/lib/mysql, up to master-bin.000001 Sat Dec 29 16:10:03 2018 - [info] Master setting check done. Sat Dec 29 16:10:03 2018 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers.. Sat Dec 29 16:10:03 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server02 --slave_ip=192.168.200.112 --slave_po rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxxSat Dec 29 16:10:03 2018 - [info] Connecting to root@192.168.200.112(server02:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to mariadb-relay-bin.000002 Temporary relay log file is /var/lib/mysql/mariadb-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sat Dec 29 16:10:04 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server03 --slave_ip=192.168.200.113 --slave_po rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxxSat Dec 29 16:10:04 2018 - [info] Connecting to root@192.168.200.113(server03:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to slave-relay-bin.000002 Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sat Dec 29 16:10:05 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=server04 --slave_ip=192.168.200.114 --slave_po rt=3306 --workdir=/tmp --target_version=5.5.56-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxxSat Dec 29 16:10:05 2018 - [info] Connecting to root@192.168.200.114(server04:22).. Checking slave recovery environment settings.. Opening /var/lib/mysql/relay-log.info ... ok. Relay log found at /var/lib/mysql, up to slave-relay-bin.000002 Temporary relay log file is /var/lib/mysql/slave-relay-bin.000002 Testing mysql connection and privileges.. done. Testing mysqlbinlog output.. done. Cleaning up test file(s).. done. Sat Dec 29 16:10:06 2018 - [info] Slaves settings check done. Sat Dec 29 16:10:06 2018 - [info] server01 (current master) +--server02 +--server03 +--server04 Sat Dec 29 16:10:06 2018 - [info] Checking master_ip_failover_script status: Sat Dec 29 16:10:06 2018 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=server01 --orig_master_ip=192.168.200.111 --orig_m aster_port=3306 IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start=== Checking the Status of the script.. OK bash: /etc/init.d/keepalived: 没有那个文件或目录 Sat Dec 29 16:10:07 2018 - [info] OK. Sat Dec 29 16:10:07 2018 - [warning] shutdown_script is not defined. Sat Dec 29 16:10:07 2018 - [info] Set master ping interval 1 seconds. Sat Dec 29 16:10:07 2018 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes. Sat Dec 29 16:10:07 2018 - [info] Starting ping health check on server01(192.168.200.111:3306).. Thu Aug 31 21:55:23 2017 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.. 注意：其中"Ping(SELECT) succeeded, waiting until MySQL doesn't respond.."说明整个系统已经 开始监控了。 关闭 MHA manager 监控，忽略操作 masterha_stop --conf=/etc/masterha/app1.cnf 发现已经将 VIP：192.168.200.100 绑定在网卡 ens32。 [root@server01 ~]# ip a | grep ens32 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32 inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1 Primary Master(192.168.200.111) 模拟主库故障 [root@server01 ~]# systemctl stop mariadb [root@server01 ~]# netstat -lnpt | grep :3306 [root@server01 ~]# ip a | grep ens32 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32 slave1(192.168.200.113)状态： MariaDB [(none)]> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.200.112 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: master-bin.000001 Read_Master_Log_Pos: 1372 Relay_Log_File: slave-relay-bin.000002 Relay_Log_Pos: 530 Relay_Master_Log_File: master-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes slave2(192.168.200.114)状态： MariaDB [(none)]> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.200.112 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: master-bin.000001 Read_Master_Log_Pos: 1372 Relay_Log_File: slave-relay-bin.000002 Relay_Log_Pos: 530 Relay_Master_Log_File: master-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Server05(192.168.200.115) 监控已经自动关闭： [root@server05 ~]# [1]+ 完成 nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 Server05(192.168.200.115) 查看监控配置文件已经发生了变化（server01 的配置已被删除）： [root@server05 ~]# cat /etc/masterha/app1.cnf [server default] manager_log=/var/log/masterha/app1/manager.log manager_workdir=/var/log/masterha/app1 master_binlog_dir=/var/lib/mysql master_ip_failover_script=/usr/local/bin/master_ip_failover password=123456 ping_interval=1 remote_workdir=/tmp repl_password=123456 repl_user=repl user=root [server2] candidate_master=1 check_repl_delay=0 hostname=server02 port=3306 [server3] hostname=server03 port=3306 [server4] hostname=server04 port=3306 Server05(192.168.200.115) 故障切换过程中的日志文件内容如下： [root@server05 ~]# tail -f /var/log/masterha/app1/manager.log Selected server02 as a new master. server02: OK: Applying all logs succeeded. server02: OK: Activated master IP address. server04: This host has the latest relay log events. server03: This host has the latest relay log events. Generating relay diff files from the latest slave succeeded. server04: OK: Applying all logs succeeded. Slave started, replicating from server02. server03: OK: Applying all logs succeeded. Slave started, replicating from server02. server02: Resetting slave info succeeded. Master failover to server02(192.168.200.112:3306) completed successfully. 故障主库修复及 VIP 切回测试 Primary Master(192.168.200.111)： [root@server01 ~]# systemctl start mariadb [root@server01 ~]# netstat -lnpt | grep :3306 tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 6131/mysqld Primary Master(192.168.200.111) 指向新的主库 [root@server01 ~]# mysql -u root -p123456 stop slave; CHANGE MASTER TO MASTER_HOST='192.168.200.112', MASTER_USER='repl', MASTER_PASSWORD='123456'; start slave; show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.200.112 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: master-bin.000001 Read_Master_Log_Pos: 1372 Relay_Log_File: mariadb-relay-bin.000002 Relay_Log_Pos: 1208 Relay_Master_Log_File: master-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Server05(192.168.200.115) 修改监控配置文件添加 server1 配置： [root@server05 ~]# vim /etc/masterha/app1.cnf [server01] hostname=server01 port=3306 Server05(192.168.200.115) 检查集群状态： [root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf -----------------------------------忽略部分信息----------------------------------- Thu Aug 31 22:20:30 2017 - [info] Alive Servers: Thu Aug 31 22:20:30 2017 - [info] server01(192.168.200.111:3306) Thu Aug 31 22:20:30 2017 - [info] server02(192.168.200.112:3306) Thu Aug 31 22:20:30 2017 - [info] server03(192.168.200.113:3306) Thu Aug 31 22:20:30 2017 - [info] server04(192.168.200.114:3306) -----------------------------------忽略部分信息----------------------------------- server02 (current master) +--server01 +--server03 +--server04 -----------------------------------忽略部分信息----------------------------------- MySQL Replication Health is OK. Server05(192.168.200.115) 开启监控 [root@server05 ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf QQ：981343876 34 / 50 Tel：13651028933 --remove_dead_master_conf --ignore_last_failover< /dev/null >/var/log/masterha/app1/manager.log 2>&1 &[1] 68199 Secondary Master(192.168.200.112) 关闭现有主库 mysql [root@server02 ~]# ip a | grep ens32 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.200.112/24 brd 192.168.200.255 scope global ens32 inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1 [root@server02 ~]# systemctl stop mariadb [root@server02 ~]# netstat -lnpt | grep :3306 Primary Master(192.168.200.111) [root@server01 ~]# ip a | grep ens32 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.200.111/24 brd 192.168.200.255 scope global ens32 inet 192.168.200.100/24 brd 192.168.200.255 scope global secondary ens32:1 slave1(192.168.200.113)状态： MariaDB [(none)]> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.200.111 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: master-bin.000002 Read_Master_Log_Pos: 923 Relay_Log_File: slave-relay-bin.000002 Relay_Log_Pos: 530 Relay_Master_Log_File: master-bin.000002 Slave_IO_Running: Yes Slave_SQL_Running: Yes slave2(192.168.200.114)状态： MariaDB [(none)]> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.200.111 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: master-bin.000002 Read_Master_Log_Pos: 923 Relay_Log_File: slave-relay-bin.000002 Relay_Log_Pos: 530 Relay_Master_Log_File: master-bin.000002 Slave_IO_Running: Yes Slave_SQL_Running: Yes Server05(192.168.200.115) 配置文件变化（已经移除故障机 server2 配置）： [root@server05 ~]# cat /etc/masterha/app1.cnf [server default] manager_log=/var/log/masterha/app1/manager.log manager_workdir=/var/log/masterha/app1 master_binlog_dir=/var/lib/mysql master_ip_failover_script=/usr/local/bin/master_ip_failover password=123456 ping_interval=1 remote_workdir=/tmp repl_password=123456 repl_user=repl user=root [server1] hostname=server01 port=3306 [server3] hostname=server03 port=3306 [server4] hostname=server04 port=3306 Server05(192.168.200.115) 监控日志： -----------------------------------忽略部分信息----------------------------------- Selected server01 as a new master. server01: OK: Applying all logs succeeded. server01: OK: Activated master IP address. server03: This host has the latest relay log events. server04: This host has the latest relay log events. Generating relay diff files from the latest slave succeeded. server04: OK: Applying all logs succeeded. Slave started, replicating from server01. server03: OK: Applying all logs succeeded. Slave started, replicating from server01. server01: Resetting slave info succeeded. Master failover to server01(192.168.200.111:3306) completed successfully. 修复 Secondary Master(192.168.200.112)主机： [root@server02 ~]# systemctl start mariadb [root@server02 ~]# netstat -lnpt | grep :3306 tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 5982/mysqld Secondary Master(192.168.200.112) 指向新的主库 [root@server02 ~]# mysql -u root -p123456 stop slave; CHANGE MASTER TO MASTER_HOST='192.168.200.111', MASTER_USER='repl', MASTER_PASSWORD='123456'; start slave; show slave status\G MariaDB [(none)]> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.200.111 Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: master-bin.000002 Read_Master_Log_Pos: 923 Relay_Log_File: mariadb-relay-bin.000003 Relay_Log_Pos: 530 Relay_Master_Log_File: master-bin.000002 Slave_IO_Running: Yes Slave_SQL_Running: Yes Server05(192.168.200.115)修改监控配置文件添加 server2 配置： [root@server05 ~]# vim /etc/masterha/app1.cnf [server2] hostname=server02 candidate_master=1 port=3306 check_repl_delay=0 Server05(192.168.200.115)检查集群状态： [root@server05 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf server01 (current master) +--server02 +--server03 +--server04 -----------------------------------忽略部分信息----------------------------------- MySQL Replication Health is OK. 十、配置读请求负载均衡 10.1、安装 LVS 和 Keepalived 192.168.200.116 和 192.168.200.117 主机： yum -y install ipvsadm kernel-devel openssl-devel keepalived 10.2、master 主机设置 192.168.200.116 lvs master 主机配置： ! Configuration File for keepalived global_defs { notification_email { crushlinux@163.com } notification_email_from crushlinux@163.com smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL vrrp_skip_check_adv_addr vrrp_strict vrrp_garp_interval 0 vrrp_gna_interval 0 } vrrp_instance VI_1 { state MASTER interface ens32 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.200.200 } } virtual_server 192.168.200.200 3306 { delay_loop 6 lb_algo wrr lb_kind DR protocol TCP real_server 192.168.200.113 3306 { weight 1 TCP_CHECK { connect_timeout 10 nb_get_retry 3 delay_before_retry 3 connect_port 3306 } } real_server 192.168.200.114 3306 { weight 1 TCP_CHECK { connect_timeout 10 nb_get_retry 3 delay_before_retry 3 connect_port 3306 } } } systemctl restart keepalived 检查 VIP 地址 [root@localhost ~]# ip add | grep ens32 2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 inet 192.168.200.116/24 brd 192.168.200.255 scope global ens32 inet 192.168.200.200/32 scope global ens32 查看负载策略 [root@localhost ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.200.200:3306 wrr -> 192.168.200.113:3306 Route 1 0 0 -> 192.168.200.114:3306 Route 1 0 0 10.3、backup 主机设置 192.168.200.40 backup 主机配置： ! Configuration File for keepalived global_defs { notification_email { crushlinux@163.com } notification_email_from crushlinux@163.com smtp_server 192.168.200.1 smtp_connect_timeout 30 router_id LVS_DEVEL vrrp_skip_check_adv_addr vrrp_strict vrrp_garp_interval 0 vrrp_gna_interval 0 } vrrp_instance VI_1 { state BACKUP interface ens32 virtual_router_id 51 priority 99 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.200.200 } } virtual_server 192.168.200.200 3306 { delay_loop 6 lb_algo wrr lb_kind DR protocol TCP real_server 192.168.200.113 3306 { weight 1 TCP_CHECK { connect_timeout 10 nb_get_retry 3 delay_before_retry 3 connect_port 3306 } } real_server 192.168.200.114 3306 { weight 1 TCP_CHECK { connect_timeout 10 nb_get_retry 3 delay_before_retry 3 connect_port 3306 } } } systemctl restart keepalived 10.4、为数据库节点设置 VIP 地址及配置 ARP 相关参数 slave1(192.168.200.113)和 slave2(192.168.200.114)主机配置 realserver 脚本： [root@server03 ~]# vim realserver.sh #!/bin/bash SNS_VIP=192.168.200.200 ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP /sbin/route add -host $SNS_VIP dev lo:0 echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce sysctl -p >/dev/null 2>&1 realserver1 检测 VIP 地址 [root@server03 ~]# bash realserver.sh [root@server03 ~]# ip a | grep lo 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet 192.168.200.200/32 brd 192.168.200.200 scope global lo:0 inet 192.168.200.113/24 brd 192.168.200.255 scope global ens32 inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 远程复制到 realserver2 主机上 [root@server03 ~]# scp realserver.sh 192.168.200.114:/root/ realserver2 检测 VIP 地址 [root@server04 ~]# bash realserver.sh [root@server04 ~]# ip a |grep lo 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet 192.168.200.200/32 brd 192.168.200.200 scope global lo:0 inet 192.168.200.114/24 brd 192.168.200.255 scope global ens32 inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 10.5、数据库读请求测试 Server05(192.168.200.115)担任 client 角色测试，连接 vip [root@server05 ~]# mysql -uroot -p123456 -h192.168.200.200 -P3306 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 297 Server version: 5.5.56-MariaDB MariaDB Server Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> exit Bye 断开后在连接测试。。。。 [root@server05 ~]# mysql -uroot -p123456 -h192.168.200.200 -P3306 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 298 Server version: 5.5.56-MariaDB MariaDB Server Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> 查看请求分流 [root@localhost ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.200.200:3306 wrr -> 192.168.200.113:3306 Route 1 1 0 -> 192.168.200.114:3306 Route 1 0 1 [root@localhost ~]# ipvsadm -Lnc IPVS connection entries pro expire state source virtual destination TCP 01:48 FIN_WAIT 192.168.200.115:35462 192.168.200.200:3306 192.168.200.114:3306 TCP 14:50 ESTABLISHED 192.168.200.115:35464 192.168.200.200:3306 192.168.200.113:3306 10.6、数据库连接故障处理 [root@localhost ~]# mysql -u root -p123456 -h192.168.200.113 -P3306 ERROR 1129 (HY000): Host '192.168.200.113' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts' [root@server03 ~]# mysqladmin -uroot -p123456 flush-hosts 附录手动切换 MHA 脚本 [root@server05 ~]# vim /usr/local/bin/master_ip_online_change #!/usr/bin/env perl # Copyright (C) 2011 DeNA Co.,Ltd. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation; either version 2 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program; if not, write to the Free Software # Foundation, Inc., # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA ## Note: This is a sample script and is not complete. Modify the script based on your environment. use strict; use warnings FATAL => 'all'; use Getopt::Long; use MHA::DBHelper; use MHA::NodeUtil; use Time::HiRes qw( sleep gettimeofday tv_interval ); use Data::Dumper; my $_tstart; my $_running_interval = 0.1; my ( $command, $orig_master_is_new_slave, $orig_master_host, $orig_master_ip, $orig_master_port, $orig_master_user, $orig_master_password, $orig_master_ssh_user, $new_master_host, $new_master_ip, $new_master_port, $new_master_user, $new_master_password, $new_master_ssh_user ); my $vip = '192.168.200.100/24'; #VIP my $key = '1'; my $ssh_start_vip = "/sbin/ifconfig ens32:$key $vip"; my $ssh_stop_vip = "/sbin/ifconfig ens32:$key down"; my $orig_master_ssh_port = 22; my $new_master_ssh_port = 22; GetOptions( 'command=s' => \$command, 'orig_master_is_new_slave' => \$orig_master_is_new_slave, 'orig_master_host=s' => \$orig_master_host, 'orig_master_ip=s' => \$orig_master_ip, 'orig_master_port=i' => \$orig_master_port, 'orig_master_user=s' => \$orig_master_user, 'orig_master_password=s' => \$orig_master_password, 'orig_master_ssh_user=s' => \$orig_master_ssh_user, 'new_master_host=s' => \$new_master_host, 'new_master_ip=s' => \$new_master_ip, 'new_master_port=i' => \$new_master_port, 'new_master_user=s' => \$new_master_user, 'new_master_password=s' => \$new_master_password, 'new_master_ssh_user=s' => \$new_master_ssh_user, 'orig_master_ssh_port=i' => \$orig_master_ssh_port, 'new_master_ssh_port=i' => \$new_master_ssh_port, ); exit &main(); sub current_time_us { my ( $sec, $microsec ) = gettimeofday(); my $curdate = localtime($sec); return $curdate . "" . sprintf( "%06d", $microsec ); } sub sleep_until { my $elapsed =tv_interval($_tstart); if ( $_running_interval > $elapsed ) { sleep( $_running_interval - $elapsed ); } } sub get_threads_util { my $dbh = shift; my $my_connection_id = shift; my $running_time_threshold = shift; my $type = shift; $running_time_threshold = 0 unless ($running_time_threshold); $type = 0 unless ($type); my @threads; my $sth = $dbh->prepare("SHOW PROCESSLIST"); $sth->execute(); while ( my $ref = $sth->fetchrow_hashref() ) { my $id = $ref->{Id}; my $user = $ref->{User}; my $host = $ref->{Host}; my $command = $ref->{Command}; my $state = $ref->{State}; my $query_time = $ref->{Time}; my $info = $ref->{Info}; $info =~ s/^\s*(.*?)\s*$/$1/ if defined($info); next if ( $my_connection_id == $id ); next if ( defined($query_time) && $query_time < $running_time_threshold ); next if ( defined($command) && $command eq "Binlog Dump" ); next if ( defined($user) && $user eq "system user" ); next if ( defined($command) && $command eq "Sleep" && defined($query_time) && $query_time >= 1 ); if ( $type >= 1 ) { next if ( defined($command) && $command eq "Sleep" ); next if ( defined($command) && $command eq "Connect" ); } if ( $type >= 2 ) { next if ( defined($info) && $info =~ m/^select/i ); next if ( defined($info) && $info =~ m/^show/i ); } push @threads, $ref; } return @threads; } sub main { if ( $command eq "stop" ) { ## Gracefully killing connections on the current master # 1. Set read_only= 1 on the new master # 2. DROP USER so that no app user can establish new connections # 3. Set read_only= 1 on the current master # 4. Kill current queries # * Any database access failure will result in script die. my $exit_code = 1; eval { ## Setting read_only=1 on the new master (to avoid accident) my $new_master_handler = new MHA::DBHelper(); # args: hostname, port, user, password, raise_error(die_on_error)_or_not $new_master_handler->connect( $new_master_ip, $new_master_port, $new_master_user, $new_master_password, 1 ); print current_time_us() . " Set read_only on the new master.. "; $new_master_handler->enable_read_only(); if ( $new_master_handler->is_read_only() ) { print "ok.\n"; } else { die "Failed!\n"; } $new_master_handler->disconnect(); # Connecting to the orig master, die if any database error happens my $orig_master_handler = new MHA::DBHelper(); $orig_master_handler->connect( $orig_master_ip, $orig_master_port, $orig_master_user, $orig_master_password, 1 ); ## Drop application user so that nobody can connect. Disabling per-session binlog beforehand $orig_master_handler->disable_log_bin_local(); print current_time_us() . " Drpping app user on the orig master..\n"; #FIXME_xxx_drop_app_user($orig_master_handler); ## Waiting for N * 100 milliseconds so that current connections can exit my $time_until_read_only = 15; $_tstart = [gettimeofday]; my @threads = get_threads_util( $orig_master_handler->{dbh}, $orig_master_handler->{connection_id} ); while ( $time_until_read_only > 0 && $#threads >= 0 ) { if ( $time_until_read_only % 5 == 0 ) { printf "%s Waiting all running %d threads are disconnected.. (max %d milliseconds)\n", current_time_us(), $#threads + 1, $time_until_read_only * 100; if ( $#threads < 5 ) { print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n" foreach (@threads); } } sleep_until(); $_tstart = [gettimeofday]; $time_until_read_only--; @threads = get_threads_util( $orig_master_handler->{dbh}, $orig_master_handler->{connection_id} ); } ## Setting read_only=1 on the current master so that nobody(except SUPER) can write print current_time_us() . " Set read_only=1 on the orig master.. "; $orig_master_handler->enable_read_only(); if ( $orig_master_handler->is_read_only() ) { print "ok.\n"; } else { die "Failed!\n"; } ## Waiting for M * 100 milliseconds so that current update queries can complete my $time_until_kill_threads = 5; @threads = get_threads_util( $orig_master_handler->{dbh}, $orig_master_handler->{connection_id} ); while ( $time_until_kill_threads > 0 && $#threads >= 0 ) { if ( $time_until_kill_threads % 5 == 0 ) { printf "%s Waiting all running %d queries are disconnected.. (max %d milliseconds)\n", current_time_us(), $#threads + 1, $time_until_kill_threads * 100; if ( $#threads < 5 ) { print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n" foreach (@threads); } } sleep_until(); $_tstart = [gettimeofday]; $time_until_kill_threads--; @threads = get_threads_util( $orig_master_handler->{dbh}, $orig_master_handler->{connection_id} ); } ## Terminating all threads print current_time_us() . " Killing all application threads..\n"; $orig_master_handler->kill_threads(@threads) if ( $#threads >= 0 ); print current_time_us() . " done.\n"; $orig_master_handler->enable_log_bin_local(); $orig_master_handler->disconnect(); ## After finishing the script, MHA executes FLUSH TABLES WITH READ LOCK eval { `ssh -p$orig_master_ssh_port $orig_master_ssh_user\@$orig_master_host \" $ssh_stop_vip \"`; }; if ($@) { warn $@; } $exit_code = 0; }; if ($@) { warn "Got Error: $@\n"; exit $exit_code; } exit $exit_code; } elsif ( $command eq "start" ) { ## Activating master ip on the new master # 1. Create app user with write privileges # 2. Moving backup script if needed # 3. Register new master's ip to the catalog database # We don't return error even though activating updatable accounts/ip failed so that we don't interrupt slaves' recovery. # If exit code is 0 or 10, MHA does not abort my $exit_code = 10; eval { my $new_master_handler = new MHA::DBHelper(); # args: hostname, port, user, password, raise_error_or_not $new_master_handler->connect( $new_master_ip, $new_master_port, $new_master_user, $new_master_password, 1 ); ## Set read_only=0 on the new master $new_master_handler->disable_log_bin_local(); print current_time_us() . " Set read_only=0 on the new master.\n"; $new_master_handler->disable_read_only(); ## Creating an app user on the new master print current_time_us() . " Creating app user on the new master..\n"; #FIXME_xxx_create_app_user($new_master_handler); $new_master_handler->enable_log_bin_local(); $new_master_handler->disconnect(); ## Update master ip on the catalog database, etc `ssh -p$new_master_ssh_port $new_master_ssh_user\@$new_master_host \" $ssh_start_vip \"`; $exit_code = 0; }; if ($@) { warn "Got Error: $@\n"; exit $exit_code; } exit $exit_code; } elsif ( $command eq "status" ) { # do nothing exit 0; } else { &usage(); exit 1; } } sub usage { print "Usage: master_ip_online_change --command=start|stop|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new _master_host=host --new_master_ip=ip --new_master_port=port\n"; die; } [root@server05 ~]# chmod +x /usr/local/bin/master_ip_online_change 编辑脚本 send_report [root@server05 ~]# vim /usr/local/bin/send_report #!/usr/bin/perl use strict; use warnings FATAL => 'all'; use Mail::Sender; use Getopt::Long; #new_master_host and new_slave_hosts are set only when recovering master succeeded my ( $dead_master_host, $new_master_host, $new_slave_hosts, $subject, $body ); my $smtp='smtp.163.com'; my $mail_from='xxxx'; my $mail_user='xxxxx'; my $mail_pass='xxxxx'; my $mail_to=['xxxx','xxxx']; GetOptions( 'orig_master_host=s' => \$dead_master_host, 'new_master_host=s' => \$new_master_host, 'new_slave_hosts=s' => \$new_slave_hosts, 'subject=s' => \$subject, 'body=s' => \$body, ); mailToContacts($smtp,$mail_from,$mail_user,$mail_pass,$mail_to,$subject,$body); sub mailToContacts { my ( $smtp, $mail_from, $user, $passwd, $mail_to, $subject, $msg ) = @_; open my $DEBUG, "> /tmp/monitormail.log" or die "Can't open the debug file:$!\n"; my $sender = new Mail::Sender { ctype => 'text/plain; charset=utf-8', encoding => 'utf-8', smtp => $smtp, from => $mail_from, auth => 'LOGIN', TLS_allowed => '0', authid => $user, authpwd => $passwd, to => $mail_to, subject => $subject, debug => $DEBUG }; $sender->MailMsg( { msg => $msg, debug => $DEBUG } ) or print $Mail::Sender::Error; return 1; } # Do whatever you want here exit 0; [root@server05 ~]# chmod +x /usr/local/bin/send_report

码农公寓

一、MHA 简介

1.1 、工作流程

相关文章