数据库管理与高可用----MHA高可用配置及故障切换

实验思路:
1.MHA架构
1）数据库安装
2)一主两从
3)MHA搭建
故障模拟
1）主库失效
2）备选主库成为主库
3)从库2将备选主库指向为主库
案例环境
1.本案例环境
服务器CentOS7.4(64位)MHA-manager/192.168.100.130
管理节点，安装manager组件
服务器CentOS7.4(64位)Mysq11/192.168.100.150 Master 节点，安装node组件
服务器CentOS7.4(64位)Mysql2/192.168.100.160 Slave 节点，安装node组件
服务器CentOS7.4(64位)Mysql3/192.168.100.170 Slave节点，安装node组件
这里操作系统是CentOS7版本，所以这里下载MHA版本是0.57版本。
2.案例需求
本案例要求通过MHA监控MySQL数据库在故障时进行自动切换，不影响业务。
3.案例实现思路
1)安装MySQL数据库
2)配置MySQL一主两从
3)安装MHA软件
4)配置无密码认证
5)配置 MySQL MHA高可用
6)模拟master故障切换
在三台MySQL 节点上分别安装数据库，MySQL版本请使用5.6.36，cmake版本请使
用2.8.6。下面只在Mysql1上面做演示，安装过程如下。

1.安装编译依赖的环境
[root@Mysql1~]# yum install ncurses-devel gcc-c++ perl-Module-Install -y

2.安装gmake编译软件
[root@Mysql1~ ]# tar zxvf cmake-2.8.6.tar.gz
[root@Mysql1 ~]# cd cmake-2.8.6
[root@Mysql1 cmake-2.8.6]# ./configure
[root@Mysql1 cmake-2.8.6]# gmake && gmake install

3.安装MySQL数据库
[root@Mysql1~ ]# tar zxvf mysql-5.6.36.tar.gz
[root@Mysql1 ~]# cd mysql-5.6.36

cmake -DCMAKE_INSTALL_PREFIX=/usr/local/mysql \
-DDEFAULT_CHARSET=utf8 \
-DDEFAULT_COLLATION=utf8_general_ci \
-DWITII_EXTRA_CHARSETS=all \
-DSYSCONFDTR=/etc

[root@Mysql1 mysql-5.6.36]# make && make install
[root@Mysql1 mysql-5.6.36]# cp support-files/my-default.cnf /etc/my.cnf
[root@Mysql1 mysql-5.6.36]# cp support-files/mysql.server /etc/rc.d/init.d/mysqld
[root@Mysql1 ~]# chmod +x /etc/rc.d/init.d/mysqld
[root@MysqI1 ~]# chkconfig --add mysqld
[root@Mysql1 ~]# echo  "PATH=$PATH:/usr/local/mysql/bin" >> /etc/profile
[root@Mysql1 ~]# source /etc/profile
[root@Mysql1 ~]# groupadd mysql
[root@Mysql1 ~]# useradd -M -s /sbin/nologin mysql -g mysql
[root@Mysql1 ~]# chown -R mysql.mysql /usr/local/mysql

source /etc/profile
groupadd mysql
useradd -M -s /sbin/nologin mysql -g mysql
chown -R mysql.mysql /usr/local/mysql

[root@Mysql1 ~]# mkdir -p /data/mysql
/usr/local/mysql/scripts/mysql_install_db \
--basedir=/usr/local/mysql \
--datadir=/usr/local/mysql/data \
--user=mysql
/usr/local/mysql/scripts/mysql_install_db \
--basedir=/usr/local/mysql \
--datadir=/usr/local/mysql/data \
--user=mysql

4.修改Master的主配置文件/etc/my.cnf文件，三台服务器的server-id不能一样
[root@Mysql1 ~]# cat /etc/my.cnf
[mysqld]
server-id = 1
log_bin = master-bin
log-slave-updates = true
配置从服务器:
在/etc/my.cnf中修改或者增加下面内容。
[root@Mysql2 ~]# vim /etc/my.cnf
server-id = 2
log_bin = master-bin
relay-log = relay-log-bin
relay-log-index = slave-relay-bin.index

5.Mysql1、Mysql2、Mysql3分别做两个软链接
[root@Mysql1 ~]# In -s /usr/local/mysql/bin/mysql /usr/sbin/
[root@Mysql1 ~]# In -s /usr/local/mysql/bin/mysqlbinlog /usr/sbin/
ln -s /usr/local/mysql/bin/mysql /usr/sbin/
ln -s /usr/local/mysql/bin/mysqlbinlog /usr/sbin/
6、Mysql1、Mysql2、Mysql3启动MySQL。

systemctl stop firewalld
setenforce 0
/usr/local/mysql/bin/mysqld_safe --user=mysql &
netstat -ntap | grep 3306

7.配置MySQL一主两从

1).MySQL主从配置相对比较简单。需要注意的是授权。步骤如下:
在所有数据库节点上授权两个用户，一个是从库同步使用，另外一个是manager使用。
grant replication slave on*.* to 'myslave'@'192.168.100.%' identified by '123';
grant all privileges on *.* to 'mha'@'192.168.100.%' identified by 'manager';
flush privileges;
grant replication slave on *.* to 'myslave'@'192.168.100.%' identified by '123';
grant all privileges on *.* to 'mha'@'192.168.100.%' identified by 'manager';
flush privileges;

2).下面三条授权按理论是不用添加的，但是做案例实验环境时候通过MHA检查MySQL主从有报错，
报两个从库通过主机名连接不上主库，所以所有数据库加上下面的授权。
grant all privileges on *.* to 'mha'@'Mysql1' identified by 'manager';
grant all privileges on *.* to 'mha'@'Mysql2' identified by 'manager';
grant all privileges on *.* to 'mha'@'Mysql3' identified by 'manager';
flush privileges;
3).在Mysq1主机上查看二进制文件和同步点
mysql> show master status;
+-------------------+----------+--------------+------------------+-------------------+
| File              | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-------------------+----------+--------------+------------------+-------------------+
| master-bin.000001 |     1294 |              |                  |                   |
+-------------------+----------+--------------+------------------+-------------------+
1 row in set (0.01 sec)

4.接下来在Mysql2和Mysql3分别执行同步。
mysql> change master to master_host='192.168.100.150',master_user='myslave',master_password='123',master_log_file='master-bin.000010',master_log_pos=1294;
flush privileges;
start slave;
stop slave;
start slave;
5.查看IO和SQL线程都是yes代表同步是否正常。
mysql> show slave status\G;
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
mysql> show master status;
"查看二进制日志路径"
mysql> show variables like 'datadir';
+---------------+--------------+
| Variable_name | Value        |
+---------------+--------------+
| datadir       | /home/mysql/ |
+---------------+--------------+
必须设置两个从库为只读模式:
set global read_only=1;
flush privileges;
注意:设置完成直接验证主从复制功能

--------------------------安装MHA软件----------------------------
1.所有服务器上都安装MHA依赖的环境，首先安装epel源。
[root@MHA-manager ~]# yum install epel-release --nogpgcheck -y

yum install -y perl-DBD-MySQL \
perl-Config-Tiny \
perl-Log-Dispatch \
perl-Parallel-ForkManager \
perl-ExtUtils-CBuilder \
perl-ExtUtils-MakeMaker \
perl-CPAN

2.MHA 软件包对于每个操作系统版本不一样，这里CentOS7.4必须选择0.57版本，
在<注意:所有服务器>上必须先安装node组件，最后在MHA-manager节点上安装manager组件，
因为manager依赖node组件，下面都是在Mysql1上操作演示安装node组件。
//上传mha4mysql-node-0.57.tar.gz
[root@Mysql1~]# tar zxvf mha4mysql-node-0.57.tar.gz
[root@Mysql1 ~]# cd mha4mysql-node-0.57
[root@Mysql1 mha4mysql-node-o.57]# perl Makefile.PL 
[root@Mysql1 mha4mysql-node-o.57]# make
[root@Mysql1 mha4mysql-node-0.57]# make && make install

3.仅在MHA-manager上安装manager组件(!注意:一定要先安装node组件才能安装manager组件)
[root@Mysql1 mha4mysql-node-0.57]# cd ~
[root@MHA-manager ~]# tar zxvf mha4mysql-manager-0.57.tar.gz
[root@MHA-manager~]# cd mha4mysql-manager-0.57
[root@MHA-manager mha4mysql-manager-0.57]# perl Makefile.PL
[root@MHA-manager mha4mysql-manager-0.57]# make
[root@MHA-manager mha4mysql-manager-o.57]# make && make install

manager安装后在/usr/local/bin下面会生成几个工具，主要包括以下几个:
masterha_check_ssh检查MHA的SSH配置状况
masterha_check_repl检查MySQL复制状况
masterha_manger启动manager的脚本
masterha_check_status检测当前MHA运行状态
masterha_master_monitor检测master是否宕机
masterha_master_switch控制故障转移（自动或者手动)
masterha_conf_host添加或删除配置的server信息
masterha_stop关闭manager

4.node安装后也会在/usr/local/bin下面会生成几个脚本(这些工具通常由MHA
Manager的脚本触发，无需人为操作）主要如下:
save_binary_logs保存和复制master的二进制日志
apply_diff_relay_logs 识别差异的中继日志事件并将其差异的事件应用于其他的slave
filter_mysqlbinlog去除不必要的ROLLBACK事件(MHA已不再使用这个工具)
purge_relay_logs清除中继日志(不会阻塞SQL线程)

5.配置无密码认证
1).在manager上配置到所有数据库节点的无密码认证
[root@MHA-manager ~]# ssh-keygen -t rsa    //一路按回车键
[root@MHA-manager ~]# ssh-copy-id 192.168.100.150   //输入yes 及150的root密码
[root@MHA-manager ~]# ssh-copy-id 192.168.100.160
[root@MHA-manager ~]# ssh-copy-id 192.168.100.170
2).在Mysql1上配置到数据库节点Mysql2和Mysql3的无密码认证
[root@Mysql1 ~]# ssh-keygen -t rsa
[root@MysqI1 ~]# ssh-copy-id 192.168.100.160
[root@Mysql1 ~]# ssh-copy-id 192.168.100.170
3).在Mysql2上配置到数据库节点MysqI1和Mysql3的无密码认证
[root@Mysql2 ~]# ssh-keygen -t rsa
[root@Mysql2 ~]# ssh-copy-id 192.168.100.150
[root@Mysql2 ~]# ssh-copy-id 192.168.100.170
4).在Mysql3上配置到数据库节点Mysql1和Mysql2的无密码认证
[root@Mysql3 ~]# ssh-keygen -t rsa
[root@Mysql3 ~]# ssh-copy-id 192.168.100.150
[root@Mysql3 ~]# ssh-copy-id 192.168.100.160

6.配置MHA

1).在manager节点上复制相关脚本到/usr/local/bin目录。
[root@mha-manager~]# cp -ra /root/mha4mysql-manager-0.57/samples/scripts /usr/local/bin
//拷贝后会有四个执行文件
[root@MHA-manager ~]#ll/usr/local/bin/scripts/
总用量32
-rwxr-xr-x 1 mysql mysql 36485月312015 master_ip_failover#自动切换时VIP管理的脚本
-rwxr-xr-x 1 mysql mysql 98725月2509:07 master_ip_online_change #在线切换时vip的管理
-rwxr-xr-x 1 mysql mysql118675月312015 power_manager #故障发生后关闭主机的脚本
-rwxr-Xr-x 1 mysql mysql 13605月312015 send_report #因故障切换后发送报警的脚本

2).复制上述的自动切换时VIP管理的脚本到/usr/local/bin目录，这里使用脚本管理VIP,
[root@mha-manager~]# cp /usr/local/bin/scripts/master_ip_failover /usr/local/bin

3).修改内容如下:(删除原有内容，直接复制)

[root@MHA-manager ~]# vim /usr/local/bin/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';

use Getopt::Long;

my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
#############################添加内容部分#########################################
my $vip = '192.168.100.200';
my $brdc = '192.168.100.255';
my $ifdev = 'ens33';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig ens33:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens33:$key down";
my $exit_code = 0;
#my $ssh_start_vip = "/usr/sbin/ip addr add $vip/24 brd $brdc dev $ifdev label $ifdev:$key;/usr/sbin/arping -q -A -c 1 -I $ifdev $vip;iptables -F;";
#my $ssh_stop_vip = "/usr/sbin/ip addr del $vip/24 dev $ifdev label $ifdev:$key";
##################################################################################
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);

exit &main();

sub main {

print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

if ( $command eq "stop" || $command eq "stopssh" ) {

my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {

my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
exit 0;
}
else {
&usage();
exit 1;
}
}
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}

sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}

4).创建MHA软件目录并拷贝配置文件。
[root@MHA-manager ~]# cd /usr/local/bin/scripts/
[root@MHA-manager scripts]# cp master_ip_online_change /usr/local/bin/
[root@MHA-manager scripts]# cp send_report /usr/local/
[root@MHA-manager ~]# mkdir /etc/masterha
[root@MHA-manager ~]# cp /root/mha4mysql-manager-0.57/samples/conf/app1.cnf /etc/masterha
[root@MHA-manager ~]# vim /etc/masterha/app1.cnf

[server default]
manager_log=/var/log/masterha/app1/manager.log
manager_workdir=/var/log/masterha/app1
master_binlog_dir=/usr/local/mysql/data
master_ip_failover_script=/usr/local/bin/master_ip_failover
master_ip_online_change_script=/usr/local/bin/master_ip_online_change
password=manager
ping_interval=1
remote_workdir=/tmp
repl_password=123
repl_user=myslave
secondary_check_script=/usr/local/bin/masterha_secondary_check -s 192.168.100.160 -s 192.168.100.170
shutdown_script=""
ssh_user=root
user=mha

[server1]    #master节点
hostname=192.168.100.150
port=3306

[server2]    #salve1
hostname=192.168.100.160
port=3306
candidate_master=1
check_repl_delay=0

[server3]    #slave2
hostname=192.168.100.170
port=3306

################配置项解释
[server default]
manager_log=/var/log/masterha/app1/manager.log  'manager日志'
manager_workdir=/var/log/masterha/app1    'manager工作目录'
master_binlog_dir=/usr/local/mysql/data    '#master保存binlog的位置，这里的路径要与master里配置的binlog的路径一直，以便mha能找到,这里也对应编译安装数据库的数据目录'
master_ip_failover_script=/usr/local/bin/master_ip_failover  '设置自动failover时候的切换脚本，也就是上边的那个脚本'
master_ip_online_change_script=/usr/local/bin/master_ip_online_change '设置手动切换时候的切换脚本'
password=manager  '#设置mysql中root用户的密码，这个密码是前文中创建监控用户的那个密码'
ping_interval=1  '设置监控主库，发送ping包的时间间隔，默认是3秒，尝试三次没有回应的时候自动进行rail'
remote_workdir=/tmp  '设置远端mysql在发生切换时binlog的保存位置'
repl_password=123  '设置复制用户的密码'
repl_user=myslave   '设置复制用户的用户'
secondary_check_script=/usr/local/bin/masterha_secondary_check -s 192.168.179.124 -s 192.168.179.125 '设置从服务器的地址'
shutdown_script="" '设置故障发生后关闭故障主机脚本（该脚本的主要作用是关闭主机放在发生脑裂,这里没有使用)'
ssh_user=root  '设置ssh的登录用户名'
user=mha

[server1]
hostname=192.168.179.123
port=3306

[server2]
candidate_master=1  '设置为候选master'
hostname=192.168.179.124
check_repl_delay=0  '默认情况下如果一个slave落后master 100M的relay logs 的话，mha将不会选择该slave作为一个新的master'
port=3306

[server3]
hostname=192.168.179.125
port=3306

5).测试ssh无密码认证，如果正常最后会输出successfully，如下所示。
[root@mha-manager~]# masterha_check_ssh -conf=/etc/masterha/app1.cnf
Fri Aug 28 12:42:32 2020 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Aug 28 12:42:32 2020 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Fri Aug 28 12:42:32 2020 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Fri Aug 28 12:42:32 2020 - [info] Starting SSH connection tests..
Fri Aug 28 12:42:38 2020 - [debug] 
Fri Aug 28 12:42:32 2020 - [debug]  Connecting via SSH from root@192.168.100.150(192.168.100.150:22) to root@192.168.100.160(192.168.100.160:22)..
Fri Aug 28 12:42:33 2020 - [debug]   ok.
Fri Aug 28 12:42:33 2020 - [debug]  Connecting via SSH from root@192.168.100.150(192.168.100.150:22) to root@192.168.100.170(192.168.100.170:22)..
Fri Aug 28 12:42:38 2020 - [debug]   ok.
Fri Aug 28 12:42:39 2020 - [debug] 
Fri Aug 28 12:42:32 2020 - [debug]  Connecting via SSH from root@192.168.100.160(192.168.100.160:22) to root@192.168.100.150(192.168.100.150:22)..
Fri Aug 28 12:42:33 2020 - [debug]   ok.
Fri Aug 28 12:42:33 2020 - [debug]  Connecting via SSH from root@192.168.100.160(192.168.100.160:22) to root@192.168.100.170(192.168.100.170:22)..
Fri Aug 28 12:42:39 2020 - [debug]   ok.
Fri Aug 28 12:42:39 2020 - [debug] 
Fri Aug 28 12:42:33 2020 - [debug]  Connecting via SSH from root@192.168.100.170(192.168.100.170:22) to root@192.168.100.150(192.168.100.150:22)..
Fri Aug 28 12:42:34 2020 - [debug]   ok.
Fri Aug 28 12:42:34 2020 - [debug]  Connecting via SSH from root@192.168.100.170(192.168.100.170:22) to root@192.168.100.160(192.168.100.160:22)..
Fri Aug 28 12:42:39 2020 - [debug]   ok.
Fri Aug 28 12:42:39 2020 - [info] All SSH connection tests passed successfully.

//这一步有问题说明秘钥分发问题

[root@mha-manager ~]# masterha_check_repl -conf=/etc/masterha/app1.cnf

Fri Aug 28 16:20:51 2020 - [info] Checking replication health on 192.168.100.160..
Fri Aug 28 16:20:51 2020 - [info]  ok.
Fri Aug 28 16:20:51 2020 - [info] Checking replication health on 192.168.100.170..
Fri Aug 28 16:20:51 2020 - [info]  ok.
Fri Aug 28 16:20:51 2020 - [info] Checking master_ip_failover_script status:
Fri Aug 28 16:20:51 2020 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.100.150 --orig_master_ip=192.168.100.150 --orig_master_port=3306 

IN SCRIPT TEST====/sbin/ifconfig ens33:1 down==/sbin/ifconfig ens33:1 192.168.100.200===

Checking the Status of the script.. OK 
Fri Aug 28 16:20:51 2020 - [info]  OK.
Fri Aug 28 16:20:51 2020 - [warning] shutdown_script is not defined.
Fri Aug 28 16:20:51 2020 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.

//注意:第一次配置需要去master上手动开启虚拟IP
[root@Mysql1 ~]# /sbin/ifconfig ens33:1 192.168.100.200/24

7.启动MHA

[root@mha-manager ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &

--remove_dead_master_conf该参数代表当发生主从切换后，老的主库的ip将会从配置文件中移除
--manger_log日志存放位置
--ignore_last_failover在缺省情况下，如果MHA 检测到连续发生宕机，且两次宕机间隔不足8小时的话，则不会进行Failover，之所以这样限制是为了避免ping-pong效应。
该参数代表忽略上次MHA触发切换产生的文件，默认情况下，MHA 发生切换后会在日志记目录，也就是上面设置的日志app1.failover.complete文件，
下次再次切换的时候如果发现该目录下存在该文件将不允许触发切换，除非在第一次切换后收到删除该文件，为了方便，这里设置为-ignore_last_failover。
隔不足8小时的话，则不会进行Failover，之所以这样限制是为了避免ping-pong效应。该
参数代表忽略上次MHA触发切换产生的文件，默认情况下，MHA 发生切换后会在日志记
目录，也就是上面设置的日志app1.failover.complete文件，下次再次切换的时候如果发现
该目录下存在该文件将不允许触发切换，除非在第一次切换后收到删除该文件，为了方便，
这里设置为-ignore_last_failover.

8.查看MHA状态，可以看到当前的master是Mysql1节点。
[root@mha-manager~]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 (pid:70465) is running(0:PING_OK), master:192.168.100.150

9.查看MHA日志，也以看到当前的master是192.168.100.150，如下所示。
[root@mha-manager ~]# cat /var/log/masterha/app1/manager.log
.......
IN SCRIPT TEST====/sbin/ifconfig ens33:1 down==/sbin/ifconfig ens33:1 192.168.100.200===
.........

10.查看 Mysql1的VIP地址192.168.100.200是否存在?，这个VIР地址不会因为
manager 节点停止MHA 服务而消失。
[root@mysql1 mha4mysql-node-0.57]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.150  netmask 255.255.255.0  broadcast 192.168.100.255
        inet6 fe80::7264:7578:de4e:190  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::9433:6cdb:ee1d:7b3d  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:17:9c:75  txqueuelen 1000  (Ethernet)
        RX packets 101492  bytes 130137031 (124.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 43150  bytes 7281255 (6.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.200  netmask 255.255.255.0  broadcast 192.168.100.255
        ether 00:0c:29:17:9c:75  txqueuelen 1000  (Ethernet)

------------------------------------验证------------------------------

[root@mha-manager ~]# tailf /var/log/masterha/app1/manager.log
//启用监控观察日志记录
[root@MysqI1~]# pkill -9 mysql /查看master变化
[root@mha-manager ~]# tailf /var/log/masterha/app1/manager.log
Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.100.150(192.168.100.150:3306)
The latest slave 192.168.100.160(192.168.100.160:3306) has all relay logs for recovery.
Selected 192.168.100.160(192.168.100.160:3306) as a new master.
192.168.100.160(192.168.100.160:3306): OK: Applying all logs succeeded.
192.168.100.160(192.168.100.160:3306): OK: Activated master IP address.
192.168.100.170(192.168.100.170:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.100.170(192.168.100.170:3306): OK: Applying all logs succeeded. Slave started, replicating from 192.168.100.160(192.168.100.160:3306)
192.168.100.160(192.168.100.160:3306): Resetting slave info succeeded.
Master failover to 192.168.100.160(192.168.100.160:3306) completed successfully.  'master已经成功切换到备用160上面'
[root@mha-manager bin]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 (pid:72487) is running(0:PING_OK), master:192.168.100.160
[root@mha-manager bin]# 

[root@mysql2 mha4mysql-node-0.57]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.160  netmask 255.255.255.0  broadcast 192.168.100.255
        inet6 fe80::1bbb:ba05:a579:42dd  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:9f:ee:79  txqueuelen 1000  (Ethernet)
        RX packets 100963  bytes 130156228 (124.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 40046  bytes 6979445 (6.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.200  netmask 255.255.255.0  broadcast 192.168.100.255
        ether 00:0c:29:9f:ee:79  txqueuelen 1000  (Ethernet)
虚拟地址也切换到mysql2
故障模拟:
在主库上:
pkill mysqld
可以看到从库的状态，其中之一肯定有切换到主库的
切换备选主库的算法:
1.一般判断从库的是从(position/GTID）判断优劣，数据有差异，最接近于master的slave，成为备选主。
2.数据一致的情况下，按照配置文件顺序，选择备选主库。
3.设定有权重(candidate_master=1)，按照权重强制指定备选主。
1）默认情况下如果一个slave落后master 100M的relay logs的话，即使有权重，也会失效。
2）如果check_repl_delay=0的话，即使落后很多日志，也强制选择其为备选主。

故障修复步骤:
1.修复db
/etc/init.d/mysqld start
2修复主从
>change master to master_host='192.168.100.150',master_port=3306,master_auto_position=1,master_user='mha',master_passwd='manager',master_log_file='master-bin.000001'
change master to master_host='192.168.100.150',master_user='myslave',master_password='123',master_log_file='master-bin.000001',master_log_pos=1294;
>start slave;
3.修改配置文件(再把这个记录添加进去，因为它检测掉失效时候会自动消失)
vi /etc/masterha/app1.cnf
[server1]
hostname=192.168.195.129
port=3306
4.启动manager(在manager那台机器上)
nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --gnore _last failover < /dev/null

dos2unix /usr/local/bin/master_ip_failover解决中英字不兼容报错的问题