优点：
 找出同步最成功的一台从服务器（也就是与主服务器数据最接近的那台从服务器）自动切换成主服务器。

 如果主机还能够访问，从主服务器上找回最新从机与主机间的数据差异。

 在每一台从服务器上操作，确定他们缺少哪些events，并分别进行补充。

 将最新的一台从服务器提升为主服务器后，将其它从服务器重新指向新的主服务器。
缺点：
 当群集内的数据库进行故障转移时，对外提供服务的虚拟IP也进行转移。

 MHA管理进程需要以后台守护进程的方式运行，并有监控机制保证MHA管理进程的正常运行。

 有监控机制保证当主机出现故障时，MHA能确定进行成功的Failover。

 当故障主机恢复后，能重新回到群集中，并成为新的Slave，自动实现重新同步。

 由于主机和从机上备份策略不同，进行故障转移后，自动调整cron中的调度（例如全备份）。

https://code.google.com/p/mysql-master-ha/downloads/list?q=label:Featured
https://code.google.com/p/mysql-master-ha/wiki/Downloads?tm=2   #（5.6）

wget https://mysql-master-ha.googlecode.com/files/mha4mysql-manager-0.55.tar.gz

wget https://mysql-master-ha.googlecode.com/files/mha4mysql-node-0.54.tar.gz

https://code.google.com/p/mysql-master-ha/wiki/Installation#Installing_MHA_Manager

Manager工具:
- masterha_check_ssh : 检查MHA的SSH配置。

- masterha_check_repl : 检查MySQL复制。

- masterha_manager : 启动MHA。

- masterha_check_status : 检测当前MHA运行状态。

- masterha_master_monitor : 监测master是否宕机。

- masterha_master_switch : 控制故障转移(自动或手动)。

- masterha_conf_host : 添加或删除配置的server信息。

Node工具:
- save_binary_logs : 保存和复制master的二进制日志。

- apply_diff_relay_logs : 识别差异的中继日志事件并应用于其它slave。

- filter_mysqlbinlog : 去除不必要的ROLLBACK事件(MHA已不再使用这个工具)。

- purge_relay_logs : 清除中继日志(不会阻塞SQL线程)。

  ## Install dependent Perl modules

  # Config::Tiny

  ## perl -MCPAN -e "install Config::Tiny"

  # Log::Dispatch

  ## perl -MCPAN -e "install Log::Dispatch"

  # Parallel::ForkManager

  ## perl -MCPAN -e "install Parallel::ForkManager"

  ## Installing MHA Manager

  $ tar -zxf mha4mysql-manager-X.Y.tar.gz

  $ perl Makefile.PL
*** Module::AutoInstall version 1.03

*** Checking for Perl dependencies...

[Core Features]

- DBI                   ...loaded. (1.616)

- DBD::mysql            ...loaded. (4.020)

- Time::HiRes           ...loaded. (1.972101)

- Config::Tiny          ...loaded. (2.20)

- Log::Dispatch         ...loaded. (2.41)

- Parallel::ForkManager ...loaded. (1.06)

- MHA::NodeConst        ...loaded. (0.54)  先装Node，这里才能通过。

*** Module::AutoInstall configuration finished.

Checking if your kit is complete...

Looks good

Generating a Unix-style Makefile

Writing Makefile for mha4mysql::manager

Writing MYMETA.yml and MYMETA.json

  $ make

  $ sudo make install

## Install DBD::mysql if not installed

  $ tar -zxf mha4mysql-node-X.Y.tar.gz

  $ perl Makefile.PL
*** Module::AutoInstall version 1.03

*** Checking for Perl dependencies...

[Core Features]

- DBI        ...loaded. (1.616)

- DBD::mysql ...loaded. (4.020)

*** Module::AutoInstall configuration finished.

Checking if your kit is complete...

Looks good

Generating a Unix-style Makefile

Writing Makefile for mha4mysql::node

Writing MYMETA.yml and MYMETA.json

  $ make

  $ sudo make install

sudo apt-get install libdbd-mysql-perl

export LC_ALL="zh_CN.UTF-8"

--relay-log-purge={|}

禁用或启用不再需要中继日志时是否自动清空它们。默认值为1(启用)。这是一个全局变量，可以用SET GLOBAL Relay_log_purge动态更改。

[]

用ssh-keygen创建公钥，一直默认回车，最后会在.ssh/下面生成id_rsa.pub
ssh-keygen -t rsa


把id_rsa.pub 文件复制到另外2台机子并导入公钥：
cat id_rsa.pub >> /root/.ssh/authorized_keys 

当然也可以直接复制里面的内容再粘帖到authorized_keys中。 这样204可以直接访问到其他2台机子了

ssh 192.168.200.201

ssh 192.168.200.202 

同理在另外2台机子上面操作，让他们之间实现无密码登入。

要是root登陆的话，需要修改ssh的配置文件：

vi /etc/ssh/sshd_config  中的 PermitRootLogin ，设置为YES，再重启SSH。

注意：Manager 要是装到某一台MySQL上，则需要自己和自己无密码登入：ssh 192.168.200.204，单独到一台服务器则不需要。

mkdir -p /usr/local/masterha/app1        #创建数据文件目录

mkdir /etc/masterha                      #创建配置文件目录

安装时解压出来的samples中有2个文件:
conf 是存放配置文件的模块
scripts是存放运行时的一些命令，可以把这几个名字移到/user/bin 中。用户下面灰色背景的几个参数中。

修改后的配置文件：[vi /etc/masterha/app1.cnf]

[server default]

user=root                                 #mysql用戶名，
#需要的权限：Super,select,create,insert,update,delete,drop,reload

password=                           #mysql密码

ssh_user=root                    #ssh免密钥登录的帐号名

repl_user=rep                    #mysql复制帐号

repl_password=             #mysql复制密码

ping_interval=                  #ping间隔，用来检测master是否正常，默认是3秒，尝试三次没有回应的时候自动进行failover

manager_workdir=/usr/local/masterha/app1       #数据目录,主要该目录的权限，需要有创建的权限

manager_log=/usr/local/masterha/manager.log    #日志文件

remote_workdir=/usr/local/masterha/app1        #另外2台机子在运行时候需要创建的目录，注意ssh-keygen帐号的权限问题

#master_ip_failover_script＝/usr/bin/master_ip_failover

#master_ip_online_change_script=/usr/bin/master_ip_online_change

#shutdown_script=/usr/bin/power_manager

#report_script=/usr/bin/send_report

#secondary_check_script= masterha_secondary_check -s remote_host1 -s remote_host2

[server1] 
hostname=192.168.200.201 
master_binlog_dir=/var/lib/mysql #binlog目录 
candidate_master= #master机宕掉后,优先启用这台作为新master

check_repl_delay=0            #默认情况下如果一个slave落后master 100M的relay logs的话，MHA将不会选择该slave作为一个新的master，因为对于这个slave的恢复需要花费很长时间，通过设置check_repl_delay=0,MHA触发切换在选择一个新的master的时候将会忽略复制延时，这个参数对于设置了candidate_master=1的主机非常有用，因为这个候选主在切换的过程中一定是新的master

[server2] 
hostname=192.168.200.202 
master_binlog_dir=/var/log/mysql 
candidate_master= 
[server3] hostname=192.168.200.204 
master_binlog_dir=/var/log/mysql 
#candidate_master=

root@db4:/usr/local/masterha# masterha_check_ssh --conf=/etc/masterha/app1.cnf

Wed Jun  ::  - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.

Wed Jun  ::  - [info] Reading application default configurations from /etc/masterha/app1.cnf..

Wed Jun  ::  - [info] Reading server configurations from /etc/masterha/app1.cnf..

Wed Jun  ::  - [info] Starting SSH connection tests..

Wed Jun  ::  - [debug]

Wed Jun  ::  - [debug]  Connecting via SSH from root@192.168.200.201(192.168.200.201:) to root@192.168.200.202(192.168.200.202:)..

Wed Jun  ::  - [debug]   ok.

Wed Jun  ::  - [debug]  Connecting via SSH from root@192.168.200.201(192.168.200.201:) to root@192.168.200.204(192.168.200.204:)..

Wed Jun  ::  - [debug]   ok.

Wed Jun  ::  - [debug]

Wed Jun  ::  - [debug]  Connecting via SSH from root@192.168.200.202(192.168.200.202:) to root@192.168.200.201(192.168.200.201:)..

Wed Jun  ::  - [debug]   ok.

Wed Jun  ::  - [debug]  Connecting via SSH from root@192.168.200.202(192.168.200.202:) to root@192.168.200.204(192.168.200.204:)..

Wed Jun  ::  - [debug]   ok.

Wed Jun  ::  - [debug]

Wed Jun  ::  - [debug]  Connecting via SSH from root@192.168.200.204(192.168.200.204:) to root@192.168.200.201(192.168.200.201:)..

Wed Jun  ::  - [debug]   ok.

Wed Jun  ::  - [debug]  Connecting via SSH from root@192.168.200.204(192.168.200.204:) to root@192.168.200.202(192.168.200.202:)..

Wed Jun  ::  - [debug]   ok.

Wed Jun  ::  - [info] All SSH connection tests passed successfully.

root@db4:/usr/local/masterha# masterha_check_repl --conf=/etc/masterha/app1.cnf

...

...

192.168.200.201 (current master)

 +--192.168.200.202

 +--192.168.200.204

Wed Jun  ::  - [info] Checking replication health on 192.168.200.202..

Wed Jun  ::  - [info]  ok.

Wed Jun  ::  - [info] Checking replication health on 192.168.200.204..

Wed Jun  ::  - [info]  ok.

Wed Jun  ::  - [warning] master_ip_failover_script is not defined.

Wed Jun  ::  - [warning] shutdown_script is not defined.

Wed Jun  ::  - [info] Got exit code  (Not master dead).

MySQL Replication Health is OK.

root@db4:~# nohup masterha_manager --conf=/etc/masterha/app1.cnf > /tmp/mha_manager.log >&

查看日志文件（/masterha/app1/manager.log）

    Cleaning up test file(s).. done.

Wed Jun  ::  - [info] Slaves settings check done.

Wed Jun  ::  - [info]

192.168.200.201 (current master)

 +--192.168.200.202

 +--192.168.200.204

Wed Jun  ::  - [warning] master_ip_failover_script is not defined.

Wed Jun  ::  - [warning] shutdown_script is not defined.

Wed Jun  ::  - [info] Set master ping interval  seconds.

Wed Jun  ::  - [info] Set secondary check script: masterha_secondary_check -s remote_host1 -s remote_host2

Wed Jun  ::  - [info] Starting ping health check on 192.168.200.201(192.168.200.201:)..

Wed Jun  ::  - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..

开启程序

nohup masterha_manager --ignore_last_failover --remove_dead_master_conf --conf=/etc/masterha/app1.cnf > /tmp/mha_manager.log >&

+++

故障切换

From:

192.168.200.202 (current master)

 +--192.168.200.201

 +--192.168.200.204

To:

192.168.200.201 (new master)

 +--192.168.200.204

+++

不先处理故障的机子，可直接运行程序，不加该参数会直接报错:

[error][/usr/local/share/perl/5.14./MHA/ServerManager.pm, ln443]  Server 192.168.200.201(192.168.200.201:) is dead, but must be alive! Check server settings.

nohup masterha_manager --ignore_last_failover --remove_dead_master_conf --conf=/etc/masterha/app1.cnf > /tmp/mha_manager.log >&

可以正常运行，但是配置文件的[server2]选项组被删除

nohup masterha_manager --ignore_last_failover --conf=/etc/masterha/app1.cnf > /tmp/mha_manager.log >&

...

...

Wed Jun  ::  - [info] Reading server configurations from /etc/masterha/app1.cnf..

Wed Jun  ::  - [info] Dead Servers:

Wed Jun  ::  - [info]   192.168.200.201(192.168.200.201:)

Wed Jun  ::  - [info] Alive Servers:

Wed Jun  ::  - [info]   192.168.200.202(192.168.200.202:)

Wed Jun  ::  - [info]   192.168.200.204(192.168.200.204:)

Wed Jun  ::  - [info] Alive Slaves:

Wed Jun  ::  - [info]   192.168.200.202(192.168.200.202:)  Version=5.5.-0ubuntu0.12.04.-log (oldest major version between slaves) log-bin:enabled

Wed Jun  ::  - [info]     Replicating from 192.168.200.201(192.168.200.201:)

....

....

Wed Jun  ::  - [info] Master is down!

Wed Jun  ::  - [info] Starting master failover.

....

....

进行5个阶段

Creating /usr/local/masterha/app1 if not exists..    ok.

...

Dumping binlog format description event, from position  to .. ok.

Wed Jun  ::  - [info] scp from root@192.168.200.201:/usr/local/masterha/app1/saved_master_binlog_from_192.168.200.201_3306_20140625234550.binlog to local:/usr/local/masterha/app1/saved_master_binlog_from_192.168.200.201_3306_20140625234550.binlog succeeded.

...

Wed Jun  ::  - [info] New master is 192.168.200.202(192.168.200.202:)

Wed Jun  ::  - [info] Starting master failover..

Wed Jun  ::  - [info]

From:

192.168.200.201 (current master)

 +--192.168.200.202

 +--192.168.200.204

To:

192.168.200.202 (new master)

 +--192.168.200.204

...

Applying log files succeeded.

Wed Jun  ::  - [info]  All relay logs were successfully applied.

Wed Jun  ::  - [info] Getting new master's binlog name and position..

Wed Jun  ::  - [info]  mysql-bin.:

Wed Jun  ::  - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.200.202', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=107, MASTER_USER='rep', MASTER_PASSWORD='xxx';

Wed Jun  ::  - [info] ** Finished master recovery successfully.

...

Wed Jun  ::  - [info]  Executed CHANGE MASTER.

Wed Jun  ::  - [info]  Slave started.

Wed Jun  ::  - [info] All new slave servers recovered successfully.

...

----- Failover Report -----

app1: MySQL Master failover 192.168.200.201 to 192.168.200.202 succeeded

Master 192.168.200.201 is down!

Check MHA Manager logs at db4:/usr/local/masterha/manager.log for details.

Started automated(non-interactive) failover.

The latest slave 192.168.200.202(192.168.200.202:) has all relay logs for recovery.

Selected 192.168.200.202 as a new master.

192.168.200.202: OK: Applying all logs succeeded.

192.168.200.204: This host has the latest relay log events.

Generating relay diff files from the latest slave succeeded.

192.168.200.204: OK: Applying all logs succeeded. Slave started, replicating from 192.168.200.202.

192.168.200.202: Resetting slave info succeeded.

Master failover to 192.168.200.202(192.168.200.202:) completed successfully.

CHANGE MASTER TO MASTER_HOST='192.168.200.202', MASTER_PORT=, MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=, MASTER_USER='rep', MASTER_PASSWORD='xxx';

一旦发生切换管理进程（Manager）将会退出，无法进行再次测试，需将故障数据库解决掉之后，重新change加入到MHA环境中来，并且要保证app1.failover.complete不存在或则加上--ignore_last_failover参数忽略，才能再次开启管理进程。

root@db4:/usr/local/masterha# masterha_master_switch --master_state=alive --conf=/etc/masterha/app1.cnf

Thu Jun  ::  - [info] MHA::MasterRotate version 0.55.

Thu Jun  ::  - [info] Starting online master switch..

...

It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.200.202(192.168.200.202:)? (YES/no): yes

...
From:
192.168.200.202 (current master)

 +--192.168.200.201

 +--192.168.200.204

To:202消失

192.168.200.201 (new master)

 +--192.168.200.204

Starting master switch from 192.168.200.202(192.168.200.202:3306) to 192.168.200.201(192.168.200.201:3306)? (yes/NO): yes

...

master_ip_online_change_script is not defined. If you do not disable writes on the current master manually, applications keep writing on the current master. Is it ok to proceed? (yes/NO): yes

...

Thu Jun  ::  - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.200.201', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin-3306.000001', MASTER_LOG_POS=107, MASTER_USER='rep', MASTER_PASSWORD='xxx';

...

Thu Jun  ::  - [info] All new slave servers switched successfully.

Thu Jun  ::  - [info]  192.168.200.201: Resetting slave info succeeded.

Thu Jun  ::  - [info] Switching master to 192.168.200.201(192.168.200.201:) completed successfully.

masterha_master_switch --master_state=alive --conf=/etc/masterha/app1.cnf --new_master_host=192.168.200.201

root@db4:/usr/local/masterha# masterha_check_status --conf=/etc/masterha/app1.cnf

app1 is stopped(:NOT_RUNNING).

root@db4:/usr/local/masterha# masterha_check_status --conf=/etc/masterha/app1.cnf

app1 monitoring program is now on initialization phase(:INITIALIZING_MONITOR). Wait for a while and try checking again.

root@db4:/usr/local/masterha# masterha_check_status --conf=/etc/masterha/app1.cnf

app1 (pid:29232) is running(0:PING_OK), master:192.168.200.201

root@db4:/usr/local/masterha# masterha_stop --conf=/etc/masterha/app1.cnf

Stopped app1 successfully.

设置：

ifconfig eth0: 192.168.100.xxx

移除：

ifconfig eth0: down

#!/usr/bin/env perl

#  Copyright (C)  DeNA Co.,Ltd.

#

#  This program is free software; you can redistribute it and/or modify

#  it under the terms of the GNU General Public License as published by

#  the Free Software Foundation; either version  of the License, or

#  (at your option) any later version.

#

#  This program is distributed in the hope that it will be useful,

#  but WITHOUT ANY WARRANTY; without even the implied warranty of

#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

#  GNU General Public License for more details.

#

#  You should have received a copy of the GNU General Public License

#   along with this program; if not, write to the Free Software

#  Foundation, Inc.,

#   Franklin Street, Fifth Floor, Boston, MA  -  USA

## Note: This is a sample script and is not complete. Modify the script based on your environment.

use strict;

use warnings FATAL => 'all';

use Getopt::Long;

my (

    $command,          $ssh_user,        $orig_master_host, $orig_master_ip,

    $orig_master_port, $new_master_host, $new_master_ip,    $new_master_port

);

my $vip = '192.168.200.50/24';

my $key = '0';

my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip";

my $ssh_stop_vip  = "/sbin/ifconfig eth0:$key down";

my $ssh_Bcast_arp = "/usr/bin/arping -c 3 -A 192.168.200.50";   #ARP回复模式，更新邻居。要是不加则服务器会自动等到vip缓存失效，期间VIP会有一定时间的不可用。

GetOptions(

    'command=s'          => \$command,

    'ssh_user=s'         => \$ssh_user,

    'orig_master_host=s' => \$orig_master_host,

    'orig_master_ip=s'   => \$orig_master_ip,

    'orig_master_port=i' => \$orig_master_port,

    'new_master_host=s'  => \$new_master_host,

    'new_master_ip=s'    => \$new_master_ip,

    'new_master_port=i'  => \$new_master_port,

);

exit &main();

sub main {

    print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

    if ( $command eq "stop" || $command eq "stopssh" ) {

        my $exit_code = ;

        eval {

            print "Disabling the VIP on old master: $orig_master_host \n";

            &stop_vip();

            $exit_code = ;

        };

        if ($@) {

            warn "Got Error: $@\n";

            exit $exit_code;

        }

        exit $exit_code;

    }

    elsif ( $command eq "start" ) {

        my $exit_code = ;

        eval {

            print "Enabling the VIP - $vip on the new master - $new_master_host \n";

            &start_vip();

            &start_arp();

            $exit_code = ;

        };

        if ($@) {

            warn $@;

            exit $exit_code;

        }

        exit $exit_code;

    }

    elsif ( $command eq "status" ) {

        print "Checking the Status of the script.. OK \n";

        exit ;

    }

    else {

        &usage();

        exit ;

    }

}

sub start_vip() {

    `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;

}

sub start_arp() {

    `ssh $ssh_user\@$new_master_host \" $ssh_Bcast_arp \"`;

}

sub stop_vip() {

    `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;

}

sub usage {

  print

"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";

}

apt-get install iputils-arping

hash -r  更新缓存路径

#!/usr/bin/env python

#-*- encoding:utf-8 -*-

#-------------------------------------------------------------------------------

# Name:        send_report.py

# Author:      zhoujy

#----------------------------------------------

import os

import sys

import time

import datetime

import smtplib

import subprocess

import fileinput

import getopt

from email.mime.text import MIMEText

from email.mime.multipart import MIMEMultipart

from email.Utils import COMMASPACE, formatdate

reload(sys)

sys.setdefaultencoding('utf8')

def send_mail(to, subject, text, from_mail, server="localhost"):

    message = MIMEMultipart()

    message['From'] = from_mail

    message['To'] = COMMASPACE.join(to)

    message['Date'] = formatdate(localtime=True)

    message['Subject'] = subject

    message.attach(MIMEText(text,_charset='utf-8'))

    smtp = smtplib.SMTP(server)

    smtp.sendmail(from_mail, to, message.as_string())

    smtp.close()

if __name__ == "__main__":

    opts,args = getopt.getopt(sys.argv[1:],"h",["orig_master_host=","new_master_host=","new_slave_hosts=","conf=","subject=","body=","app_vip=","new_master_ssh_port=","ssh_user="])

#    print opts,args

    for lines in opts:

        key,values = lines

        if key == '--orig_master_host':

            orig_master_host = values

        if key == '--new_master_host':

            new_master_host = values

        if key == '--new_slave_hosts':

            new_slave_hosts = values

        if key == '--subject':

            subject = values

        if key == '--body':

            body = values

#    text = sys.stdin.read()

    mail_list = ['zjy@xxx.com']

    send_mail(mail_list, subject.encode("utf8"), body, "MHA_Monitor@smtp.dxy.cn", server="192.168.220.251")

sysbench --test=oltp --mysql-table-engine=innodb --mysql-host=192.168.200.50 --mysql-db=mha_test --oltp-table-size= --mysql-user=zjy --mysql-password=# prepare

stop slave io_thread;

sysbench --num-threads= --test=oltp --mysql-table-engine=innodb --mysql-host=192.168.200.50 --mysql-db=mha_test --oltp-table-size= --mysql-user=zjy --mysql-password=aaaaa# run

start slave io_thread;

/etc/init.d/mysql stop