自动failover
slave1:
stop slave io_thread slave2
stop slave io_thread server1:
create database sbtest;
/usr/local/bin/sysbench --test=/root/sysbench-0.5/sysbench/tests/db/oltp.lua --oltp-table-size=10000 --oltp-tables-count=1 --mysql-user=root --mysql-password=123 -- mysql-
host=10.24.220.232 --mysql-port=3306 --num-threads=4 --max-requests=0 --max-time=30 --report-interval=3 prepare
/usr/local/bin/sysbench --test=/root/sysbench-0.5/sysbench/tests/db/oltp.lua --oltp-table-size=10000 --oltp-tables-count=1 --mysql-user=root --mysql-password=123 -- mysql-
host=10.24.220.232 --mysql-port=3306 --num-threads=4 --max-requests=0 --max-time=30 --report-interval=3 run
slave2:
start slave io_thread slave1
start slave io_thread
server1
service mysql.server stop
Sun May :: - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun May :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Sun May :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Sun May :: - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun May :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Sun May :: - [info] Reading server configuration from /etc/masterha/app1.conf..
:):
Sun May :: - [debug] Connected to: slave1(10.24.220.70:), user=root
Sun May :: - [debug] Number of slave worker threads on host slave1(10.24.220.70:):
Sun May :: - [debug] Connected to: slave2(10.169.214.33:), user=root
Sun May :: - [debug] Number of slave worker threads on host slave2(10.169.214.33:):
Sun May :: - [debug] Comparing MySQL versions..
Sun May :: - [debug] Comparing MySQL versions done.
Sun May :: - [debug] Connecting to servers done.
Sun May :: - [info] GTID failover mode =
Sun May :: - [info] Dead Servers:
Sun May :: - [info] Alive Servers:
Sun May :: - [info] server1(10.24.220.232:)
Sun May :: - [info] slave1(10.24.220.70:)
Sun May :: - [info] slave2(10.169.214.33:)
Sun May :: - [info] Alive Slaves:
Sun May :: - [info] slave1(10.24.220.70:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Sun May :: - [info] GTID ON
Sun May :: - [debug] Relay log info repository: FILE
Sun May :: - [info] Replicating from 10.24.220.232(10.24.220.232:)
Sun May :: - [info] Primary candidate for the new Master (candidate_master is set)
Sun May :: - [info] slave2(10.169.214.33:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Sun May :: - [info] GTID ON
Sun May :: - [debug] Relay log info repository: FILE
Sun May :: - [info] Replicating from 10.24.220.232(10.24.220.232:)
Sun May :: - [info] Not candidate for the new Master (no_master is set)
Sun May :: - [info] Current Alive Master: server1(10.24.220.232:)
Sun May :: - [info] Checking slave configurations..
Sun May :: - [info] Checking replication filtering settings..
Sun May :: - [info] binlog_do_db= , binlog_ignore_db=
Sun May :: - [info] Replication filtering check ok.
Sun May :: - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Sun May :: - [info] Checking SSH publickey authentication settings on the current master..
Sun May :: - [debug] SSH connection test to server1, option -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o BatchMode=yes -o ConnectTimeout=, timeout
Sun May :: - [info] HealthCheck: SSH to server1 is reachable.
Sun May :: - [info]
server1(10.24.220.232:) (current master)
+--slave1(10.24.220.70:)
+--slave2(10.169.214.33:) Sun May :: - [info] Checking master_ip_failover_script status:
Sun May :: - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=server1 --orig_master_ip=10.24.220.232 --orig_master_port= IN SCRIPT TEST====/sbin/ifconfig eth0: down==/sbin/ifconfig eth0: 10.24.220.100/=== Checking the Status of the script.. OK
Sun May :: - [info] OK.
Sun May :: - [warning] shutdown_script is not defined.
Sun May :: - [debug] Disconnected from server1(10.24.220.232:)
Sun May :: - [debug] Disconnected from slave1(10.24.220.70:)
Sun May :: - [debug] Disconnected from slave2(10.169.214.33:)
Sun May :: - [debug] SSH check command: exit
Sun May :: - [info] Set master ping interval seconds.
Sun May :: - [info] Set secondary check script: /usr/bin/masterha_secondary_check -s 10.169.214.33 -s 10.24.220.70 --user=root --master_host=10.24.220.232 --master_port=
Sun May :: - [info] Starting ping health check on server1(10.24.220.232:)..
Sun May :: - [debug] Connected on master.
Sun May :: - [debug] Set short wait_timeout on master: seconds
Sun May :: - [debug] Trying to get advisory lock..
Sun May :: - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Sun May :: - [warning] Got error on MySQL select ping: (MySQL server has gone away)
Sun May :: - [info] Executing SSH check script: exit
Sun May :: - [debug] SSH connection test to server1, option -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o BatchMode=yes -o ConnectTimeout=, timeout
Sun May :: - [info] Executing secondary network check script: /usr/bin/masterha_secondary_check -s 10.169.214.33 -s 10.24.220.70 --user=root --master_host=10.24.220.232 --master_port= --user=root --master_host=server1 --master_ip=10.24.220.232 --master_port= --master_user=root --master_password= --ping_type=SELECT
Sun May :: - [info] HealthCheck: SSH to server1 is reachable.
Monitoring server 10.169.214.33 is reachable, Master is not reachable from 10.169.214.33. OK.
Monitoring server 10.24.220.70 is reachable, Master is not reachable from 10.24.220.70. OK.
Sun May :: - [info] Master is not reachable from all other monitoring servers. Failover should start.
Sun May :: - [warning] Got error on MySQL connect: (Lost connection to MySQL server at 'reading initial communication packet', system error: )
Sun May :: - [warning] Connection failed time(s)..
Sun May :: - [warning] Got error on MySQL connect: (Lost connection to MySQL server at 'reading initial communication packet', system error: )
Sun May :: - [warning] Connection failed time(s)..
Sun May :: - [warning] Got error on MySQL connect: (Lost connection to MySQL server at 'reading initial communication packet', system error: )
Sun May :: - [warning] Connection failed time(s)..
Sun May :: - [warning] Master is not reachable from health checker!
Sun May :: - [warning] Master server1(10.24.220.232:) is not reachable!
Sun May :: - [warning] SSH is reachable.
Sun May :: - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.conf again, and trying to connect to all servers to check server status..
Sun May :: - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun May :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Sun May :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Sun May :: - [debug] Skipping connecting to dead master server1(10.24.220.232:).
Sun May :: - [debug] Connecting to servers..
Sun May :: - [debug] Connected to: slave1(10.24.220.70:), user=root
Sun May :: - [debug] Number of slave worker threads on host slave1(10.24.220.70:):
Sun May :: - [debug] Connected to: slave2(10.169.214.33:), user=root
Sun May :: - [debug] Number of slave worker threads on host slave2(10.169.214.33:):
Sun May :: - [debug] Comparing MySQL versions..
Sun May :: - [debug] Comparing MySQL versions done.
Sun May :: - [debug] Connecting to servers done.
Sun May :: - [info] GTID failover mode =
Sun May :: - [info] Dead Servers:
Sun May :: - [info] server1(10.24.220.232:)
Sun May :: - [info] Alive Servers:
Sun May :: - [info] slave1(10.24.220.70:)
Sun May :: - [info] slave2(10.169.214.33:)
Sun May :: - [info] Alive Slaves:
Sun May :: - [info] slave1(10.24.220.70:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Sun May :: - [info] GTID ON
Sun May :: - [debug] Relay log info repository: FILE
Sun May :: - [info] Replicating from 10.24.220.232(10.24.220.232:)
Sun May :: - [info] Primary candidate for the new Master (candidate_master is set)
Sun May :: - [info] slave2(10.169.214.33:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Sun May :: - [info] GTID ON
Sun May :: - [debug] Relay log info repository: FILE
Sun May :: - [info] Replicating from 10.24.220.232(10.24.220.232:)
Sun May :: - [info] Not candidate for the new Master (no_master is set)
Sun May :: - [info] Checking slave configurations..
Sun May :: - [info] Checking replication filtering settings..
Sun May :: - [info] Replication filtering check ok.
Sun May :: - [info] Master is down!
Sun May :: - [info] Terminating monitoring script.
Sun May :: - [info] Got exit code (Master dead).
Sun May :: - [info] MHA::MasterFailover version 0.56.
Sun May :: - [info] Starting master failover.
Sun May :: - [info]
Sun May :: - [info] * Phase : Configuration Check Phase..
Sun May :: - [info]
Sun May :: - [debug] Skipping connecting to dead master server1.
Sun May :: - [debug] Connecting to servers..
Sun May :: - [debug] Connected to: slave1(10.24.220.70:), user=root
Sun May :: - [debug] Number of slave worker threads on host slave1(10.24.220.70:):
Sun May :: - [debug] Connected to: slave2(10.169.214.33:), user=root
Sun May :: - [debug] Number of slave worker threads on host slave2(10.169.214.33:):
Sun May :: - [debug] Comparing MySQL versions..
Sun May :: - [debug] Comparing MySQL versions done.
Sun May :: - [debug] Connecting to servers done.
Sun May :: - [info] GTID failover mode =
Sun May :: - [info] Dead Servers:
Sun May :: - [info] server1(10.24.220.232:)
Sun May :: - [info] Checking master reachability via MySQL(double check)...
Sun May :: - [info] ok.
Sun May :: - [info] Alive Servers:
Sun May :: - [info] slave1(10.24.220.70:)
Sun May :: - [info] slave2(10.169.214.33:)
Sun May :: - [info] Alive Slaves:
Sun May :: - [info] slave1(10.24.220.70:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Sun May :: - [info] GTID ON
Sun May :: - [debug] Relay log info repository: FILE
Sun May :: - [info] Replicating from 10.24.220.232(10.24.220.232:)
Sun May :: - [info] Primary candidate for the new Master (candidate_master is set)
Sun May :: - [info] slave2(10.169.214.33:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Sun May :: - [info] GTID ON
Sun May :: - [debug] Relay log info repository: FILE
Sun May :: - [info] Replicating from 10.24.220.232(10.24.220.232:)
Sun May :: - [info] Not candidate for the new Master (no_master is set)
Sun May :: - [info] Starting GTID based failover.
Sun May :: - [info]
Sun May :: - [info] ** Phase : Configuration Check Phase completed.
Sun May :: - [info]
Sun May :: - [info] * Phase : Dead Master Shutdown Phase..
Sun May :: - [info]
Sun May :: - [info] Forcing shutdown so that applications never connect to the current master..
Sun May :: - [info] Executing master IP deactivation script:
Sun May :: - [info] /usr/local/bin/master_ip_failover --orig_master_host=server1 --orig_master_ip=10.24.220.232 --orig_master_port= --command=stopssh --ssh_user=root
Sun May :: - [debug] Stopping IO thread on slave2(10.169.214.33:)..
Sun May :: - [debug] Stopping IO thread on slave1(10.24.220.70:).. IN SCRIPT TEST====/sbin/ifconfig eth0: down==/sbin/ifconfig eth0: 10.24.220.100/=== Disabling the VIP on old master: server1
Sun May :: - [debug] Stop IO thread on slave2(10.169.214.33:) done.
Sun May :: - [debug] Stop IO thread on slave1(10.24.220.70:) done.
SIOCSIFFLAGS: Cannot assign requested address
Sun May :: - [info] done.
Sun May :: - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Sun May :: - [info] * Phase : Dead Master Shutdown Phase completed.
Sun May :: - [info]
Sun May :: - [info] * Phase : Master Recovery Phase..
Sun May :: - [info]
Sun May :: - [info] * Phase 3.1: Getting Latest Slaves Phase..
Sun May :: - [info]
Sun May :: - [debug] Fetching current slave status..
Sun May :: - [debug] Fetching current slave status done.
Sun May :: - [info] The latest binary log file/position on all slaves is log.:
Sun May :: - [info] Retrieved Gtid Set: 191f7a9f-ffa2-11e5-a825-00163e00242a:-
Sun May :: - [info] Latest slaves (Slaves that received relay log files to the latest):
Sun May :: - [info] slave1(10.24.220.70:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Sun May :: - [info] GTID ON
Sun May :: - [debug] Relay log info repository: FILE
Sun May :: - [info] Replicating from 10.24.220.232(10.24.220.232:)
Sun May :: - [info] Primary candidate for the new Master (candidate_master is set)
Sun May :: - [info] slave2(10.169.214.33:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Sun May :: - [info] GTID ON
Sun May :: - [debug] Relay log info repository: FILE
Sun May :: - [info] Replicating from 10.24.220.232(10.24.220.232:)
Sun May :: - [info] Not candidate for the new Master (no_master is set)
Sun May :: - [info] The oldest binary log file/position on all slaves is log.:
Sun May :: - [info] Retrieved Gtid Set: 191f7a9f-ffa2-11e5-a825-00163e00242a:-
Sun May :: - [info] Oldest slaves:
Sun May :: - [info] slave1(10.24.220.70:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Sun May :: - [info] GTID ON
Sun May :: - [debug] Relay log info repository: FILE
Sun May :: - [info] Replicating from 10.24.220.232(10.24.220.232:)
Sun May :: - [info] Primary candidate for the new Master (candidate_master is set)
Sun May :: - [info] slave2(10.169.214.33:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Sun May :: - [info] GTID ON
Sun May :: - [debug] Relay log info repository: FILE
Sun May :: - [info] Replicating from 10.24.220.232(10.24.220.232:)
Sun May :: - [info] Not candidate for the new Master (no_master is set)
Sun May :: - [info]
Sun May :: - [info] * Phase 3.3: Determining New Master Phase..
Sun May :: - [info]
Sun May :: - [info] Searching new master from slaves..
Sun May :: - [info] Candidate masters from the configuration file:
Sun May :: - [info] slave1(10.24.220.70:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Sun May :: - [info] GTID ON
Sun May :: - [debug] Relay log info repository: FILE
Sun May :: - [info] Replicating from 10.24.220.232(10.24.220.232:)
Sun May :: - [info] Primary candidate for the new Master (candidate_master is set)
Sun May :: - [info] Non-candidate masters:
Sun May :: - [info] slave2(10.169.214.33:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Sun May :: - [info] GTID ON
Sun May :: - [debug] Relay log info repository: FILE
Sun May :: - [info] Replicating from 10.24.220.232(10.24.220.232:)
Sun May :: - [info] Not candidate for the new Master (no_master is set)
Sun May :: - [info] Searching from candidate_master slaves which have received the latest relay log events..
Sun May :: - [info] New master is slave1(10.24.220.70:)
Sun May :: - [info] Starting master failover..
Sun May :: - [info]
From:
server1(10.24.220.232:) (current master)
+--slave1(10.24.220.70:)
+--slave2(10.169.214.33:) To:
slave1(10.24.220.70:) (new master)
+--slave2(10.169.214.33:)
Sun May :: - [info]
Sun May :: - [info] * Phase 3.3: New Master Recovery Phase..
Sun May :: - [info]
Sun May :: - [info] Waiting all logs to be applied..
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [debug] Sql Thread Done: , Worker Thread done: , Ended workers:
Sun May :: - [info] done.
Sun May :: - [debug] Stopping slave IO/SQL thread on slave1(10.24.220.70:)..
Sun May :: - [debug] done.
Sun May :: - [info] Getting new master's binlog name and position..
Sun May :: - [info] log.:
Sun May :: - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='slave1 or 10.24.220.70', MASTER_PORT=, MASTER_AUTO_POSITION=, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Sun May :: - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: log., , 191f7a9f-ffa2-11e5-a825-00163e00242a:-
Sun May :: - [info] Executing master IP activate script:
Sun May :: - [info] /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=server1 --orig_master_ip=10.24.220.232 --orig_master_port= --new_master_host=slave1 --new_master_ip=10.24.220.70 --new_master_port= --new_master_user='root' --new_master_password=''
Unknown option: new_master_user
Unknown option: new_master_password IN SCRIPT TEST====/sbin/ifconfig eth0: down==/sbin/ifconfig eth0: 10.24.220.100/=== Enabling the VIP - 10.24.220.100/ on the new master - slave1
Sun May :: - [info] OK.
Sun May :: - [info] Setting read_only= on slave1(10.24.220.70:)..
Sun May :: - [info] ok.
Sun May :: - [info] ** Finished master recovery successfully.
Sun May :: - [info] * Phase : Master Recovery Phase completed.
Sun May :: - [info]
Sun May :: - [info] * Phase : Slaves Recovery Phase..
Sun May :: - [info]
Sun May :: - [info]
Sun May :: - [info] * Phase 4.1: Starting Slaves in parallel..
Sun May :: - [info]
Sun May :: - [info] -- Slave recovery on host slave2(10.169.214.33:) started, pid: . Check tmp log /var/log/masterha/app1/slave2_3306_20160515204944.log if it takes time..
Sun May :: - [info]
Sun May :: - [info] Log messages from slave2 ...
Sun May :: - [info]
Sun May :: - [info] Resetting slave slave2(10.169.214.33:) and starting replication from the new master slave1(10.24.220.70:)..
Sun May :: - [debug] Stopping slave IO/SQL thread on slave2(10.169.214.33:)..
Sun May :: - [debug] done.
Sun May :: - [info] Executed CHANGE MASTER.
Sun May :: - [debug] Starting slave IO/SQL thread on slave2(10.169.214.33:)..
Sun May :: - [debug] done.
Sun May :: - [info] Slave started.
Sun May :: - [info] gtid_wait(191f7a9f-ffa2-11e5-a825-00163e00242a:-) completed on slave2(10.169.214.33:). Executed events.
Sun May :: - [info] End of log messages from slave2.
Sun May :: - [info] -- Slave on host slave2(10.169.214.33:) started.
Sun May :: - [info] All new slave servers recovered successfully.
Sun May :: - [info]
Sun May :: - [info] * Phase : New master cleanup phase..
Sun May :: - [info]
Sun May :: - [info] Resetting slave info on the new master..
Sun May :: - [debug] Clearing slave info..
Sun May :: - [debug] Stopping slave IO/SQL thread on slave1(10.24.220.70:)..
Sun May :: - [debug] done.
Sun May :: - [debug] SHOW SLAVE STATUS shows new master does not replicate from anywhere. OK.
Sun May :: - [info] slave1: Resetting slave info succeeded.
Sun May :: - [info] Master failover to slave1(10.24.220.70:) completed successfully.
Sun May :: - [info] Deleted server1 entry from /etc/masterha/app1.conf .
Sun May :: - [debug] Disconnected from slave1(10.24.220.70:)
Sun May :: - [debug] Disconnected from slave2(10.169.214.33:)
Sun May :: - [info] ----- Failover Report ----- app1: MySQL Master failover server1(10.24.220.232:) to slave1(10.24.220.70:) succeeded Master server1(10.24.220.232:) is down! Check MHA Manager logs at monitor:/var/log/masterha/app1/manager.log for details. Started automated(non-interactive) failover.
Invalidated master IP address on server1(10.24.220.232:)
Selected slave1(10.24.220.70:) as a new master.
slave1(10.24.220.70:): OK: Applying all logs succeeded.
slave1(10.24.220.70:): OK: Activated master IP address.
slave2(10.169.214.33:): OK: Slave started, replicating from slave1(10.24.220.70:)
slave1(10.24.220.70:): Resetting slave info succeeded.
Master failover to slave1(10.24.220.70:) completed successfully.