一、keepalived介绍
keepalived软件最开始是转为负载均衡软件LVS而设计，用来管理和监控LVS集群系统中各个服务节点的状态，后来又加入了可实现高可用的VRRP功能。所以Keepalived除了能管理LVS以外，还可以作为其他服务（如：Nginx、Haproxy、MySQL）的高可用解决方案的软件。Keepalived是类似工作在lay3、lay4和lay7的交换机制的软件。
Keepalived软件是通过VRRP协议实现高可用功能。VRRP（虚拟路由器冗余协议）目的就是为了解决静态路由单点故障的问题，它能够保证当个别节点宕机时，整个网络还可以正常地运行。所以Keepalived一方面有配置管理LVS的功能，还可以对LVS下面的节点进行健康监测，另一方面又可以实现系统网络服务的高可用功能。
二、Keepalived的三个功能
1、管理LVS负载均衡软件
2、实现对LVS集群节点健康检查功能
3、作为系统网络服务的高可用功能（重点）
Keepalived的作用是检测Web服务器的状态，如果有1台web或MySQL服务器宕机或故障，Keepalived检测到后，会将故障的Web服务器或MySQL服务器从集群当中剔除，而当服务器恢复正常后，Keepalived会自动将剔除的服务器重新加入到集群当中，这些工作无需人工参与，需要人工参与的是服务器故障的修复。
三、Keepalived的工作原理
Keepalived高可用之间是通过VRRP进行通信的。那什么是VRRP协议呢？
（1）VRRP，全称Virtual Router Redundancy Protocol，中文为虚拟路由冗余协议，VRRP的出现是为了解决静态路由的单点故障。
（2）VRRP是通过一种竞选协议机制来决定将路由任务交给某台VRRP路由器的。
（3）VRRP用IP多播的方式（默认多播地址：224.0.0.18）实现高可用对之通信。
（4）工作做时，主节点发包，备用节点接包，当备用节点接收不到主节点发送的数据包时，就会启动接管程序接管主节点的资源。备用节点可以有多个，通过优先级竞选，但一般的Keepalived系统运行工作中都是一对。
（5）VRRP使用了加密协议加密数据，但是目录官方还是推荐以明文的方式配置认证类型和密码。
明确了VRRP协议，再看Keepalived工作原理：
Keepalived高可用对之间是通过VRRP进行通信，VRRP通过竞选机制来确定主备，主的优先级高于备，因此工作时，主会优先获得所有资源，备节点处于等待状态，当主宕机后，备用节点则会接管主节点资源，然后顶替主节点对外提供服务。
在Keepalived服务对之间，只有作为主的服务器会一直发送VRRP广播包，告诉备用节点主节点还活着，此时备用节点不会抢占主，当主不可用时，即备监听不到主发送的广播包时，就会启动相关的服务接管资源，保证业务的连续性，接管速度最快可以小于1秒。
四、Keepalived高可用服务部署
1、环境说明
| Hostname | IP | 角色说明 |
| lb01 | 192.168.56.12 | keepalived MASTER |
| lb02 | 192.168.56.13 | keepalived BACKUP |
2、部署Keepalived
（1）安装keepalived
[root@lb01 ~]# yum install -y keepalived
[root@lb02 ~]# yum install -y keepalived
[root@lb01 ~]# rpm -qa keepalived
keepalived-1.3.-.el7.x86_64
[root@lb02 ~]# rpm -qa keepalived
keepalived-1.3.-.el7.x86_64
（2）keepalived.conf配置文件高可用部分解析
[root@lb01 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs { #定义服务故障报警的E-mail地址，可配多个地址，可选配置
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc #指定发送邮件的发件人，即发送人地址，可选配置
smtp_server 192.168.200.1 #指定发送邮件的smtp服务器，本机开启了sendmail或postfix就可以使用上面的默认地址发送邮件，可选配置
smtp_connect_timeout #链接smtp超时时间，可选配置
router_id LVS_DEVEL #Keepalived服务器的路由标识，在同一局域网内该标识具有唯一性
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval
vrrp_gna_interval
} vrrp_instance VI_1 { #VRRP实例定义区块，定义了一个VI_1的实例，每个vrrp_instance实例可以认为是Keepalived服务的一个实例或作为一个业务服务，在主节点中有的vrrp_instance实例，备用节点也要存在，这样故障才能接管。
state MASTER #定义Keepalived的主备状态，只能有MASTER和BACKUP两种状态，并且状态字符要大写
interface eth0 #定义Keepalived使用的网卡接口
virtual_router_id #虚拟路由ID标识，这个标识最好是一个数字，并且唯一。MASTER和BACKUP配置中相同实例的这个id必须一致，否则会脑裂。
priority #优先级配置，数值越大，实例优先级越高，建议MASTER和BACKUP相差50以上为佳。
advert_int #同步通知间隔，也就是MASTER和BACKUP之间通信检查时间间隔，单位为秒，默认为1.
authentication { #权限认证配置
auth_type PASS #认证类型有PASS、AH2中，官方推荐PASS，不超过8个字符，同一实例MASTER和BACKUP使用相同密码才能正常通信。
auth_pass #认证密码
}
virtual_ipaddress { #虚拟IP地址，可以配置多个IP地址，每个一行，配置时最好明确指定子网掩码和虚拟IP绑定的网络接口。
192.168.200.16
192.168.200.17
192.168.200.18
}
}
3、Keepalived高可用服务单实例演示
（1）配置Keepalived主服务器lb01 MASTER
[root@lb01 keepalived]# cp keepalived.conf keepalived.conf.bak
[root@lb01 keepalived]# vim keepalived.conf
! Configuration File for keepalived global_defs {
notification_email {
@qq.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout
router_id lb01
} vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id
priority
advert_int
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
192.168.56.20/ dev eth0 label eth0:
}
}
[root@lb01 keepalived]# systemctl start keepalived //配置完成启动keepalived
[root@lb01 keepalived]# ip addr |grep 192.168.56.20 //查看是否有配置的虚拟IP：192.168.56.20
inet 192.168.56.20/ scope global secondary eth0:
（2）配置Keepalived备服务器lb02 BACKUP
[root@lb02 keepalived]# cp keepalived.conf keepalived.conf.bak
[root@lb02 keepalived]# vim keepalived.conf
! Configuration File for keepalived global_defs {
notification_email {
@qq.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout
router_id lb02
} vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id
priority
advert_int
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
192.168.56.20/ dev eth0 label eth0:
}
}
[root@lb02 keepalived]# systemctl start keepalived //配置完成启动keepalived
[root@lb02 keepalived]# ip addr |grep 192.168.56.20 //查看是否有配置的虚拟IP：192.168.56.20，备用服务器查看是不存在虚拟IP的，如果有返回结果，说明脑裂了
（3）高可用主备服务器切换测试
（）停止主上的keepalived服务，查看lb01和lb02的虚拟ip
[root@lb01 keepalived]# systemctl stop keepalived //停止主上的keepalived服务
[root@lb01 keepalived]# ip addr |grep 192.168.56.20 //lb01上停止keepalived后，查看lb01上是不存在虚拟ip：192.168.56.20
[root@lb02 keepalived]# ip addr |grep 192.168.56.20 //lb02上可以看到虚拟ip：192.168.56.20，实现了VIP漂移
inet 192.168.56.20/ scope global secondary eth0: （）重新启动主上的keepalived服务，查看lb01和lb02的虚拟ip
[root@lb01 keepalived]# systemctl start keepalived //重新启动lb01上的keepalived
[root@lb01 keepalived]# ip addr |grep 192.168.56.20 //可以看到虚拟ip又重新回到了lb01上
inet 192.168.56.20/ scope global secondary eth0:
[root@lb02 keepalived]# ip addr |grep 192.168.56.20 //lb02上再查询虚拟ip信息是不存在虚拟ip的
4、Keepalived双实例双主模式演示
（1）修改lb01和lb02的主配置文件，增加一个实例vrrp_VI2
[root@lb01 keepalived]# cat keepalived.conf
! Configuration File for keepalived global_defs {
notification_email {
@qq.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout
router_id lb01
} vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id
priority
advert_int
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
192.168.56.20/ dev eth0 label eth0:
}
} vrrp_instance VI_2 { //增加一个vrrp实例VI2
state BACKUP
interface eth0
virtual_router_id
priority
advert_int
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
192.168.56.30/ dev eth0 label eth0: //虚拟ip为192.168.56.30
}
} [root@lb02 keepalived]# cat keepalived.conf
! Configuration File for keepalived global_defs {
notification_email {
@qq.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout
router_id lb02
} vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id
priority
advert_int
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
192.168.56.20/ dev eth0 label eth0:
}
} vrrp_instance VI_2 { //增加一个vrrp实例VI2
state MASTER
interface eth0
virtual_router_id
priority
advert_int
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
192.168.56.30/ dev eth0 label eth0: //虚拟ip为192.168.56.30
}
}
（2）在lb01和lb02上分别重启Keepalived服务，观察初始VIP设置情况
[root@lb01 keepalived]# systemctl restart keepalived
[root@lb01 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"
inet 192.168.56.20/ scope global secondary eth0: [root@lb02 keepalived]# systemctl restart keepalived
[root@lb02 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"
inet 192.168.56.30/ scope global secondary eth0:
启动lb01的Keepalived服务后，初始状态启动了192.168.56.20这个VIP地址，即由VI_1实例配置的VIP对外提供服务。
启动lb02的Keepalived服务后，初始状态启动了192.168.56.30这个VIP地址，即由VI_2实例配置的VIP对外提供服务。
（3）高可用故障切换测试
[root@lb01 keepalived]# systemctl stop keepalived //停止lb01的keepalived服务
[root@lb01 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30" //在lb01上是无法查看到vip
[root@lb02 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30" //在lb02上是可以查看到2个vip地址的
inet 192.168.56.30/ scope global secondary eth0:
inet 192.168.56.20/ scope global secondary eth0:
[root@lb01 keepalived]# systemctl start keepalived //重新启动lb01上的keepalived服务
[root@lb01 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30" //可以看到vip地址192.168.56.20飘移回来了
inet 192.168.56.20/ scope global secondary eth0: 同理测试停止lb02上的keepalived服务查看vip信息 [root@lb02 keepalived]# systemctl stop keepalived
[root@lb02 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"
[root@lb01 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"
inet 192.168.56.20/ scope global secondary eth0:
inet 192.168.56.30/ scope global secondary eth0:
[root@lb02 keepalived]# systemctl start keepalived
[root@lb02 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"
inet 192.168.56.30/ scope global secondary eth0:
[root@lb01 keepalived]# ip addr |egrep "192.168.56.20|192.168.56.30"
inet 192.168.56.20/ scope global secondary eth0:
五、Nginx负载均衡配置Keepalived服务
1、环境说明：
| Hostname | IP | 角色说明 |
| lb01 | 192.168.56.12 | Nginx+Keepalived（MASTER） |
| lb02 | 192.168.56.13 | Nginx+Keepalived（BACKUP） |
| web01 | 192.168.56.11 | web01服务-->Nginx |
| web02 | 192.168.0.130 | web02服务-->Nginx |
2、配置web01和web02
[root@web01 vhosts]# cat www.abc.org.conf
server {
listen ;
server_name 192.168.56.11;
root /vhosts/html/www;
index index.html index.htm index.php;
}
[root@web02 vhosts]# cat www.abc.org.conf
server {
listen ;
server_name 192.168.0.130;
root /vhosts/html/www;
index index.html index.htm index.php;
} 测试web01和web02的主页，进行区分
[root@localhost vhosts]# curl 192.168.56.11
welcome to 192.168.56.11
[root@localhost vhosts]# curl 192.168.0.130:
welcome to use 192.168.0.130
3、在lb01和lb02上配置Nginx负载均衡
[root@lb01 keepalived]# cat /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid; include /usr/share/nginx/modules/*.conf; events {
worker_connections 1024;
} http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048; include /etc/nginx/mime.types;
default_type application/octet-stream; include /etc/nginx/conf.d/*.conf;
upstream web_server_pool {
server 192.168.56.11:80 weight=1;
server 192.168.0.130:8080 weight=1;
}
server {
listen 80;
server_name 192.168.56.20; //此处的server_name需要配置VIP的地址
location / {
proxy_pass http://web_server_pool;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
}
} }
4、在lb01和lb02上配置Keepalived服务
[root@lb01 keepalived]# cat keepalived.conf
! Configuration File for keepalived global_defs {
notification_email {
@qq.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout
router_id lb01
} vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id
priority
advert_int
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
192.168.56.20/ dev eth0 label eth0:
}
} [root@lb02 keepalived]# cat keepalived.conf
! Configuration File for keepalived global_defs {
notification_email {
@qq.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout
router_id lb02
} vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id
priority
advert_int
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
192.168.56.20/ dev eth0 label eth0:
}
} 注意lb01和lb02中Keepalived配置的不同之处
5、访问测试
直接访问：http://192.168.56.20，可以看到刷新页面，分别得到不同的结果，说明Nginx的负载均衡功能实现了，如图：
再停止lb01上的keepalived，再查看是否能够保持访问
[root@lb01 keepalived]# systemctl stop keepalived
[root@lb01 keepalived]# ip addr |grep "192.168.56.20"
[root@lb02 keepalived]# ip addr |grep "192.168.56.20" //可以看到停止lb01上的keepalived后，vip在lb02上
inet 192.168.56.20/ scope global secondary eth0:
再进行访问：http://192.168.56.20，一样可以保持访问结果，这就实现了Keepalived的高可用功能，如图：
6、解决Nginx监控检查的问题
按照前面的操作，顺利地实现了Nginx的反向代理和负载均衡，也实现了Keepalived的高可用功能，在默认情况下，Keepalived仅仅在对方机器宕机或者Keepalived服务停止时才会接管也业务，而在实际工作当中，也会有其中一台负载均衡器的Nginx宕机了，而Keepalived服务还在运行，这就会导致用户访问的VIP：192.168.56.20无法找到对应的服务。尝试把lb01的Nginx停止，再查看访问情况
（）首先先进行访问测试，可以看到都是正常的
[root@localhost vhosts]# curl 192.168.56.20
welcome to 192.168.56.110
[root@localhost vhosts]# curl 192.168.56.20
welcome to use 192.168.0.130 （）停止lb01上的nginx，查看vip依旧还在lb01上
[root@lb01 keepalived]# systemctl stop nginx
[root@lb01 keepalived]# ip addr |grep "192.168.56.20"
inet 192.168.56.20/ scope global secondary eth0: （）再进行测试访问，发现连接被拒绝
[root@localhost vhosts]# curl 192.168.56.20
curl: () Failed connect to 192.168.56.20:; Connection refused
那么，如何解决这种业务服务宕机还可以将IP漂移到备用节点上呢？这就需要Keepalived监测脚本了。首先先写一个脚本，如下：
#!/bin/bash
d=`date --date today +%Y%m%d_%H:%M:%S`
counter=$(ps -C nginx --no-heading |wc -l)
if [ "${counter}" = "" ]; then
systemctl start nginx.service
sleep
counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "" ]; then
echo "$d nginx was down.keepalived will stop." >> /var/log/check_ng.log
systemctl stop keepalived
fi
fi
此处在监测到nginx进程为0时，会重新启动nginx，再进行统计nginx的进程数量，如果依旧为0，则将keepalived服务停止，启用高可用故障切换。实验阶段，为了看到效果，使用一下脚本，只要监测到了nginx进程数为0，即刻停止keepalived服务，脚本如下：
此脚本在lb01和lb02上都需要存在的，脚本路径：/etc/keepalived/check_nginx.sh
[root@lb01 keepalived]# cat check_nginx.sh
#!/bin/bash
d=`date --date today +%Y%m%d_%H:%M:%S`
counter=$(ps -C nginx --no-heading|wc -l)
if [ $counter -eq ]; then
echo "$d nginx was down.keepalived will stop." >> /var/log/check_ng.log
systemctl stop keepalived
fi
再对lb01和lb02的keepalived.conf配置文件进行修改，增加脚本模块：
[root@lb01 keepalived]# cat keepalived.conf
! Configuration File for keepalived global_defs {
notification_email {
@qq.com
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout
router_id lb01
} vrrp_script chk_nginx { #定义vrrp脚本，检测nginx进程，此处一定要注意和"{"的空格，如果没有空格，会导致脚本不会执行，切记切记！！！
script "/etc/keepalived/check_nginx.sh" #执行脚本，当Nginx服务有问题，就停掉Keepalived
interval 2 #监测的间隔时间为2s
weight
} vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id
priority
advert_int
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
192.168.56.20/
}
track_script {
chk_nginx #在vrrp实例VI_1启用chk_nginx这个脚本
}
}
下面测试过程和结果：
（1）在lb01上查看keepalived的vip和进程以及nignx的端口
[root@lb01 keepalived]# !ip
ip addr |grep "192.168.56.20"
inet 192.168.56.20/ scope global secondary eth0
[root@lb01 keepalived]# netstat -tulnp |grep nginx
tcp 0.0.0.0: 0.0.0.0:* LISTEN /nginx: master
[root@lb01 keepalived]# ps -ef |grep keepalived
root : ? :: /usr/sbin/keepalived -D
root : ? :: /usr/sbin/keepalived -D
root : ? :: /usr/sbin/keepalived -D （2）模拟Nginx故障，停止Nginx服务，再查看（1）中的相关信息
[root@lb01 keepalived]# systemctl stop nginx
[root@lb01 keepalived]# !nets
netstat -tulnp |grep nginx
[root@lb01 keepalived]# !ip
ip addr |grep "192.168.56.20"
[root@lb01 keepalived]# ps -ef |grep keepalived
root : pts/ :: grep --color=auto keepalived
（3）在lb02上查看VIP信息是否存在，并验证web服务访问是否正常
[root@lb02 keepalived]# !ip
ip addr |grep "192.168.56.20"
inet 192.168.56.20/ scope global secondary eth0
[root@localhost vhosts]# curl 192.168.56.20
welcome to use 192.168.0.130
[root@localhost vhosts]# curl 192.168.56.20
welcome to 192.168.56.110
通过上述的脚本监测，可以实现了真正的Nginx+Keepalived高可用故障切换功能。
7、写一个监测Keepalived脑裂的脚本
为了防止高可用功能出现脑裂现象，还可以在备用服务器上写一个监测脚本，如果可以ping通主节点并且备用节点有VIP就报警。
（1）在lb02上写一个监测脚本并执行
[root@lb02 keepalived]# cat check_split_brain.sh
#!/bin/bash
lb01_vip="192.168.56.20"
lb01_ip="192.168.56.12"
while true
do
ping -c -W $lb01_ip &>/dev/null
if [ $? -eq -a `ip add|grep "$lb01_vip"|wc -l` -eq ]
then
echo "ha is split brain.warning."
else
echo "ha is ok."
fi
sleep
done
[root@lb02 keepalived]# sh check_split_brain.sh
ha is ok.
ha is ok.
ha is ok.
正常情况下，主节点还活着，VIP 192.168.56.20就在主节点上，不会报警，提示：ha is ok
（2）模拟脑裂：停止主节点lb01上的Keepalived，查看lb02上的脚本执行情况
[root@lb01 keepalived]# systemctl stop keepalived
[root@lb02 keepalived]# sh check_split_brain.sh
ha is ok.
ha is split brain.warning.
ha is split brain.warning.
从上可以看到脚本会报警有脑裂的错误，即可将此叫脚本放在zabbix监控服务当中，实现脑裂报警。