LVS+Keepalived 高可用群集部署

2023-11-28 22:46:28

一、LVS+Keepalived 高可用群集

在这个高度信息化的 IT 时代，企业的生产系统、业务运营、销售和支持，以及日常管理等环节越来越依赖于计算机信息和服务，对高可用（HA）技术的应用需求不断提高，以便提供持续的、不间断的计算机系统或网络服务。

Keepalived是一个基于VRRP协议来实现的LVS服务高可用方案，可以解决静态路由出现的单点故障问题。

1、Keepalived工具介绍

专为LVS和HA设计的一款健康检查工具

? 支持故障自动切换（Failover）

? 支持节点健康状态检查（Health Checking）

? 官方网站：http://www.keepalived.org/

2、工作原理

在一个LVS服务集群中通常有主服务器(MASTER)和备份服务器(BACKUP)两种角色的服务器，但是对外表现为一个虚拟IP，主服务器会发送VRRP通告信息给备份服务器，当备份服务器收不到VRRP消息的时候，即主服务器异常的时候，备份服务器就会接管虚拟IP，继续提供服务，从而保证了高可用性。

3、Keepalived实现原理剖析

Keepalived采用VRRP热备份协议实现Linux服务器的多机热备功能

4、 VRRP （虚拟路由冗余协议）

是针对路由器的一种备份解决方案

由多台路由器组成一个热备组，通过共用的虚拟IP地址对外提供服务

每个热备组内同时只有一台主路由器提供服务，其他路由器处于冗余状态

若当前在线的路由器失效，则其他路由器会根据设置的优先级自动接替虚拟IP地址，继续提供服务

5、故障转移机制

Keepalived高可用服务之间的故障切换转移，是通过VRRP 来实现的。

在 Keepalived服务正常工作时，主 Master节点会不断地向备节点发送（多播的方式）心跳消息，用以告诉备Backup节点自己还活着，当主 Master节点发生故障时，就无法发送心跳消息，备节点也就因此无法继续检测到来自主 Master节点的心跳了，于是调用自身的接管程序，接管主Master节点的 IP资源及服务。

而当主 Master节点恢复时，备Backup节点又会释放主节点故障时自身接管的IP资源及服务，恢复到原来的备用角色。

二、LVS+Keepalived 高可用群集的搭建步骤

环境准备

主DR 服务器： 192.168.229.90
备DR 服务器： 192.168.229.80
Web 服务器1： 192.168.229.70
Web 服务器2： 192.168.229.60
nfs 服务器： 192.168.229.50
客户端：192.168.229.200
vip：192.168.229.100

1、配置负载调度器（主、备相同；192.168.229.90；192.168.229.80）

systemctl stop firewalld.service
setenforce 0

yum -y install ipvsadm keepalived
modprobe ip_vs
cat /proc/net/ip_vs

1.1 配置keeplived（主、备DR 服务器上都要设置）　　

cd /etc/keepalived/
cp keepalived.conf keepalived.conf.bak
vim keepalived.conf
......
global_defs {						#定义全局参数
--10行--修改，邮件服务指向本地
	smtp_server 127.0.0.1
--12行--修改，指定服务器（路由器）的名称，主备服务器名称须不同，主为LVS_01，备为LVS_02
	router_id LVS_01
}

vrrp_instance VI_1 {				#定义VRRP热备实例参数
--20行--修改，指定热备状态，主为MASTER，备为BACKUP
    state MASTER
--21行--修改，指定承载vip地址的物理接口
    interface ens33
--22行--修改，指定虚拟路由器的ID号，每个热备组保持一致	
    virtual_router_id 10
--23行--修改，指定优先级，数值越大优先级越高，主为100，备为99
    priority 100
    advert_int 1					#通告间隔秒数(心跳频率)
    authentication {				#定义认证信息,每个热备组保持一致
		auth_type PASS				#认证类型
--27行--修改，指定验证密码，主备服务器保持一致
        auth_pass abc123
    }
    virtual_ipaddress {				#指定群集vip地址
        192.168.229.100
    }
}
--36行--修改，指定虚拟服务器地址(VIP)、端口，定义虚拟服务器和Web服务器池参数
virtual_server 192.168.229.100 80 {
    delay_loop 6					#健康检查的间隔时间(秒)
    lb_algo rr						#指定调度算法，轮询(rr)
--39行--修改，指定群集工作模式，直接路由(DR)
    lb_kind DR
    persistence_timeout 50			#连接保持时间(秒)
    protocol TCP					#应用服务采用的是 TCP协议
--43行--修改，指定第一个Web节点的地址、端口
    real_server 192.168.229.90 80 {
        weight 1					#节点的权重
--45行--删除，添加以下健康检查方式		
        TCP_CHECK {
			connect_port 80			#添加检查的目标端口
			connect_timeout 3		#添加连接超时(秒)
			nb_get_retry 3			#添加重试次数
			delay_before_retry 4	#添加重试间隔
		}
	}

	real_server 192.168.229.80 80 {		#添加第二个 Web节点的地址、端口
        weight 1
        TCP_CHECK {
			connect_port 80
			connect_timeout 3
			nb_get_retry 3
			delay_before_retry 4
		}
	}
##删除后面多余的配置##
}


systemctl start keepalived
ip addr show dev ens33				#查看虚拟网卡vip

1.2 配置分发策略（#keepalived配置好后，可以不用再配ipvsadm 分发策略）　　

ipvsadm-save > /etc/sysconfig/ipvsadm
systemctl start ipvsadm

ipvsadm -C
ipvsadm -A -t 192.168.229.100:80 -s rr
ipvsadm -a -t 192.168.229.100:80 -r 192.168.229.70:80 -g
ipvsadm -a -t 192.168.229.100:80 -r 192.168.229.60:80 -g
ipvsadm

ipvsadm -ln
ipvsadm-save > /etc/sysconfig/ipvsadm

#如果没有vip的分发策略。则重启keepalived 服务

1.3 调整内核 proc 响应参数，关闭linux内核的重定向参数响应　　

vim /etc/sysctl.conf
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0

sysctl -p

2、部署共享存储（NFS服务器：192.168.229.50）

systemctl stop firewalld.service
systemctl disable firewalld.service
setenforce 0

yum install nfs-utils rpcbind -y
systemctl start nfs.service
systemctl start rpcbind.service
systemctl enable nfs.service
systemctl enable rpcbind.service

mkdir /opt/accp /opt/benet
chmod 777 /opt/accp /opt/benet

vim /etc/exports
/usr/share *(ro,sync)
/opt/accp 192.168.229.0/24(rw,sync)
/opt/benet 192.168.229.0/24(rw,sync)

--发布共享---
exportfs -rv

3、配置节点服务器（192.168.229.70；192.168.229.60）　　

systemctl stop firewalld
setenforce 0

yum -y install httpd
systemctl start httpd

vim /etc/sysconfig/network-scripts/ifcfg-lo:0
DEVICE=lo:0
ONBOOT=yes
IPADDR=192.168.229.100
NETMASK=255.255.255.255

service network restart 或 systemctl restart network
ifup lo:0
ifconfig lo:0
route add -host 192.168.229.100 dev lo:0

vim /etc/sysctl.conf

net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

sysctl -p
--192.168.229.70---
mount.nfs 192.168.229.50:/opt/accp /var/www/html

--192.168.229.60---
mount.nfs 192.168.229.50:/opt/benet /var/www/html

4、测试验证

在客户端访问 http://192.168.229.100/ ，默认网关指向 192.168.229.100

再在主服务器关闭 keepalived 服务后在测试,systemctl stop keepalived

三、实例操作：LVS+Keepalived 高可用群集

1、配置负载调度器（主、备相同；192.168.229.90；192.168.229.80）

1.1 关闭防火墙并安装ipvsadm和keepalived工具

主DR 服务器： 192.168.229.90

[root@192 ~]# systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@192 ~]# setenforce 0
setenforce: SELinux is disabled
[root@192 ~]# yum -y install ipvsadm keepalived
[root@192 ~]# modprobe ip_vs
[root@192 ~]# cat /proc/net/ip_vs
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port Forward Weight ActiveConn InActConn

备DR 服务器： 192.168.229.80　　

[root@192 ~]# systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@192 ~]# setenforce 0
setenforce: SELinux is disabled
[root@192 ~]# yum -y install ipvsadm keepalived
[root@192 ~]# modprobe ip_vs
[root@192 ~]# cat /proc/net/ip_vs
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port Forward Weight ActiveConn InActConn

1.2 配置keeplived（主、备DR 服务器上都要设置）　

主DR 服务器： 192.168.229.90

[root@192 ~]# cd /etc/keepalived/
[root@192 keepalived]# ls
keepalived.conf
[root@192 keepalived]# cp keepalived.conf{,.bak}
[root@192 keepalived]# ls
keepalived.conf  keepalived.conf.bak
[root@192 keepalived]# vim keepalived.conf
[root@192 keepalived]# systemctl start keepalived.service 
[root@192 keepalived]# ip addr show dev ens33 
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:4a:3e:ba brd ff:ff:ff:ff:ff:ff
    inet 192.168.229.90/24 brd 192.168.229.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.229.100/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::d20e:a6c8:5f4a:ce2b/64 scope link 
       valid_lft forever preferred_lft forever

备DR 服务器： 192.168.229.80

[root@192 ~]# cd /etc/keepalived/
[root@192 keepalived]# ls
keepalived.conf
[root@192 keepalived]# cp keepalived.conf{,.bak}
[root@192 keepalived]# ls
keepalived.conf  keepalived.conf.bak
[root@192 keepalived]# vim keepalived.conf
[root@192 keepalived]# systemctl start keepalived.service 
[root@192 keepalived]# ip addr show dev ens33 
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:4a:3e:ba brd ff:ff:ff:ff:ff:ff
    inet 192.168.229.90/24 brd 192.168.229.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.229.100/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::d20e:a6c8:5f4a:ce2b/64 scope link 
       valid_lft forever preferred_lft forever

1.3 调整内核 proc 响应参数，关闭linux内核的重定向参数响应　　

主DR 服务器： 192.168.229.90

[root@192 keepalived]# vim /etc/sysctl.conf 
[root@192 keepalived]# sysctl -p
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0

备DR 服务器： 192.168.229.80

[root@192 keepalived]# vim /etc/sysctl.conf 
[root@192 keepalived]# sysctl -p
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0

2、部署共享存储（NFS服务器：192.168.229.50）

[root@192 ~]# systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@192 ~]# setenforce 0
setenforce: SELinux is disabled
[root@192 ~]# rpm -q rpcbind nfs-utils
rpcbind-0.2.0-42.el7.x86_64
nfs-utils-1.3.0-0.48.el7.x86_64
[root@192 ~]# mkdir /opt/ly /opt/weq
[root@192 ~]# echo ‘<h1>This is node server1!</h1>‘ > /opt/ly/index.html
[root@192 ~]# echo ‘<h1>This is node server2!</h1>‘ > /opt/weq/index.html
[root@192 ~]# ls /opt/ly /opt/weq
/opt/ly:
index.html

/opt/weq:
index.html
[root@192 ~]# vim /etc/exports
[root@192 ~]# chmod 777 /opt/ly /opt/weq
[root@192 ~]# showmount
clnt_create: RPC: Port mapper failure - Unable to receive: errno 111 (Connection refused)
[root@192 ~]# showmount -e
clnt_create: RPC: Port mapper failure - Unable to receive: errno 111 (Connection refused)
[root@192 ~]# systemctl start rpcbind
[root@192 ~]# systemctl start nfs
[root@192 ~]# systemctl enable rpcbind
[root@192 ~]# systemctl enable nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
[root@192 ~]# exportfs -rv
exporting 192.168.229.0/24:/opt/weq
exporting 192.168.229.0/24:/opt/ly
[root@192 ~]# showmount -e
Export list for 192.168.229.50:
/opt/weq 192.168.229.0/24
/opt/ly  192.168.229.0/24

3、配置节点服务器（192.168.229.70；192.168.229.60）　　

Web 服务器1： 192.168.229.70

[root@192 ~]# systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@192 ~]# setenforce 0
setenforce: SELinux is disabled
[root@192 ~]# yum -y install httpd
[root@192 ~]# systemctl start httpd.service 
[root@192 ~]# cd /etc/sysconfig/network-scripts/
[root@192 network-scripts]# ls
ifcfg-ens33  ifdown-ib    ifdown-ppp       ifdown-tunnel  ifup-ib    ifup-plusb   ifup-Team         network-functions
ifcfg-lo     ifdown-ippp  ifdown-routes    ifup           ifup-ippp  ifup-post    ifup-TeamPort     network-functions-ipv6
ifdown       ifdown-ipv6  ifdown-sit       ifup-aliases   ifup-ipv6  ifup-ppp     ifup-tunnel
ifdown-bnep  ifdown-isdn  ifdown-Team      ifup-bnep      ifup-isdn  ifup-routes  ifup-wireless
ifdown-eth   ifdown-post  ifdown-TeamPort  ifup-eth       ifup-plip  ifup-sit     init.ipv6-global
[root@192 network-scripts]# vim ifcfg-lo
[root@192 network-scripts]# cat ifcfg-lo
DEVICE=lo:0
IPADDR=192.168.229.100
NETMASK=255.255.255.255
ONBOOT=yes
[root@192 network-scripts]# ifup lo:0
[root@192 network-scripts]# ifconfig lo:0
lo:0: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 192.168.229.100  netmask 255.255.255.255
        loop  txqueuelen 1  (Local Loopback)
[root@192 network-scripts]# route add -host 192.168.229.100 dev lo:0
[root@192 network-scripts]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.229.2   0.0.0.0         UG    100    0        0 ens33
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
192.168.229.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33
192.168.229.100 0.0.0.0         255.255.255.255 UH    0      0        0 lo
[root@192 network-scripts]# vim /etc/sysctl.conf 
[root@192 network-scripts]# sysctl -p
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

[root@192 network-scripts]# showmount -e 192.168.229.50
Export list for 192.168.229.50:
/opt/weq 192.168.229.0/24
/opt/ly  192.168.229.0/24
[root@192 network-scripts]# mount.nfs 192.168.229.50:/opt/ly /var/www/html/
[root@192 network-scripts]# ls /var/www/html/
index.html
[root@192 network-scripts]# cat /var/www/html/index.html 
<h1>This is node server1!</h1>

Web 服务器2： 192.168.229.60

[root@192 ~]# systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@192 ~]# setenforce 0
setenforce: SELinux is disabled
[root@192 ~]# yum -y install httpd
[root@192 ~]# systemctl start httpd.service 
[root@192 ~]# cd /etc/sysconfig/network-scripts/
[root@192 network-scripts]# ls
ifcfg-ens33  ifdown-ib    ifdown-ppp       ifdown-tunnel  ifup-ib    ifup-plusb   ifup-Team         network-functions
ifcfg-lo     ifdown-ippp  ifdown-routes    ifup           ifup-ippp  ifup-post    ifup-TeamPort     network-functions-ipv6
ifdown       ifdown-ipv6  ifdown-sit       ifup-aliases   ifup-ipv6  ifup-ppp     ifup-tunnel
ifdown-bnep  ifdown-isdn  ifdown-Team      ifup-bnep      ifup-isdn  ifup-routes  ifup-wireless
ifdown-eth   ifdown-post  ifdown-TeamPort  ifup-eth       ifup-plip  ifup-sit     init.ipv6-global
[root@192 network-scripts]# vim ifcfg-lo
[root@192 network-scripts]# cat ifcfg-lo
DEVICE=lo:0
IPADDR=192.168.229.100
NETMASK=255.255.255.255
ONBOOT=yes
[root@192 network-scripts]# ifup lo:0
[root@192 network-scripts]# ifconfig lo:0
lo:0: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 192.168.229.100  netmask 255.255.255.255
        loop  txqueuelen 1  (Local Loopback)
[root@192 network-scripts]# route add -host 192.168.229.100 dev lo:0
[root@192 network-scripts]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.229.2   0.0.0.0         UG    100    0        0 ens33
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
192.168.229.0   0.0.0.0         255.255.255.0   U     100    0        0 ens33
192.168.229.100 0.0.0.0         255.255.255.255 UH    0      0        0 lo
[root@192 network-scripts]# vim /etc/sysctl.conf 
[root@192 network-scripts]# sysctl -p
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

[root@cm network-scripts]# showmount -e 192.168.229.50
Export list for 192.168.229.50:
/opt/weq 192.168.229.0/24
/opt/ly  192.168.229.0/24
[root@cm network-scripts]# mount.nfs 192.168.229.50:/opt/weq /var/www/html/
[root@cm network-scripts]# ls /var/www/html/
index.html
[root@cm network-scripts]# cat /var/www/html/index.html 
<h1>This is node server2!</h1>