文章目录
IP
SERVICES
192.168.72.55
keepalived+nginx
192.168.72.56
keepalived+nginx
192.168.72.57
keepalived+nginx
192.168.72.100
VIP
如果需要给kube-apiserver提供高可用，建议在部署kubernetes集群的时候就预先部署好keepalived+nginx，
否则的话，需要修改kube-apiserver的证书，将vip加入到证书内，
并且修改kube-config相关的秘钥（kube-controller、kube-scheduler、kubelet、kube-proxy）都需要重新生成，将原有的kube-apiserver的认证地址修改为VIP
重启kube-apiserver、kube-controller、kube-scheduler、kubelet、kube-proxy服务，最终就可以实现高可用
所以，在kubernetes集群部署前先完成keepalived+nginx就会方便很多
为什么选择keepalived？
因为keepalived有后端服务健康检测机制，检测到后端的nginx（nginx为了使用upstream模块实现负载均衡）服务有故障，就会将自己（keepalived）关闭，使VIP漂移到另外两个节点中的一个，用来保证和实现kube-apiserver服务的高可用
此次部署，是基于了我博客前面的 suse 12 二进制部署 Kubernetets 1.19.7系列，最后的分发脚本和启动服务里面的数组需要自行修改(修改成自己的ip或者主机名即可，主机名需要提前做好hosts解析)
k8s-01:~ # cd /opt/k8s/packages/
k8s-01:/opt/k8s/packages # wget http://nginx.org/download/nginx-1.16.1.tar.gz
k8s-01:/opt/k8s/packages # tar xf nginx-1.16.1.tar.gz
k8s-01:~ # cd /opt/k8s/packages/nginx-1.16.1/
k8s-01:/opt/k8s/packages/nginx-1.16.1 # ./configure --prefix=$(pwd)/nginx-prefix \
--with-stream \
--without-http \
--without-http_uwsgi_module && \
make && \
make install
k8s-01:~ # cd /opt/k8s/conf/
k8s-01:/opt/k8s/conf # cat > kube-nginx.conf <<EOF
worker_processes 1;
events {
worker_connections 1024;
}
stream {
upstream backend {
hash \$remote_addr consistent;
server 192.168.72.55:6443 max_fails=3 fail_timeout=30s;
server 192.168.72.56:6443 max_fails=3 fail_timeout=30s;
server 192.168.72.57:6443 max_fails=3 fail_timeout=30s;
}
server {
listen *:8443;
proxy_connect_timeout 1s;
proxy_pass backend;
}
}
EOF
k8s-01:~ # cd /opt/k8s/conf/
k8s-01:/opt/k8s/conf # cat > kube-nginx.service <<EOF
[Unit]
Description=kube-apiserver nginx proxy
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=forking
ExecStartPre=/opt/k8s/server/kube-nginx/sbin/nginx \
-c /opt/k8s/server/kube-nginx/conf/kube-nginx.conf \
-p /opt/k8s/server/kube-nginx -t
ExecStart=/opt/k8s/server/kube-nginx/sbin/nginx \
-c /opt/k8s/server/kube-nginx/conf/kube-nginx.conf \
-p /opt/k8s/server/kube-nginx
ExecReload=/opt/k8s/server/kube-nginx/sbin/nginx \
-c /opt/k8s/server/kube-nginx/conf/kube-nginx.conf \
-p /opt/k8s/server/kube-nginx -s reload
PrivateTmp=true
Restart=always
RestartSec=5
StartLimitInterval=0
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
#!/usr/bin/env bash
source /opt/k8s/bin/k8s-env.sh
for host in ${MASTER_IPS[@]}
do
printf "\e[1;34m${host}\e[0m\n"
ssh root@${host} "mkdir -p /opt/k8s/server/kube-nginx/{conf,logs,sbin}"
scp /opt/k8s/packages/nginx-1.16.1/nginx-prefix/sbin/nginx ${host}:/opt/k8s/server/kube-nginx/sbin/
scp /opt/k8s/conf/kube-nginx.conf ${host}:/opt/k8s/server/kube-nginx/conf/
scp /opt/k8s/conf/kube-nginx.service ${host}:/etc/systemd/system/
done
#!/usr/bin/env bash
source /opt/k8s/bin/k8s-env.sh
for host in ${MASTER_IPS[@]}
do
printf "\e[1;34m${host}\e[0m\n"
ssh root@${host} "systemctl daemon-reload && \
systemctl enable kube-nginx --now && \
systemctl status kube-nginx | grep Active"
done
k8s-01:~ # cd /opt/k8s/packages/
k8s-01:/opt/k8s/packages # wget https://www.keepalived.org/software/keepalived-2.2.0.tar.gz
k8s-01:/opt/k8s/packages # tar xf keepalived-2.2.0.tar.gz
k8s-01:/opt/k8s/packages/keepalived-2.2.0 # ./configure --prefix=$(pwd)/keepalived-prefix && \
make && \
make install
k8s-01:~ # cd /opt/k8s/conf/
k8s-01:/opt/k8s/conf # cat > keepalived.conf.template <<EOF
! Configuration File for keepalived
global_defs {
router_id 192.168.0.50
}
vrrp_script chk_nginx {
script "/etc/keepalived/check_port.sh 8443"
interval 2
weight -20
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 251
priority 100
advert_int 1
mcast_src_ip ##NODE_IP##
nopreempt
authentication {
auth_type PASS
auth_pass 11111111
}
track_script {
chk_nginx
}
virtual_ipaddress {
192.168.72.100
}
}
EOF
为了避免keepalived服务出现问题，修复后重启keepalived，出现IP漂移回来的情况，这里选择了3个都是BACKUP的模式，减少数据的丢失
k8s-01:~ # cd /opt/k8s/conf/
k8s-01:/opt/k8s/conf # cat > check_port.sh <<"EOF"
CHK_PORT=$1
if [ -n "$CHK_PORT" ];then
PORT_PROCESS=$(ss -lt|grep $CHK_PORT|wc -l)
if [ $PORT_PROCESS -eq 0 ];then
echo "Port $CHK_PORT Is Not Used,End."
exit 1
fi
else
echo "Check Port Cant Be Empty!"
fi
EOF
k8s-01:~ # cd /opt/k8s/conf/
k8s-01:/opt/k8s/conf # cat > keepalived.service <<EOF
[Unit]
Description=LVS and VRRP High Availability Monitor
After=syslog.target network-online.target
[Service]
Type=forking
PIDFile=/var/run/keepalived.pid
KillMode=process
EnvironmentFile=-/etc/sysconfig/keepalived
ExecStart=/usr/sbin/keepalived \$KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP \$MAINPID
[Install]
WantedBy=multi-user.target
EOF
#!/usr/bin/env bash
source /opt/k8s/bin/k8s-env.sh
for (( i=0; i < 3; i++ ))
do
sed -e "s/##NODE_IP##/${MASTER_IPS[i]}/" /opt/k8s/conf/keepalived.conf.template > \
/opt/k8s/conf/keepalived.conf-${MASTER_IPS[i]}.template
done
for host in ${MASTER_IPS[@]}
do
printf "\e[1;34m${host}\e[0m\n"
ssh root@${host} "mkdir -p /etc/keepalived"
scp /opt/k8s/packages/keepalived-2.2.0/keepalived-prefix/sbin/keepalived ${host}:/usr/sbin/
scp /opt/k8s/packages/keepalived-2.2.0/keepalived-prefix/etc/sysconfig/keepalived ${host}:/etc/sysconfig/
scp /opt/k8s/conf/keepalived.conf-${host}.template ${host}:/etc/keepalived/keepalived.conf
scp /opt/k8s/conf/check_port.sh ${host}:/etc/keepalived/
scp /opt/k8s/conf/keepalived.service ${host}:/etc/systemd/system/
done
for host in ${MASTER_IPS[@]}
do
printf "\e[1;34m${host}\e[0m\n"
ssh root@${host} "systemctl daemon-reload && \
systemctl enable keepalived --now && \
systemctl status keepalived | grep Active"
done
#!/usr/bin/env bash
source /opt/k8s/bin/k8s-env.sh
for host in ${MASTER_IPS[@]}
do
printf "\e[1;34m${host}\e[0m\n"
ssh root@${host} "ip a | grep 192.168.72.100"
done
ping 192.168.72.100 -c 1