使用kubeadm部署一套高可用k8s集群

有疑问的地方可以看官方文档

准备环境

我的机器如下, 系统为ubuntu20.04, kubernetes版本1.21.0

hostname	IP	硬件配置(最低)
ha	10.0.0.10	1核1G
k8s-master0	10.0.0.20	4核4G
k8s-master1	10.0.0.21	4核4G
k8s-master2	10.0.0.22	4核4G
k8s-work0	10.0.0.30	4核4G

基础环境配置

基础环境是不管master还是work都需要的环境

1. 禁用swap
2. 确保每个节点上 MAC 地址和 product_uuid 的唯一性

product_uuid可以这样查看

sudo cat /sys/class/dmi/id/product_uuid

1. 修改hostname
2. 允许 iptables 检查桥接流量

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf

br_netfilter

EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

EOF

sudo sysctl --system

1. 安装runtime
   
   这里安装docker

curl -fsSL get.docker.com | bash

修改docker配置

cat > /etc/docker/daemon.json << EOF

{

  "registry-mirrors": ["https://go38p9zi.mirror.aliyuncs.com"],

  "exec-opts": ["native.cgroupdriver=systemd"],

  "log-driver": "json-file",

  "log-opts": {

    "max-size": "100m"

  }

}

EOF

1. 安装kubeadm、kubelet 和 kubectl

kubectl 并不是每台机器都需要安装

更新 apt 包索引并安装使用 Kubernetes apt 仓库所需要的包：

sudo apt-get update

sudo apt-get install -y apt-transport-https ca-certificates curl

下载 Google Cloud 公开签名秘钥：

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

添加 Kubernetes apt 仓库：

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

更新 apt 包索引，安装 kubelet、kubeadm 和 kubectl，并锁定其版本：

sudo apt-get update

sudo apt-get install -y kubelet kubeadm kubectl

sudo apt-mark hold kubelet kubeadm kubectl

准备负载均衡

在ha机器上执行

安装haproxy

apt install -y haproxy

配置haproxy

--- 前面保持默认配置 ---

frontend k8s_api_fe

    bind :6443

    default_backend k8s_api_be

    mode tcp

    option tcplog

backend k8s_api_be

    balance source

    mode tcp

    server      k8s-master0 10.0.0.20:6443 check

    server      k8s-master1 10.0.0.21:6443 check

    server      k8s-master2 10.0.0.22:6443 check

frontend http_ingress_traffic_fe

    bind :80

    default_backend http_ingress_traffic_be

    mode tcp

    option tcplog

backend http_ingress_traffic_be

    balance source

    mode tcp

    server      k8s-work0 10.0.0.30:80 check

frontend https_ingress_traffic_fe

    bind *:443

    default_backend https_ingress_traffic_be

    mode tcp

    option tcplog

backend https_ingress_traffic_be

    balance source

    mode tcp

    server      k8s-work0 10.0.0.30:443 check

创建集群

kubeadm init

在k8s-master0上执行

kubeadm init \

--apiserver-advertise-address=10.0.0.10 \

--service-cidr=10.96.0.0/12 \

--pod-network-cidr=10.244.0.0/16

其他参数:

--image-repository registry.aliyuncs.com/k8sxio

--kubernetes-version v1.18.0

安装flannel插件

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

获取join命令, 增加新的节点

node

kubeadm init 后会输出在终端上, 有效期2小时, 超时后可以重新生成

生成添加命令:

kubeadm token create --print-join-command

master

生成证书, 记录输出结果

kubeadm init phase upload-certs --upload-certs

获取加入命令

kubeadm token create --print-join-command

上面两步可以简化成

echo "`kubeadm token create --print-join-command` --control-plane --certificate-key `kubeadm init phase upload-certs --upload-certs | tail -1`"

移除节点

移除节点

kubectl drain k8s-node1 --ignore-daemonsets

kubectl delete node k8s-node1

如果是master节点还需要移除etcd member

kubectl exec -it -n kube-system etcd-k8s-master0 /bin/sh

etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list

可以看到有多少个节点

etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove 12637f5ec2bd02b8

通过ID来删除