K8S v1.18.x 部署-Kubeadm方式-5：部署Kubernetes HA集群

2024-01-03 16:41:22

高可用拓扑方案
kubeadm提供了2种部署一个高可用kubernetes集群的不同方式：

Stacked控制平面节点
这种方法所需基础设施较少。etcd成员节点和控制平面节点位于同一节点上。

使用外部etcd节点
这种方法所需基础设施较多。控制平面的节点和etcd成员节点是分开的。

定制控制平面配置
kubeadm通过ClusterConfiguration对象公开了extraArgs字段，它可以覆盖传递给控制平面组件(如APIServer、ControllerManager和Scheduler)的默认参数。

其中各组件配置使用如下字段定义：apiServer、controllerManager、scheduler。

其中extraArgs字段由key: value对组成。覆盖控制平面组件的参数时:

将适当的字段添加到配置中；
向字段添加要覆盖的参数值；
用--config运行kubeadm init。
创建种子节点的配置文件
仅在Master1节点操作。

kubeadm config print init-defaults > kubeadm-default-config.yaml

kubeadm-default-config.yaml示例

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:

groups:
- system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
- signing
- authentication
  kind: InitConfiguration
  localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
  nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-prod-m1
  taints:
- effect: NoSchedule
  key: node-role.kubernetes.io/master
  
  apiServer:
  timeoutForControlPlane: 4m0s
  apiVersion: kubeadm.k8s.io/v1beta2
  certificatesDir: /etc/kubernetes/pki
  clusterName: kubernetes
  controllerManager: {}
  dns:
  type: CoreDNS
  etcd:
  local:
  dataDir: /var/lib/etcd
  imageRepository: k8s.gcr.io
  kind: ClusterConfiguration
  kubernetesVersion: v1.18.0
  networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  scheduler: {}
  kubeadm-default-config.yaml组成说明：
InitConfiguration：用于定义一些初始化配置，如初始化使用的token以及apiserver地址等；
ClusterConfiguration：用于定义apiserver、etcd、network、scheduler、controller-manager等master组件相关配置项；
KubeletConfiguration：用于定义kubelet组件相关的配置项；
KubeProxyConfiguration：用于定义kube-proxy组件相关的配置项。
默认的kubeadm-default-config.yaml文件中只有InitConfiguration、ClusterConfiguration两部分。通过如下操作生成另外两部分的示例文件：

生成KubeletConfiguration示例文件: kubeadm-kubelet-config.yaml

kubeadm config print init-defaults --component-configs KubeletConfiguration

生成KubeProxyConfiguration示例文件: kubeadm-kubeproxy-config.yaml

kubeadm config print init-defaults --component-configs KubeProxyConfiguration
根据上述3分示例文件，整合生成kubeadm-config.yaml文件，并根据规划，调整POD、DNS、Cluster IP等配置信息：

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:

groups:
- system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
- signing
- authentication
  kind: InitConfiguration
  localAPIEndpoint:
  APIServer程序绑定的IP, 此处为K8S-PROD-M1节点IP
  
  advertiseAddress: 192.168.122.11
  
  APIServer程序绑定的端口,默认为6443
  
  bindPort: 6443
  nodeRegistration:
  criSocket: /var/run/dockershim.sock
  
  当前节点名称
  
  name: k8s-prod-m1
  taints:
- effect: NoSchedule
  key: node-role.kubernetes.io/master
  
  apiServer:
  
  APIServer相关配置
  
  extraArgs:
  
  审计日志相关配置
  
  audit-log-maxage: "20"
  audit-log-maxbackup: "10"
  audit-log-maxsize: "100"
  audit-log-path: "/var/log/kube-audit/audit.log"
  audit-policy-file: "/etc/kubernetes/audit-policy.yaml"
  audit-log-format: json
  
  开启审计日志配置, 所以需要宿主机上的审计配置
  
  extraVolumes:
- name: "audit-config"
  hostPath: "/etc/kubernetes/audit-policy.yaml"
  mountPath: "/etc/kubernetes/audit-policy.yaml"
  readOnly: true
  pathType: "File"
- name: "audit-log"
  hostPath: "/var/log/kube-audit"
  mountPath: "/var/log/kube-audit"
  pathType: "DirectoryOrCreate"
  timeoutForControlPlane: 4m0s
  apiVersion: kubeadm.k8s.io/v1beta2
  certificatesDir: /etc/kubernetes/pki
  clusterName: kubernetes
  APIServer实际访问地址: VIP:PORT
  
  controlPlaneEndpoint: "192.168.122.40:16443"
  controllerManager: {}
  dns:
  type: CoreDNS
  etcd:
  local:
  
  Etcd Volume 本地路径,最好修改为独立的磁盘
  
  dataDir: /data/apps/etcd
  
  可修改为自己的镜像仓库
  
  imageRepository: harbor.cluster.local/library
  kind: ClusterConfiguration
  
  指定部署版本：v1.18.8
  
  kubernetesVersion: v1.18.8
  networking:
  dnsDomain: cluster.local
  
  SVC子网，SVC IP 地址取值范围
  
  serviceSubnet: 10.96.0.0/12
  
  POD子网，和网络插件配置的要一致
  
  podSubnet: 10.244.0.0/16
  scheduler: {}
  
  Kubelet相关配置
  
  apiVersion: kubelet.config.k8s.io/v1beta1
  kind: KubeletConfiguration
  clusterDNS:
  
  CoreDNS 默认ip地址
10.96.0.10
clusterDomain: cluster.local

Kube-proxy相关配置

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "ipvs"
ipvs:
minSyncPeriod: 5s
syncPeriod: 5s

加权轮询调度

scheduler: "wrr"
配置说明：
配置文加中的192.168.122.40为后续配置的HAProxy API代理IP；
advertiseAddress: 192.168.122.11与bindPort: 6443为程序绑定的地址与端口；
controlPlaneEndpoint: "192.168.122.40:16443" 为实际访问APIServer的地址；
这样配置是为了维持APIServer的HA, 所以LB节点上部署一个HAProxy做4层代理APIServer。

创建日志审计策略文件
vi /etc/kubernetes/audit-policy.yaml

This is required

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:

"RequestReceived"
rules:
level: RequestResponse
resources:
- group: ""
  resources: ["pods"]
level: Metadata
resources:
- group: ""
  resources: ["pods/log", "pods/status"]
level: None
resources:
- group: ""
  resources: ["configmaps"]
  resourceNames: ["controller-leader"]
level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core API group
  resources: ["endpoints", "services"]
level: None
userGroups: ["system:authenticated"]
nonResourceURLs:
- "/api*" # Wildcard matching.
- "/version"
level: Request
resources:
- group: "" # core API group
  resources: ["configmaps"]
  namespaces: ["kube-system"]
level: Metadata
resources:
- group: "" # core API group
  resources: ["secrets", "configmaps"]
level: Request
resources:
- group: "" # core API group
- group: "extensions" # Version of group should NOT be included.
level: Metadata
omitStages:
- "RequestReceived"

准备镜像(可选)

Master和Worker节点上需要准备下面镜像。

需要准备的镜像有

kubead安装后，根据该kubeadm config images list命令判定所需的镜像：

k8s.gcr.io/kube-apiserver:v1.18.8
k8s.gcr.io/kube-controller-manager:v1.18.8
k8s.gcr.io/kube-scheduler:v1.18.8
k8s.gcr.io/kube-proxy:v1.18.8
k8s.gcr.io/pause:3.2
k8s.gcr.io/etcd:3.4.3-0
k8s.gcr.io/coredns:1.6.7
也可根据创建好种子节点的kubeadm-config.yaml文件后，通过下面的命令判断要获取的镜像文件及版本:

获取要下载的镜像

[root@K8S-PROD-M1 workspace]# kubeadm config images list --config kubeadm-config.yaml
W0708 15:55:19.714210 3943 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
k8s.gcr.io/kube-apiserver:v1.18.8
k8s.gcr.io/kube-controller-manager:v1.18.8
k8s.gcr.io/kube-scheduler:v1.18.8
k8s.gcr.io/kube-proxy:v1.18.8
k8s.gcr.io/pause:3.2
k8s.gcr.io/etcd:3.4.3-0
k8s.gcr.io/coredns:1.6.7

根据配置文件下载镜像到本地

kubeadm config images pull --config kubeadm-config.yaml
这种方式在国内没法直接下载镜像，需要*！

Docker Hub上获取

gcr_image_download.sh

#!/bin/bash

images=(
kube-apiserver:v1.18.8
kube-controller-manager:v1.18.8
kube-scheduler:v1.18.8
kube-proxy:v1.18.8
pause:3.2
etcd:3.4.3-0
)

for imageName in ${images[@]} ; do
docker pull aiotceo/$imageName
docker tag aiotceo/$imageName harbor.cluster.local/library/$imageName
docker push harbor.cluster.local/library/$imageName
docker rmi aiotceo/$imageName
done

docker pull coredns/coredns:1.6.7
docker tag coredns/coredns:1.6.7 harbor.cluster.local/library/coredns:1.6.7
docker push harbor.cluster.local/library/coredns:1.6.7
docker rmi coredns/coredns:1.6.7

docker pull quay.io/coreos/flannel:v0.12.0-amd64
docker tag quay.io/coreos/flannel:v0.12.0-amd64 harbor.cluster.local/library/flannel:v0.12.0-amd64
docker push harbor.cluster.local/library/flannel:v0.12.0-amd64
docker rmi quay.io/coreos/flannel:v0.12.0-amd64
说明：私有Harbor搭建请参考: Harbor部署-Docker Compose方式

初始化集群-初始化种子节点
kubeadm init --config=kubeadm-config.yaml --upload-certs

参数说明

--upload-certs: 会在加入master节点的时候自动拷贝证书

初始化输出
...
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.122.40:16443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:fb327a5dba129f7cec83d00f54bef6ee1d475a925aaa5573943a584fce4c23f8

初始化流程

kubeadm init主要执行了以下操作：

[init]

指定版本进行初始化操作；

[preflight]

初始化前的检查和下载所需要的Docker镜像文件。

[kubelet-start]

生成kubelet的配置文件”"var/lib/kubelet/config.yaml"，没有这个文件kubelet无法启动，所以初始化之前的kubelet实际上启动不会成功。

[certificates]

生成Kubernetes使用的证书，存放在/etc/kubernetes/pki目录中。

[kubeconfig]

生成kubeConfig文件，存放在/etc/kubernetes目录中，组件之间通信需要使用对应文件。

[control-plane]

使用/etc/kubernetes/manifest目录下的YAML文件，安装Master 组件。

[etcd]

使用/etc/kubernetes/manifest/etcd.yaml安装Etcd服务。

[wait-control-plane]

等待control-plan部署的Master组件启动。

[apiclient]

检查Master组件服务状态。

[uploadconfig]

更新配置。

[kubelet]

使用ConfigMap配置kubelet。

[patchnode]

更新CNI信息到Node上，通过注释的方式记录。

[mark-control-plane]

为当前节点打标签，打了角色Master，和不可调度标签，这样默认就不会使用Master节点来运行Pod。

[bootstrap-token]

生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到。

[addons]

安装附加组件CoreDNS和kube-proxy

命令行方式

kubeadm init --apiserver-advertise-address=192.168.122.11 --apiserver-bind-port=6443 --control-plane-endpoint=192.168.122.40 --ignore-preflight-errors=swap --image-repository=harbor.cluster.local/library --kubernetes-version=v1.18.8 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --service-dns-domain=cluster.local

配置kubectl
无论在Master节点或Worker节点，要能够执行kubectl命令则必须进行配置，有下面两种配置方式。

通过配置文件
[root@K8S-PROD-M1 ~]# mkdir -p $HOME/.kube
[root@K8S-PROD-M1 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@K8S-PROD-M1 ~]# chown $(id -u):$(id -g) $HOME/.kube/config
通过环境变量
echo 'export KUBECONFIG=/etc/kubernetes/admin.conf' >> ~/.bashrc

source ~/.bashrc
测试kubectl
配置好kubectl后，查看当前集群状态：

[root@K8S-PROD-M1 workspace]# kubectl get no
NAME STATUS ROLES AGE VERSION
K8S-PROD-M1 NotReady master 14m v1.18.4

[root@K8S-PROD-M1 workspace]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused
etcd-0 Healthy {"health":"true"}

[root@K8S-PROD-M1 workspace]# kubectl get all --all-namespaces=true
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-86c44fc579-fqsrl 0/1 Pending 0 14m
kube-system pod/coredns-86c44fc579-kdbhs 0/1 Pending 0 14m
kube-system pod/etcd-K8S-PROD-M1 1/1 Running 0 15m
kube-system pod/kube-apiserver-K8S-PROD-M1 1/1 Running 0 15m
kube-system pod/kube-controller-manager-K8S-PROD-M1 1/1 Running 0 15m
kube-system pod/kube-proxy-d89q2 1/1 Running 1 14m
kube-system pod/kube-scheduler-K8S-PROD-M1 1/1 Running 0 15m

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 15m
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 15m

NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 15m

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/coredns 0/2 2 0 15m

NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/coredns-86c44fc579 2 2 0 14m
由于未安装网络插件，CoreDNS处于pending状态，node处于NotReady状态。执行kubectl get cs命令异常原因：

原因是/etc/kubernetes/manifests下的kube-controller-manager.yaml和kube-scheduler.yaml设置的默认端口是0。
解決方法：

kube-controller-manager

将配置文件(/etc/kubernetes/manifests/kube-controller-manager.yaml)中的--port=0注释掉即可：

[root@K8S-PROD-M1 ~]# vi /etc/kubernetes/manifests/kube-controller-manager.yaml
...

--node-cidr-mask-size=24
#- --port=0
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
...
kube-scheduler

将配置文件(/etc/kubernetes/manifests/kube-scheduler.yaml)中的--port=0注释掉即可：

[root@K8S-PROD-M1 ~]# vi /etc/kubernetes/manifests/kube-scheduler.yaml
...

--leader-elect=true
#- --port=0
image: harbor.cluster.local/library/kube-scheduler:v1.18.8
...
重啓kubelet

systemctl restart kubelet.service

部署网络插件
Kubernetes支持多种网络方案，这里以部署Fannel为例。

获取部署文件
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

修改Flannel配置

更新镜像地址

由于kube-flannel.yml文件指定的镜像从coreos镜像仓库拉取，可能拉取失败，替换成自己的镜像仓库：

查看

[root@K8S-PROD-M1 workspace]# grep -i "flannel:" kube-flannel.yml
image: quay.io/coreos/flannel:v0.12.0-amd64
image: quay.io/coreos/flannel:v0.12.0-amd64

替换

[root@K8S-PROD-M1 workspace]# sed -i 's#quay.io/coreos/flannel:v0.12.0-amd64#harbor.cluster.local/library/flannel:v0.12.0-amd64#' kube-flannel.yml

修改Pods的IP段与运行模式

确保net-conf.json中："Network": "10.244.0.0/16"与规划的一致，"Type": "vxlan"：

...
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
...

删除不需要的DaemonSet

根据OS版本，保留kube-flannel-ds-amd64，删除下面不需要的DaemonSet，：kube-flannel-ds-arm、kube-flannel-ds-arm64、kube-flannel-ds-ppc64le、kube-flannel-ds-s390x.

安装Flannel
[root@K8S-PROD-M1 workspace]# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
查看集群状态
Node和Pod状态，全部OK, 所有的核心组件都正常运行：

[root@K8S-PROD-M1 workspace]# kubectl get no
NAME STATUS ROLES AGE VERSION
K8S-PROD-M1 Ready master 68m v1.18.8

[root@K8S-PROD-M1 workspace]# kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-86c44fc579-fqsrl 1/1 Running 0 68m
coredns-86c44fc579-kdbhs 1/1 Running 0 68m
etcd-K8S-PROD-M1 1/1 Running 0 69m
kube-apiserver-K8S-PROD-M1 1/1 Running 0 69m
kube-controller-manager-K8S-PROD-M1 1/1 Running 0 69m
kube-flannel-ds-amd64-x9qlj 1/1 Running 0 3m59s
kube-proxy-d89q2 1/1 Running 1 68m
kube-scheduler-K8S-PROD-M1 1/1 Running 0 69m
验证网络
[root@K8S-PROD-M1 workspace]# kubectl run flannel-net-test --image=alpine --replicas=1 sleep 3600
[root@K8S-PROD-M1 workspace]# kubectl get pod
NAME READY STATUS RESTARTS AGE
flannel-net-test-7bd9fb9d88-g2kc8 1/1 Running 0 9s

[root@K8S-PROD-M1 workspace]# kubectl exec -it flannel-net-test-7bd9fb9d88-g2kc8 sh
/ # ifconfig
...
完成Kubernetes集群部署
执行kubeadm init命名后有两条kubeadm join命令，其中--control-plane为加入Master节点命令；另外token有时效性，如果提示token失效，请自行创建一个新的token：

Master节点上创建新的join token

[root@K8S-PROD-M1 kubernetes]# kubeadm token create --print-join-command
W0716 14:52:34.678321 27243 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join 192.168.122.60:16443 --token 0wbr4z.cp63s7zins4lkcgh --discovery-token-ca-cert-hash sha256:3abcb890f33bf37c9f0d0232df89e4af77353be3154dcb860080f8f46110cefa

Master节点上创建新的certificate-key

[root@K8S-PROD-M1 ~]# kubeadm init phase upload-certs --upload-certs
W0716 14:57:33.514126 32010 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
be4184ac51619ce7fbfc3d913558a869c75fdfb808c18cdc431d9be457f95b61
如果token超时将产生下面异常：

......
error execution phase preflight: couldn't validate the identity of the API Server: could not find a JWS signature in the cluster-info ConfigMap for token ID "abcdef"
To see the stack trace of this error execute with --v=5 or higher
如果certificate-key时效超时将产生异常：

......
error execution phase control-plane-prepare/download-certs: error downloading certs: error downloading the secret: Secret "kubeadm-certs" was not found in the "kube-system" Namespace. This Secret might have expired. Please, run kubeadm init phase upload-certs --upload-certs on a control plane to generate a new one
To see the stack trace of this error execute with --v=5 or higher
加入Master节点

创建日志审计策略文件

其他两台服务器创建

ssh K8S-PROD-M2 "mkdir -p /etc/kubernetes/"
ssh K8S-PROD-M3 "mkdir -p /etc/kubernetes/"

拷贝日志审计策略文件

K8S-PROD-M2 节点

scp /etc/kubernetes/audit-policy.yaml K8S-PROD-M2:/etc/kubernetes/

K8S-PROD-M3 节点

scp /etc/kubernetes/audit-policy.yaml K8S-PROD-M3:/etc/kubernetes/

分别join master

先测试API Server连通性

curl -k https://127.0.0.1:6443

返回如下信息

{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
"reason": "Forbidden",
"details": {

},
"code": 403
}

增加额外的配置

用于区分不同的Master中的apiserver-advertise-address与apiserver-bind-port：

kubeadm join 192.168.122.40:16443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:fb327a5dba129f7cec83d00f54bef6ee1d475a925aaa5573943a584fce4c23f8 \
--control-plane --certificate-key abdfbadb5acc7c7b7868badf53d323a4fc6deef5402603dd75ce602315a183d5 \
--apiserver-advertise-address 192.168.122.13 \
--apiserver-bind-port 6443
特别注意：添加第二个Master节点后，执行kubectl命令可能会不正常，这是因为此时ETCD只有2个节点，选举时会产生错误，等到第三个Master即节点部署后，集群状态将正常。

设置kubelet开机启动

[root@K8S-PROD-M1 ~]# systemctl enable kubelet.service && systemctl status kubelet.service
[root@K8S-PROD-M2 ~]# systemctl enable kubelet.service && systemctl status kubelet.service
[root@K8S-PROD-M3 ~]# systemctl enable kubelet.service && systemctl status kubelet.service
加入Worker节点
kubeadm join 192.168.122.40:16443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:fb327a5dba129f7cec83d00f54bef6ee1d475a925aaa5573943a584fce4c23f8

设置kubelet开机启动

[root@K8S-PROD-W1 ~]# systemctl enable kubelet.service && systemctl status kubelet.service
[root@K8S-PROD-W2 ~]# systemctl enable kubelet.service && systemctl status kubelet.service
[root@K8S-PROD-W3 ~]# systemctl enable kubelet.service && systemctl status kubelet.service

再次查看集群状态

[root@K8S-PROD-M1 pki]# kubectl get no
NAME STATUS ROLES AGE VERSION
k8s-prod-m1 Ready master 82m v1.18.8
k8s-prod-m2 Ready master 29m v1.18.8
k8s-prod-m3 Ready master 27m v1.18.8
k8s-prod-w1 Ready <none> 5m21s v1.18.8
k8s-prod-w2 Ready <none> 3m53s v1.18.8
k8s-prod-w3 Ready <none> 2m49s v1.18.8

码农公寓

kubeadm-default-config.yaml示例

effect: NoSchedulekey: node-role.kubernetes.io/master

生成KubeletConfiguration示例文件: kubeadm-kubelet-config.yaml

生成KubeProxyConfiguration示例文件: kubeadm-kubeproxy-config.yaml

APIServer程序绑定的IP, 此处为K8S-PROD-M1节点IP

APIServer程序绑定的端口,默认为6443

当前节点名称

effect: NoSchedulekey: node-role.kubernetes.io/master

APIServer相关配置

审计日志相关配置

开启审计日志配置, 所以需要宿主机上的审计配置

APIServer实际访问地址: VIP:PORT

Etcd Volume 本地路径,最好修改为独立的磁盘

可修改为自己的镜像仓库

指定部署版本：v1.18.8

SVC子网，SVC IP 地址取值范围

POD子网，和网络插件配置的要一致

podSubnet: 10.244.0.0/16scheduler: {}

Kubelet相关配置

CoreDNS 默认ip地址

10.96.0.10clusterDomain: cluster.local

Kube-proxy相关配置

加权轮询调度