一、部署Kubelet
1.1 集群规划
主机名 角色 IP
HDSS7-21 kubelet 10.4.7.21
HDSS7-22 kubelet 10.4.7.22
注意：部署以10.4.7.21为例，10.4.7.22节点类似
1.2 签发kubelet证书
证书签发需要在10.4.7.200上操作
[root@hdss7-200 ~]# cd /opt/certs/
注意：将所有可能的kubelet服务器的IP都加进去，后期如果需要再加入其他IP节点的话就需要重新签发此证书，有计划的将证书替换成最新的，最好避免后期加入新的节点。
[root@hdss7-200 certs]# vim kubelet-csr.json
{
"CN": "k8s-kubelet",
"hosts": [
"127.0.0.1",
"10.4.7.10",
"10.4.7.21",
"10.4.7.22",
"10.4.7.23",
"10.4.7.24",
"10.4.7.25",
"10.4.7.26",
"10.4.7.27",
"10.4.7.28"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "beijing",
"L": "beijing",
"O": "od",
"OU": "ops"
}
]
}
生成证书
[root@hdss7-200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kubelet-csr.json | cfssl-json -bare kubelet
注意私钥文件的属性权限是600
certs]# ll kubelet*
-rw-r--r-- 1 root root 1115 6月 10 00:04 kubelet.csr
-rw-r--r-- 1 root root 452 6月 10 00:03 kubelet-csr.json
-rw------- 1 root root 1675 6月 10 00:04 kubelet-key.pem
-rw-r--r-- 1 root root 1468 6月 10 00:04 kubelet.pem
分发证书
[root@hdss7-200 certs]# scp kubelet.pem kubelet-key.pem hdss7-21:/opt/kubernetes/server/bin/certs/
[root@hdss7-200 certs]# scp kubelet.pem kubelet-key.pem hdss7-22:/opt/kubernetes/server/bin/certs/
1.3 创建kubelet的配置
在10.4.7.21,22服务器上操作
1.3.1 set-cluster：创建需要连接的集群信息，可以创建多个k8s信息（会将ca.pem证书编码后嵌入到/opt/kubernetes/conf/kubelet.kubeconfig配置文件中）
注意：10.4.7.10是apiserver的VIP，之前我们在10.4.7.11/21上部署的nginx就是代理10.4.7.21/22的apiserver集群，部署的keepalived的VIP就是10.4.7.10
[root@hdss7-21 ~]# cd /opt/kubernetes/
[root@hdss7-21 conf]# kubectl config set-cluster myk8s --certificate-authority=/opt/kubernetes/server/bin/certs/ca.pem --embed-certs=true --server=https://10.4.7.10:7443 --kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
Cluster "myk8s" set.
[root@hdss7-21 conf]# ll /opt/kubernetes/conf/
总用量 8
-rw-r--r-- 1 root root 2223 6月 8 22:00 audit.yaml
-rw------- 1 root root 1986 6月 10 00:14 kubelet.kubeconfig
1.3.2 set-credentials：创建用户账号，即用户登录的客户端私有证书，可以创建多个证书（将client.pem证书和client-key.pem私钥编码后嵌入到kubelet.kubeconfig文件中）
[root@hdss7-21 conf]# kubectl config set-credentials k8s-node --client-certificate=/opt/kubernetes/server/bin/certs/client.pem --client-key=/opt/kubernetes/server/bin/certs/client-key.pem --embed-certs=true --kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
User "k8s-node" set.
1.3.3 set-context：设置context，即确定账号和集群对应关系
[root@hdss7-21 conf]# kubectl config set-context myk8s-context --cluster=myk8s --user=k8s-node --kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
Context "myk8s-context" created.
1.3.4 use-context：设置当前使用哪个context
[root@hdss7-21 conf]# kubectl config use-context myk8s-context --kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig
Switched to context "myk8s-context".
把此配置传送给10.4.7.22，那么在22上就不需要重复操作以上4个步骤了
[root@hdss7-21 conf]# scp /opt/kubernetes/conf/kubelet.kubeconfig hdss7-22:/opt/kubernetes/conf/
1.4 授权k8s-node用户
此步骤只需要在一台master节点上操作就行（10.4.7.21）
授权k8s-node用户绑定集群角色system:node，让k8s-node拥有具备运算节点的权限
[root@hdss7-21 conf]# vim /opt/kubernetes/conf/k8s-node.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: k8s-node
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: k8s-node
[root@hdss7-21 conf]# kubectl create -f /opt/kubernetes/conf/k8s-node.yaml
clusterrolebinding.rbac.authorization.k8s.io/k8s-node created
创建资源（会存储到etcd中）
[root@hdss7-21 conf]# kubectl get clusterrolebinding k8s-node
NAME AGE
k8s-node 51s
注意：查看7443端口是否正常启动，非常重要，7443端口无法连接会导致node节点无法加入到master节点
~]# telnet 10.4.7.10 7443
Trying 10.4.7.10...
Connected to 10.4.7.10.
Escape character is ‘^]‘.
^]
telnet> q
删除资源命令如下
[root@hdss7-21 conf]# kubectl delete -f k8s-node.yaml
clusterrolebinding.rbac.authorization.k8s.io "k8s-node" deleted
查看资源配置
[root@hdss7-21 conf]# kubectl get clusterrolebinding k8s-node -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: "2021-06-10T13:51:06Z"
name: k8s-node
resourceVersion: "12725"
selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/k8s-node
uid: e70f91af-c9f2-11eb-aaf3-000c29e396b1
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:node
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: k8s-node
1.5 准备pause基础镜像
因为kubelet在启动容器时需要有一个基础镜像初始化网络空间等来帮助我们去启动容器，从而让我们能够启动pod；
将pause镜像放入到harbor私有仓库中，仅在10.4.7.200上操作，确保harbor和docker运行正常，提前检查
下载镜像
[root@hdss7-200 ~]# docker image pull kubernetes/pause
打标签
[root@hdss7-200 ~]# docker image tag kubernetes/pause:latest harbor.od.com/public/pause:latest
登录harbor
[root@hdss7-200 ~]# docker login -u admin harbor.od.com
推送pause镜像到harbor私有仓库
[root@hdss7-200 ~]# docker image push harbor.od.com/public/pause:latest
1.6 创建kubelet启动脚本
在node节点创建启动脚本，并启动kubelet，在10.4.7.21/22上操作，以21为例
22上修改--hostname-override项
[root@hdss7-21 ~]# vim /opt/kubernetes/server/bin/kubelet-startup.sh
#!/bin/sh
WORK_DIR=$(dirname $(readlink -f $0))
[ $? -eq 0 ] && cd $WORK_DIR || exit
/opt/kubernetes/server/bin/kubelet --anonymous-auth=false --cgroup-driver systemd --cluster-dns 192.168.0.2 --cluster-domain cluster.local --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --fail-swap-on="false" --client-ca-file ./certs/ca.pem --tls-cert-file ./certs/kubelet.pem --tls-private-key-file ./certs/kubelet-key.pem --hostname-override hdss7-21.host.com --image-gc-high-threshold 20 --image-gc-low-threshold 10 --kubeconfig ../../conf/kubelet.kubeconfig --log-dir /data/logs/kubernetes/kube-kubelet --pod-infra-container-image harbor.od.com/public/pause:latest --root-dir /data/kubelet
1.7 添加权限，创建目录
[root@hdss7-21 ~]# chmod +x /opt/kubernetes/server/bin/kubelet-startup.sh
[root@hdss7-21 ~]# mkdir -p /data/logs/kubernetes/kube-kubelet /data/kubelet
1.8 配置supervisor配置
[root@hdss7-21 ~]# vim /etc/supervisord.d/kube-kubelet.ini
[program:kube-kubelet-7-21]
command=/opt/kubernetes/server/bin/kubelet-startup.sh
numprocs=1
directory=/opt/kubernetes/server/bin
autostart=true
autorestart=true
startsecs=30
startretries=3
exitcodes=0,2
stopsignal=QUIT
stopwaitsecs=10
user=root
redirect_stderr=true
stdout_logfile=/data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
stdout_logfile_maxbytes=64MB
stdout_logfile_backups=5
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
1.9 启动服务并检查
[root@hdss7-21 ~]# supervisorctl update
kube-kubelet-7-21: added process group
[root@hdss7-21 ~]# supervisorctl status
etcd-server-7-21 RUNNING pid 1172, uptime 1:06:46
kube-apiserver-7-21 RUNNING pid 1183, uptime 1:06:46
kube-controller-manager-7-21 RUNNING pid 1167, uptime 1:06:46
kube-kubelet-7-21 RUNNING pid 2280, uptime 0:01:44
kube-scheduler-7-21 RUNNING pid 1169, uptime 1:06:46
[root@hdss7-22 ~]# tail -100f /data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
I0713 21:44:08.453953 2265 kubelet_node_status.go:75] Successfully registered node hdss7-21.host.com
I0713 21:44:08.509328 2265 cpu_manager.go:155] [cpumanager] starting with none policy
I0713 21:44:08.509382 2265 cpu_manager.go:156] [cpumanager] reconciling every 10s
I0713 21:44:08.509441 2265 policy_none.go:42] [cpumanager] none policy: Start
W0713 21:44:08.644794 2265 manager.go:540] Failed to retrieve checkpoint for "kubelet_internal_checkpoint": checkpoint is not found
I0713 21:44:08.878478 2265 reconciler.go:154] Reconciler: start to sync state
出现如上表示正常启动
查看node节点是否加入到集群中
[root@hdss7-21 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready <none> 13s v1.14.10
hdss7-22.host.com NotReady <none> 0s v1.14.10
别急，需要加载一会
[root@hdss7-21 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready <none> 55m v1.14.10
hdss7-22.host.com Ready <none> 54m v1.14.10
1.10 修改节点角色
[root@hdss7-21 ~]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/node=
node/hdss7-21.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-21.host.com node-role.kubernetes.io/master=
node/hdss7-21.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-22.host.com node-role.kubernetes.io/master=
node/hdss7-22.host.com labeled
[root@hdss7-21 ~]# kubectl label node hdss7-22.host.com node-role.kubernetes.io/node=
node/hdss7-22.host.com labeled
[root@hdss7-21 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com Ready master,node 57m v1.14.10
hdss7-22.host.com Ready master,node 57m v1.14.10
1.11 安装部署其他节点
在10.4.7.22上同样操作
1.12 报错排查
在10.4.7.21（10.4.7.10）master节点上查看node，发现无任何资源可访问，如下
[root@hdss7-21 ~]# kubectl get node
No resources found.
查看kubectl日志
[root@hdss7-21 ~]# tail -100f /data/logs/kubernetes/kube-kubelet/kubelet.stdout.log
第一种如下：
failed to ensure node lease exists connect: no route to host
原因：这是根本就没有10.4.7.10这个ip，或者无法连接到此ip，ping一下，检查是否可以连接，添加此虚IP重新，重新执行本章1.2步骤即可
第二种如下：
E0611 20:41:55.908234 1414 kubelet.go:2246] node "hdss7-22.host.com" not found
E0611 20:41:56.008667 1414 kubelet.go:2246] node "hdss7-22.host.com" not found
这个报错可以忽略
第三种如下：
E0611 20:41:55.838167 1414 reflector.go:126] k8s.io/client-go/informers/factory.go:133: Failed to list *v1beta1.RuntimeClass: Get https://10.4.7.10:7443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 10.4.7.10:7443: connect: connection refused
这就说明虽然找到了这个服务器，但是拒绝连接，测试一下端口的连通性，发现被拒绝，无此端口
[root@hdss7-21 ~]# telnet 10.4.7.10 7443
Trying 10.4.7.10...
telnet: connect to address 10.4.7.10: Connection refused
检查虚IP是否正确配置，检查1.2步骤是否正确执行，在本地telnet一下，是否正确启动7443端口，正常启动后重启一下kube-kubelet-7-21，kube-kubelet-7-22服务即可
正确的日志如下：
I0611 21:06:18.917499 9153 kubelet_node_status.go:72] Attempting to register node hdss7-22.host.com
I0611 21:06:18.947122 9153 kubelet_node_status.go:75] Successfully registered node hdss7-22.host.com
I0611 21:06:18.989477 9153 kubelet.go:1825] skipping pod synchronization - container runtime status check may not have completed yet.
I0611 21:06:19.015529 9153 cpu_manager.go:155] [cpumanager] starting with none policy
I0611 21:06:19.015565 9153 cpu_manager.go:156] [cpumanager] reconciling every 10s
二、部署kube-proxy