一台k8s主控节点失去远程连接，重新连接后，发现服务器运行时间变短。另外一台主控节点的运行时间是18天，怀疑这台机器宕机重启了。
[root@hdss7-21 ~]# uptime 15:56:20 up 22:01, 1 user, load average: 0.38, 0.35, 0.41
检查集群运行状态：
[root@hdss7-21 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {"health": "true"}
etcd-0 Healthy {"health": "true"}
etcd-2 Healthy {"health": "true"}
[root@hdss7-21 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
hdss7-21.host.com NotReady master,node 9d v1.15.2
hdss7-22.host.com Ready master,node 9d v1.15.2
[root@hdss7-21 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-ds-bwtkl 1/1 Running 0 6d
nginx-ds-kvhsv 1/1 Running 0 6d
上面的命令输出结果，显示node节点有问题。
发现docker没有启动，重启docker：
[root@hdss7-21 ~]# systemctl status docker * docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled) Active: inactive (dead) Docs: https://docs.docker.com [root@hdss7-21 ~]# systemctl start docker
supervisor的显示也异常：
[root@hdss7-21 ~]# supervisorctl status etcd-server-7-21 RUNNING pid 4008, uptime 22:05:21 kube-apiserver-7-21 RUNNING pid 4021, uptime 22:05:21 kube-controller-manager-7-21 RUNNING pid 4024, uptime 22:05:21 kube-kubelet-7-21 FATAL Exited too quickly (process log may have details) kube-proxy-7-21 RUNNING pid 4028, uptime 22:05:21 kube-scheduler-7-21 RUNNING pid 4032, uptime 22:05:21
使用supervisor把kubelet启动：
[root@hdss7-21 ~]# supervisorctl stop kube-kubelet-7-21 kube-kubelet-7-21: ERROR (not running) [root@hdss7-21 ~]# supervisorctl start kube-kubelet-7-21 kube-kubelet-7-21: started
k8s集群恢复正常：
[root@hdss7-21 ~]# supervisorctl status etcd-server-7-21 RUNNING pid 4008, uptime 22:14:00 kube-apiserver-7-21 RUNNING pid 4021, uptime 22:14:00 kube-controller-manager-7-21 RUNNING pid 4024, uptime 22:14:00 kube-kubelet-7-21 RUNNING pid 126981, uptime 0:00:40 kube-proxy-7-21 RUNNING pid 4028, uptime 22:14:00 kube-scheduler-7-21 RUNNING pid 4032, uptime 22:14:00 [root@hdss7-21 ~]# kubectl get node NAME STATUS ROLES AGE VERSION hdss7-21.host.com Ready master,node 9d v1.15.2 hdss7-22.host.com Ready master,node 9d v1.15.2
学习k8s，老男孩教育的王导值得拥有。
supervisor的原理不太懂，但先搞定问题再说。