1、k8s 集群架构描述
- kubeadm v1.10.2创建k8s集群。
- master节点高可用，三节点（10.18.60.3、10.18.60.4、10.18.60.5）。
- LVS实现master三节点代理。
2、K8S集群证书过期，日志报错如下
Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid
3、故障排定，查看证书
# openssl x509 -noout -text -in apiserver-kubelet-client.crt | grep Not
Not Before: May 22 01:58:06 2018 GMT
Not After : May 22 01:58:07 2019 GMT # 2019-5-22日过期
4、kubeadm 命令介绍（v1.10.2）
注：本次只需用到以下两个参数命令，其它参数不做介绍
# kubeadm alpha phase certs -h # 创建证书 Usage: kubeadm alpha phase certs [command] Available Commands: all Generates all PKI assets necessary to establish the control plane apiserver Generates an API server serving certificate and key apiserver-etcd-client Generates a client certificate for the API server to connect to etcd securely apiserver-kubelet-client Generates a client certificate for the API server to connect to the kubelets securely ca Generates a self-signed kubernetes CA to provision identities for components of the cluster etcd-ca Generates a self-signed CA to provision identities for etcd etcd-healthcheck-client Generates a client certificate for liveness probes to healthcheck etcd etcd-peer Generates an etcd peer certificate and key etcd-server Generates an etcd serving certificate and key front-proxy-ca Generates a front proxy CA certificate and key for a Kubernetes cluster front-proxy-client Generates a front proxy CA client certificate and key for a Kubernetes cluster sa Generates a private key for signing service account tokens along with its public key
# kubeadm alpha phase kubeconfig -h # 生成配置文件（例如：admin.conf|controller-manager.conf|kubelet.conf|scheduler.conf） Usage: kubeadm alpha phase kubeconfig [command] Available Commands: admin Generates a kubeconfig file for the admin to use and for kubeadm itself all Generates all kubeconfig files necessary to establish the control plane and the admin kubeconfig file controller-manager Generates a kubeconfig file for the controller manager to use kubelet Generates a kubeconfig file for the kubelet to use. Please note that this should be used *only* for bootstrapping purposes. scheduler Generates a kubeconfig file for the scheduler to use user Outputs a kubeconfig file for an additional user
# kubeadm alpha phase certs apiserver -h
--apiserver-advertise-address string 填写本机apiserver ip。
--apiserver-cert-extra-sans strings 多master节点，在创建apiserver证书时，需要指定每个节点的IP，代理IP、域名。
--cert-dir string The path where to save the certificates (default "/etc/kubernetes/pki")
--config string Path to kubeadm config file (WARNING: Usage of a configuration file is experimental)
-h, --help help for apiserver
--service-cidr string Alternative range of IP address for service VIPs, from which derives the internal API server VIP that will be added to the API Server serving cert (default "10.96.0.0/12")
--service-dns-domain string Alternative domain for services, to use for the API server serving cert (default "cluster.local")
5、备份节点配置文件与证书
# cp -rfp /etc/kubernetes /etc/kubernetes.2019.5.23
6、创建证书
注：
1、因为之前三个master节点的配置文件中全部填写的是LVS VIP（没做域名解析），为了今后切换方便给VIP配置了一个域名，而apiserver证书中没有配置该域名的认证，所以利用openssl对apiserver证书做了重签替换，并设置apiserver证书有效期10年，所以这次证书过期不涉及apiserver，只需要对apiserver-kubelet-client与front-proxy-client证书重新创建，而下边给出了对apiserver证书重签的命令。
2、创建证书时需要配置VPN，kubeadm需要连接国外
# kubeadm alpha phase certs apiserver --apiserver-advertise-address 10.18.60.3 --apiserver-cert-extra-sans 10.18.60.4 --apiserver-cert-extra-sans 10.18.60.5 --apiserver-cert-extra-sans 10.16.60.6 --apiserver-cert-extra-sans k8s.m.api # 创建apiserver证书 # kubeadm alpha phase certs apiserver-kubelet-client # 创建apiserver-kubelet-client证书 # kubeadm alpha phase certs front-proxy-client # 创建front-proxy-client证书
7、创建配置文件（admin.conf|controller-manager.conf|kubelet.conf|scheduler.conf）
# 创建完会看到/etc/kubernetes下面出现了配置文件 # kubeadm alpha phase kubeconfig all --apiserver-advertise-address 10.18.60.3
8、准备替换
因为我三节点的代理IP配置了域名，所以需要做下替换。
# sed -i 's/10\.18\.60\.3/k8s.m.api/g' admin.conf
# sed -i 's/10\.18\.60\.3/k8s.m.api/g' controller-manager.conf
# sed -i 's/10\.18\.60\.3/k8s.m.api/g' scheduler.conf
# sed -i 's/10\.5\.38\.39/k8s.m.api/g' kubelet.conf
# grep 'host:' /etc/kubernetes/manifests/kube-apiserver.yaml
host: k8s.m.api
# 直接覆盖（注意文件权限） # cp -rfp /etc/kubernetes/admin.conf ~/.kube/config
# kubelet 客户端签发的不需要备份 # rm -rf /var/lib/kubelet/pki/*
9、重启服务
# 重启本机所有docker容器 # docker restart $(docker ps -q) # 重启kubelet # systemctl restart kubelet.service
10、验证
# 可以看到已经恢复 # kubectl get node
11、恢复其它master节点
# 注意拷贝kubelet.conf文件到其它服务器（其它服务器自己生成）
# scp admin.conf controller-manager.conf scheduler.conf 10.18.60.4:/etc/kubernetes # scp admin.conf controller-manager.conf scheduler.conf 10.18.60.5:/etc/kubernetes
# scp apiserver-kubelet-client.crt apiserver-kubelet-client.key front-proxy-client.crt front-proxy-client.key 10.18.60.4:/etc/kubernetes/pki
# scp apiserver-kubelet-client.crt apiserver-kubelet-client.key front-proxy-client.crt front-proxy-client.key 10.18.60.5:/etc/kubernetes/pki
# 其它两个节点只生成各自的kubelet配置文件
# kubeadm alpha phase kubeconfig kubelet
# grep 'server:' kubelet.conf
server: https://k8s.m.api:6443
# 直接覆盖（注意文件权限） # cp -rfp /etc/kubernetes/admin.conf ~/.kube/config
# kubelet 客户端签发的不需要备份 # rm -rf /var/lib/kubelet/pki/*
# 重启本机所有docker容器 # docker restart $(docker ps -q) # 重启kubelet # systemctl restart kubelet.service
# 可以看到已经恢复 # kubectl get node