Kubernetes全栈架构师（二进制高可用安装k8s集群扩展篇）--学习笔记

2023-07-22 09:24:10

目录

  • 二进制Metrics&Dashboard安装
  • 二进制高可用集群可用性验证
  • 生产环境k8s集群关键性配置
  • Bootstrapping: Kubelet启动过程
  • Bootstrapping: CSR申请和证书颁发原理
  • Bootstrapping: 证书自动续期原理

二进制Metrics&Dashboard安装

  • 安装CoreDNS
  • 安装Metrics Server
  • 安装dashboard

安装CoreDNS

安装对应版本（推荐）

cd /root/k8s-ha-install/

如果更改了k8s service的网段需要将coredns的serviceIP改成k8s service网段的第十个IP

sed -i "s#10.96.0.10#10.96.0.10#g" CoreDNS/coredns.yaml

安装coredns

kubectl create -f CoreDNS/coredns.yaml

安装最新版CoreDNS（不推荐）

git clone https://github.com/coredns/deployment.git

cd deployment/kubernetes

# ./deploy.sh -s -i 10.96.0.10 | kubectl apply -f -

serviceaccount/coredns created

clusterrole.rbac.authorization.k8s.io/system:coredns created

clusterrolebinding.rbac.authorization.k8s.io/system:coredns created

configmap/coredns created

deployment.apps/coredns created

service/kube-dns created

查看状态

kubectl get po -n kube-system -l k8s-app=kube-dns

状态

NAME                      READY   STATUS    RESTARTS   AGE

coredns-fb4874468-nr5nx   1/1     Running   0          49s

强制删除一直处于Terminating的pod

[root@k8s-master01 ~]# kubectl get po -n kube-system -l k8s-app=kube-dns

NAME                      READY   STATUS        RESTARTS   AGE

coredns-fb4874468-fgs2h   1/1     Terminating   0          6d20h

[root@k8s-master01 ~]# kubectl delete pods coredns-fb4874468-fgs2h --grace-period=0 --force -n kube-system

warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.

pod "coredns-fb4874468-fgs2h" force deleted

[root@k8s-master01 ~]# kubectl get po -n kube-system -l k8s-app=kube-dns

No resources found in kube-system namespace.

安装Metrics Server

在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率。

安装metrics server

cd /root/k8s-ha-install/metrics-server-0.4.x/

kubectl  create -f .

等待metrics server启动然后查看状态

kubectl  top node

节点状态

NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%

k8s-master01   263m         13%    1239Mi          66%

k8s-master02   213m         10%    1065Mi          57%

k8s-master03   207m         10%    1050Mi          56%

k8s-node01     89m          4%     514Mi           27%

k8s-node02     158m         7%     493Mi           26%

查看pod状态

kubectl  top po -A

pod状态

NAMESPACE     NAME                                      CPU(cores)   MEMORY(bytes)

kube-system   calico-kube-controllers-cdd5755b9-4fzg9   3m           18Mi

kube-system   calico-node-8xg62                         26m          60Mi

kube-system   calico-node-dczxz                         24m          60Mi

kube-system   calico-node-gn8ws                         23m          62Mi

kube-system   calico-node-qmwkd                         26m          60Mi

kube-system   calico-node-zfw8n                         25m          59Mi

kube-system   coredns-fb4874468-nr5nx                   3m           10Mi

kube-system   metrics-server-64c6c494dc-9x727           2m           18Mi

安装dashboard

  • 安装指定版本dashboard
  • 安装最新版dashboard
  • 登录dashboard

Dashboard用于展示集群中的各类资源，同时也可以通过Dashboard实时查看Pod的日志和在容器中执行一些命令等。

安装指定版本dashboard

cd /root/k8s-ha-install/dashboard/

kubectl  create -f .

安装最新版dashboard

官方GitHub地址：https://github.com/kubernetes/dashboard

可以在官方dashboard查看到最新版dashboard

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.3.1/aio/deploy/recommended.yaml

创建管理员用户

vim admin.yaml

# 添加以下内容

apiVersion: v1

kind: ServiceAccount

metadata:

  name: admin-user

  namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

  name: admin-user

  annotations:

    rbac.authorization.kubernetes.io/autoupdate: "true"

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: cluster-admin

subjects:

- kind: ServiceAccount

  name: admin-user

  namespace: kube-system

执行

kubectl apply -f admin.yaml -n kube-system

登录dashboard

在谷歌浏览器（Chrome）启动文件中加入启动参数，用于解决无法访问Dashboard的问题，因为使用的证书是自签名（属性->快捷方式->目标，粘贴到最后）

 --test-type --ignore-certificate-errors

更改dashboard的svc为NodePort：

kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard

修改 type: ClusterIP 为 type:NodePort

修改完成之后会暴露一个端口号，查看端口号：

kubectl get svc kubernetes-dashboard -n kubernetes-dashboard

端口号

NAME                   TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE

kubernetes-dashboard   NodePort   10.108.217.183   <none>        443:31874/TCP   9m37s

根据自己的实例端口号，通过任意安装了kube-proxy的宿主机或者VIP的IP+端口即可访问到dashboard：访问Dashboard：https://192.168.232.236:31874（请更改18282为自己的端口），选择登录方式为令牌（即token方式）

查看token值：

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

token值

Name:         admin-user-token-9c4tz

Namespace:    kube-system

Labels:       <none>

Annotations:  kubernetes.io/service-account.name: admin-user

              kubernetes.io/service-account.uid: d1f2e528-0ef8-4c6b-a384-a18fbca6bc54

Type:  kubernetes.io/service-account-token

Data

====

ca.crt:     1411 bytes

namespace:  11 bytes

token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IlNCbEdFa1RQZElhbTBRb29aTTNCTUE1dTJ2enBCeGZxMWJwbmpfZHBXdkEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLTljNHR6Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJkMWYyZTUyOC0wZWY4LTRjNmItYTM4NC1hMThmYmNhNmJjNTQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.KFH5ed0kJEaU1HSpxkitJxqKJGnSNAWogNSGjGn1wEh7R9zKYkAfNLES6Vl3GU9jvxBCEZW415ZFILr96kpgl_88mD-K-AMgQxKLdpghYDx_CnsLtI6e8rLTNkaPS2Uo3sYAy9U280Niop14Yzuar5FQ3AfSbeXGcF_9Jrgyeh5XWPA0h69Au8pUEOkVdpADmuIaFSqfTnmkOSdGqCgFb_QsUqvjo4ifIxKnN6uW8wfR1s4esWkPq569xhCINaUY6g3rnT1jfVTU2XmrURrKOVok0OfSmtXTKCSs2jliEdmx7qEFTrw2KCPnTfORUtTnmdZ2ZnGGx9Fvf_hGaKk1FQ

二进制高可用集群可用性验证

安装busybox

[root@k8s-master01 ~]# cat<<EOF | kubectl apply -f -

apiVersion: v1

kind: Pod

metadata:

  name: busybox

  namespace: default

spec:

  containers:

  - name: busybox

    image: busybox:1.28

    command:

      - sleep

      - "3600"

    imagePullPolicy: IfNotPresent

  restartPolicy: Always

EOF

查看状态

[root@k8s-master01 ~]# kubectl get po

NAME      READY   STATUS    RESTARTS   AGE

busybox   1/1     Running   0          29s
  • Pod必须能解析Service
  • Pod必须能解析跨namespace的Service
  • 每个节点都必须要能访问Kubernetes的kubernetes svc 443和kube-dns的service 53
  • Pod和Pod之前要能通（同namespace能通信、跨namespace能通信、跨机器能通信）

集群安装成功后默认的kubernetes server

[root@k8s-master01 ~]# kubectl get svc

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE

kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d22h

Pod必须能解析Service

[root@k8s-master01 ~]# kubectl exec  busybox -n default -- nslookup kubernetes

Server:    10.96.0.10

Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes

Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

Pod必须能解析跨namespace的Service

[root@k8s-master01 ~]# kubectl exec  busybox -n default -- nslookup kube-dns.kube-system

Server:    10.96.0.10

Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kube-dns.kube-system

Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

每个节点都必须要能访问Kubernetes的kubernetes svc 443和kube-dns的service 53（发送键输入到所有的会话）

[root@k8s-master01 ~]# yum install telnet -y

[root@k8s-master01 ~]# telnet 10.96.0.1 443

Trying 10.96.0.1...

Connected to 10.96.0.1.

Escape character is '^]'.

[root@k8s-master01 ~]# kubectl get svc -n kube-system

NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE

kube-dns         ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP   53m

metrics-server   ClusterIP   10.107.95.145   <none>        443/TCP                  4h33m

You have new mail in /var/spool/mail/root

[root@k8s-master01 ~]# telnet 10.96.0.10 53

Trying 10.96.0.10...

Connected to 10.96.0.10.

Escape character is '^]'.

[root@k8s-master01 ~]# curl 10.96.0.10:53

curl: (52) Empty reply from server

Pod和Pod之前要能通（同namespace能通信、跨namespace能通信、跨机器能通信）（取消发送键输入到所有的会话）

[root@k8s-master01 ~]# kubectl get po -n kube-system -owide

NAME                                      READY   STATUS    RESTARTS        AGE     IP                NODE           NOMINATED NODE   READINESS GATES

calico-kube-controllers-cdd5755b9-4fzg9   1/1     Running   0               4h59m   192.168.232.131   k8s-node01     <none>           <none>

calico-node-8xg62                         1/1     Running   0               4h59m   192.168.232.129   k8s-master02   <none>           <none>

calico-node-dczxz                         1/1     Running   0               4h59m   192.168.232.131   k8s-node01     <none>           <none>

calico-node-gn8ws                         1/1     Running   0               4h59m   192.168.232.128   k8s-master01   <none>           <none>

calico-node-qmwkd                         1/1     Running   0               4h59m   192.168.232.130   k8s-master03   <none>           <none>

calico-node-zfw8n                         1/1     Running   2 (4h59m ago)   4h59m   192.168.232.132   k8s-node02     <none>           <none>

coredns-fb4874468-fgs2h                   1/1     Running   0               56m     172.25.92.66      k8s-master02   <none>           <none>

metrics-server-64c6c494dc-9x727           1/1     Running   0               4h35m   172.27.14.193     k8s-node02     <none>           <none>

# 进入k8s-master02

[root@k8s-master01 ~]# kubectl exec -ti calico-node-8xg62  -n kube-system -- bash

Defaulted container "calico-node" out of: calico-node, install-cni (init), flexvol-driver (init)

# ping k8s-master03

[root@k8s-master02 /]# ping 192.168.232.130

PING 192.168.232.130 (192.168.232.130) 56(84) bytes of data.

64 bytes from 192.168.232.130: icmp_seq=1 ttl=64 time=0.416 ms

64 bytes from 192.168.232.130: icmp_seq=2 ttl=64 time=0.240 ms

64 bytes from 192.168.232.130: icmp_seq=3 ttl=64 time=0.191 ms

# 退出k8s-master02

[root@k8s-master02 /]# exit

exit

[root@k8s-master01 ~]# kubectl get po -owide

NAME      READY   STATUS    RESTARTS   AGE   IP             NODE           NOMINATED NODE   READINESS GATES

busybox   1/1     Running   0          22m   172.25.92.69   k8s-master02   <none>           <none>

# 进入k8s-master02容器ping k8s-master03

[root@k8s-master01 ~]# kubectl exec -ti busybox -- sh

/ # ping 192.168.232.130

PING 192.168.232.130 (192.168.232.130): 56 data bytes

64 bytes from 192.168.232.130: seq=0 ttl=63 time=0.329 ms

64 bytes from 192.168.232.130: seq=1 ttl=63 time=0.452 ms

64 bytes from 192.168.232.130: seq=2 ttl=63 time=0.675 ms

创建一个带有三个副本的deployment

[root@k8s-master01 ~]# kubectl create deploy nginx --image=nginx --replicas=3

deployment.apps/nginx created

[root@k8s-master01 ~]# kubectl get deploy

NAME    READY   UP-TO-DATE   AVAILABLE   AGE

nginx   2/3     3            2           35s

# 查看部署所在的NODE

[root@k8s-master01 ~]# kubectl get po -owide

NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE           NOMINATED NODE   READINESS GATES

busybox                  1/1     Running   0          28m     172.25.92.69     k8s-master02   <none>           <none>

nginx                    1/1     Running   0          2m37s   172.18.195.4     k8s-master03   <none>           <none>

nginx-6799fc88d8-lbhgm   1/1     Running   0          54s     172.25.244.197   k8s-master01   <none>           <none>

nginx-6799fc88d8-nq2gz   1/1     Running   0          54s     172.17.125.1     k8s-node01     <none>           <none>

nginx-6799fc88d8-tzgz8   1/1     Running   0          54s     172.27.14.194    k8s-node02     <none>           <none>

# 删除

[root@k8s-master01 ~]# kubectl delete deploy nginx

deployment.apps "nginx" deleted

[root@k8s-master01 ~]# kubectl delete po busybox

生产环境k8s集群关键性配置

  • docker配置
  • controller-manager配置
  • kubelet配置
  • kubelet-conf.yml
  • 安装总结

docker配置

（发送键输入到所有的会话）

vim /etc/docker/daemon.json

{

    "registry-mirrors": [

    "https://registry.docker-cn.com",

    "http://hub-mirror.c.163.com",

    "https://docker.mirrors.ustc.edu.cn"

  ],

 "exec-opts": ["native.cgroupdriver=systemd"],

 "max-concurrent-downloads": 10, # 并发下载的线程数

 "max-concurrent-uploads": 5, # 并发上传的线程数

 "log-opts": {

     "max-size": "300m", # 限制日志文件最大容量，超过则分割

     "max-file": "2" # 日志保存最大数量

 },

 "live-restore": true # 更改docker配置之后需要重启docker才能生效，这个参数可以使得重启docker不影响正在运行的容器进程

}

更新配置

systemctl daemon-reload

controller-manager配置

注释新版本已经默认配置，设置证书过期时间

（发送键输入到所有的会话，取消node节点）

vim /usr/lib/systemd/system/kube-controller-manager.service

# --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true \

--cluster-signing-duration=876000h0m0s \

更新配置

systemctl daemon-reload

systemctl restart kube-controller-manager

kubelet配置

（发送键输入到所有的会话）

vim /etc/systemd/system/kubelet.service.d/10-kubelet.conf

[Service]

Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.kubeconfig --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig"

Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"

Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml  --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"

Environment="KUBELET_EXTRA_ARGS=--tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384    --image-pull-progress-deadline=30m"

ExecStart=

ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

k8s默认加密方式会被识别为漏洞，需要修改加密方式

--tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

kubelet下载镜像的deadline，避免重复循环加载

--image-pull-progress-deadline=30m

kubelet-conf.yml

vim /etc/kubernetes/kubelet-conf.yml

# 添加如下配置

rotateServerCertificates: true

allowedUnsafeSysctls: # 允许修改内核，才能修改内核参数，比如修改最大并发量，但是涉及到安全问题，所以按需配置

 - "net.core*"

 - "net.ipv4.*"

kubeReserved: # 预留资源，生产环境需要设置高一点，预留足够资源

  cpu: "10m"

  memory: 10Mi

  ephemeral-storage: 10Mi

systemReserved:

  cpu: "10m"

  memory: 20Mi

  ephemeral-storage: 1Gi

更新配置

systemctl daemon-reload

systemctl restart kubelet

查看日志

tail -f /var/log/messages

验证

[root@k8s-master01 ~]# kubectl get node

NAME           STATUS   ROLES    AGE    VERSION

k8s-master01   Ready    <none>   7h3m   v1.22.0-beta.1

k8s-master02   Ready    <none>   7h3m   v1.22.0-beta.1

k8s-master03   Ready    <none>   7h3m   v1.22.0-beta.1

k8s-node01     Ready    <none>   7h3m   v1.22.0-beta.1

k8s-node02     Ready    <none>   7h3m   v1.22.0-beta.1

安装总结

  • kubeadm
  • 二进制
  • 自动化安装
  • 安装需要注意的细节

自动化安装（Ansible）

  • Master节点安装不需要写自动化。
  • 添加Node节点，playbook。

安装需要注意的细节

  • 上面的细节配置
  • 生产环境中etcd一定要和系统盘分开，一定要用ssd硬盘。
  • Docker数据盘也要和系统盘分开，有条件的话可以使用ssd硬盘

Bootstrapping: Kubelet启动过程

Bootstrapping：自动为node节点颁发证书

二进制高可用安装生成证书的时候，我们为每个k8s组件都生成了一个kubeconfig文件，比如controller-manager.kubeconfig，配置中保存了apiserver的一些信息，以及它去连接apiserver的证书

kube-controller-manager.service指定了kubeconfig，组件启动的时候会通过config文件去连接apiserver进行认证，通讯

[root@k8s-master01 kubernetes]# vim /usr/lib/systemd/system/kube-controller-manager.service

[Unit]

Description=Kubernetes Controller Manager

Documentation=https://github.com/kubernetes/kubernetes

After=network.target

[Service]

ExecStart=/usr/local/bin/kube-controller-manager \

      --v=2 \

      --logtostderr=true \

      --address=127.0.0.1 \

      --root-ca-file=/etc/kubernetes/pki/ca.pem \

      --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \

      --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \

      --service-account-private-key-file=/etc/kubernetes/pki/sa.key \

      --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \ # 指定了kubeconfig

      --leader-elect=true \

      --use-service-account-credentials=true \

      --node-monitor-grace-period=40s \

      --node-monitor-period=5s \

      --pod-eviction-timeout=2m0s \

      --controllers=*,bootstrapsigner,tokencleaner \

      --allocate-node-cidrs=true \

      --cluster-cidr=172.16.0.0/12 \

      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \

      --node-cidr-mask-size=24

Restart=always

RestartSec=10s

[Install]

WantedBy=multi-user.target

kube-controller-manager.service和kube-scheduler.service是我们自己颁发证书，但是kubelet不建议通过这种方式，建议使用TLS BootStrapping，它会自动地为node节点的kubelet颁发证书，那么它是如何生成的呢？

使用node02节点举例

[root@k8s-node02 ~]# cd /etc/kubernetes

[root@k8s-node02 kubernetes]# ls

bootstrap-kubelet.kubeconfig  kubelet-conf.yml  kubelet.kubeconfig  kube-proxy.conf  kube-proxy.kubeconfig  manifests  pki

当我们配置一个节点的时候，节点上面是没有kubelet.kubeconfig文件的，它是通过bootstrap-kubelet.kubeconfig与apiserver进行交互，然后自动申请了一个kubelet.kubeconfig文件，kubelet启动的时候如果缺少kubelet.kubeconfig文件，就会这样申请

删除kubelet.kubeconfig

[root@k8s-node02 kubernetes]# rm -rf kubelet.kubeconfig

查看kubelet配置文件

[root@k8s-node02 kubernetes]# cat /etc/systemd/system/kubelet.service.d/10-kubelet.conf

[Service]

Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig"

Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"

Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"

Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384    --image-pull-progress-deadline=30m" "

ExecStart=

ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

在配置中可以看到指定了bootstrap-kubeconfig文件

--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

指定了kubeconfig文件，但是这个文件是没有的，kubelet启动的时候会通过前面描述的方式申请一个新的kubelet.kubeconfig

--kubeconfig=/etc/kubernetes/kubelet.kubeconfig

重启kubelet，可以看到生成了kubelet.kubeconfig证书，生成之后就可以和apiserver进行交互

[root@k8s-node02 kubernetes]# systemctl restart kubelet

[root@k8s-node02 kubernetes]# ls

bootstrap-kubelet.kubeconfig  kubelet-conf.yml  kubelet.kubeconfig  kube-proxy.conf  kube-proxy.kubeconfig  manifests  pki

TLS BootStrapping 官方文档：https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#initialization-process

在一个k8s集群中，工作节点的组件kubelet和kube-proxy需要连接master节点的组件，尤其是apiserver。为了确保连接是私有的，强烈建议为每一个客户端配置一个TLS证书在所有的节点上

kubelet启动过程

  • 查找kubeconfig文件，文件一般位于/etc/kubernetes/kubelet.kubeconfig
  • 从kubeconfig文件中检索APIServer的URL和证书
  • 然后去和APIServer进行交互

查看证书

[root@k8s-node02 kubernetes]# more kubelet.kubeconfig

apiVersion: v1

clusters:

- cluster:

    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ1RENDQXN5Z0F3SUJBZ0lVYlFPRnd0dDVmdTlFZndnNUFjMnB2a1JYWWFFd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2R6

RUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEV6QVJCZ05WQkFvVENrdDFZbVZ5Ym1WMFpYTXhHakFZQmdOVkJBc1RFVXQxWW1WeWJtVjBaWE10

CmJXRnVkV0ZzTVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1DQVhEVEl4TURjd09UQTNNelV3TUZvWUR6SXgKTWpFd05qRTFNRGN6TlRBd1dqQjNNUXN3Q1FZRFZRUUdFd0pEVGpFUU1BNEdBMVVFQ0JNSFFtVnBh

bWx1WnpFUQpNQTRHQTFVRUJ4TUhRbVZwYW1sdVp6RVRNQkVHQTFVRUNoTUtTM1ZpWlhKdVpYUmxjekVhTUJnR0ExVUVDeE1SClMzVmlaWEp1WlhSbGN5MXRZVzUxWVd3eEV6QVJCZ05WQkFNVENtdDFZbVZ5Ym1WMFpY

TXdnZ0VpTUEwR0NTcUcKU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQzhaMUgya2d2QTltZDVnQ0Z3TjNSeGFOck9KQ3F3ampJOApNRG5TZkk2ZldzcjlDR2dvb0VpQTZRdDk0c2szcDNkMkZ0c3hNaWNkdHRO

QVJZQWJQU3JSdkZBdGhkeGpvNWVCCjFTYVFXcXh5ckp3ZFN4UW5hUkMyaXZPRm55NUNmU0VOekYyVnBOQVIwVTVLUjRLWko2MzQyQk1yYzF3aEE5VjkKd05aaXVySi95emZPSTY5dzJaWUFwQTVYaldDSXczOXQzdjBr

WVM3clZkQkhkMWFnUzJMcWNMb0dlZnJvT1BzZgp1ZEZCa0tkbnE5T0tLejRSVHpsQ2hnMXFTSk1wK0xmT1hiYTUzRmQ5c01WRFhvS3lDQVk4N1E3RHltWmlsZ2F5Cjl3YTFGZnNXajRKQzJzVy9lSWtWczQzazV4QVR2

ems0ZmF6QVhZZDdvUXUrbStKZUZvL1JBZ01CQUFHalpqQmsKTUE0R0ExVWREd0VCL3dRRUF3SUJCakFTQmdOVkhSTUJBZjhFQ0RBR0FRSC9BZ0VDTUIwR0ExVWREZ1FXQkJTLwpnN2xDaXBuZlkycGhVbWY4RXdXdE5o

ZE9LREFmQmdOVkhTTUVHREFXZ0JTL2c3bENpcG5mWTJwaFVtZjhFd1d0Ck5oZE9LREFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBamdRcE1hbndsaHN3dkNRWkd4NitwTXdqVm00LzNQbnYKcWZNU1F0YmVFWEpPRndY

WGN5T0h0clphRVNrbTV2eHZiWkRwMXZFcFc0aWN0V3U4emhGS3JKQWtRN0NKNHZDaAo3THBnVUdFSFFzTkJEU09reUxPcUhEN1RNOHFZV3hvWUZPbWFhdm1KelMwbFhZRkx1VmNYTm1GTTBPWlRsL01OCjVpdE5vMEVz

bWxBaUVYTFpJRkdpL0dZQkZXRHBQNVB2SlJIdUptd2JVQTdiMkNoVmhreFBXa0FYaDgvZjNzMjUKOWZSTC9ya0VTMGJqQVlHQy9lTGRDL09uUng2VFRyMHVRTUFSVjBqZGs3Qkg1dElXQjFtakthNjViR1Z6SlBZYQox

anNIRVhvOUo0MnBaQ3lWNnQxTklzRkFrL3pyazJzSFZHbFZzUzB4ZHpPaDhZaEhsKzVwVEE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

    server: https://192.168.232.236:8443

  name: default-cluster

contexts:

- context:

    cluster: default-cluster

    namespace: default

    user: default-auth

  name: default-context

current-context: default-context

kind: Config

preferences: {}

users:

- name: default-auth

  user:

    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem

    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

查看kubelet.kubeconfig证书有效期：

[root@k8s-node02 kubernetes]# echo "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ1RENDQXN5Z0F3SUJBZ0lVYlFPRnd0dDVmdTlFZndnNUFjMnB2a1JYWWFFd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2R6

RUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEV6QVJCZ05WQkFvVENrdDFZbVZ5Ym1WMFpYTXhHakFZQmdOVkJBc1RFVXQxWW1WeWJtVjBaWE10

CmJXRnVkV0ZzTVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1DQVhEVEl4TURjd09UQTNNelV3TUZvWUR6SXgKTWpFd05qRTFNRGN6TlRBd1dqQjNNUXN3Q1FZRFZRUUdFd0pEVGpFUU1BNEdBMVVFQ0JNSFFtVnBh

bWx1WnpFUQpNQTRHQTFVRUJ4TUhRbVZwYW1sdVp6RVRNQkVHQTFVRUNoTUtTM1ZpWlhKdVpYUmxjekVhTUJnR0ExVUVDeE1SClMzVmlaWEp1WlhSbGN5MXRZVzUxWVd3eEV6QVJCZ05WQkFNVENtdDFZbVZ5Ym1WMFpY

TXdnZ0VpTUEwR0NTcUcKU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQzhaMUgya2d2QTltZDVnQ0Z3TjNSeGFOck9KQ3F3ampJOApNRG5TZkk2ZldzcjlDR2dvb0VpQTZRdDk0c2szcDNkMkZ0c3hNaWNkdHRO

QVJZQWJQU3JSdkZBdGhkeGpvNWVCCjFTYVFXcXh5ckp3ZFN4UW5hUkMyaXZPRm55NUNmU0VOekYyVnBOQVIwVTVLUjRLWko2MzQyQk1yYzF3aEE5VjkKd05aaXVySi95emZPSTY5dzJaWUFwQTVYaldDSXczOXQzdjBr

WVM3clZkQkhkMWFnUzJMcWNMb0dlZnJvT1BzZgp1ZEZCa0tkbnE5T0tLejRSVHpsQ2hnMXFTSk1wK0xmT1hiYTUzRmQ5c01WRFhvS3lDQVk4N1E3RHltWmlsZ2F5Cjl3YTFGZnNXajRKQzJzVy9lSWtWczQzazV4QVR2

ems0ZmF6QVhZZDdvUXUrbStKZUZvL1JBZ01CQUFHalpqQmsKTUE0R0ExVWREd0VCL3dRRUF3SUJCakFTQmdOVkhSTUJBZjhFQ0RBR0FRSC9BZ0VDTUIwR0ExVWREZ1FXQkJTLwpnN2xDaXBuZlkycGhVbWY4RXdXdE5o

ZE9LREFmQmdOVkhTTUVHREFXZ0JTL2c3bENpcG5mWTJwaFVtZjhFd1d0Ck5oZE9LREFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBamdRcE1hbndsaHN3dkNRWkd4NitwTXdqVm00LzNQbnYKcWZNU1F0YmVFWEpPRndY

WGN5T0h0clphRVNrbTV2eHZiWkRwMXZFcFc0aWN0V3U4emhGS3JKQWtRN0NKNHZDaAo3THBnVUdFSFFzTkJEU09reUxPcUhEN1RNOHFZV3hvWUZPbWFhdm1KelMwbFhZRkx1VmNYTm1GTTBPWlRsL01OCjVpdE5vMEVz

bWxBaUVYTFpJRkdpL0dZQkZXRHBQNVB2SlJIdUptd2JVQTdiMkNoVmhreFBXa0FYaDgvZjNzMjUKOWZSTC9ya0VTMGJqQVlHQy9lTGRDL09uUng2VFRyMHVRTUFSVjBqZGs3Qkg1dElXQjFtakthNjViR1Z6SlBZYQox

anNIRVhvOUo0MnBaQ3lWNnQxTklzRkFrL3pyazJzSFZHbFZzUzB4ZHpPaDhZaEhsKzVwVEE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

U5EIENFUlRJRklDQVRFLS0tLS0K" | base64 --decode >/tmp/1

然后使用OpenSSL即可查看证书过期时间（100年有效期）：

[root@k8s-node02 kubernetes]# openssl x509 -in /tmp/1 -noout -dates

notBefore=Jul  9 07:35:00 2021 GMT

notAfter=Jun 15 07:35:00 2121 GMT

Bootstrapping: CSR申请和证书颁发原理

官方文档：https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#bootstrap-initialization

1、 Kubelet启动

2、 Kubelet查看kubelet.kubeconfig文件，假设没有这个文件

3、 Kubelet会查看本地的bootstrap.kubeconfig

4、 Kubelet读取bootstrap.kubeconfig文件，检索apiserver的url和一个token

5、 Kubelet链接apiserver，使用这个token进行认证

a) Apiserver会识别tokenid，apiserver会查看该tokenid对于的bootstrap的一个secret

secret就是二进制高可用安装集群的时候创建的bootstrap.secret.yaml

如果要修改bootstrap.secret.yaml的token-id和token-secret，需要保证 c8ad9c 字符串一致的，并且位数是一样的。还要保证上个命令的黄色字体：c8ad9c.2e4d610cf3e7426e与你修改的字符串要一致

[root@k8s-master01 kubernetes]# more bootstrap-kubelet.kubeconfig

apiVersion: v1

clusters:

- cluster:

    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQ1RENDQXN5Z0F3SUJBZ0lVYlFPRnd0dDVmdTlFZndnNUFjMnB2a1JYWWFFd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2R6

RUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFXcHBibWN4RURBT0JnTlZCQWNUQjBKbAphV3BwYm1jeEV6QVJCZ05WQkFvVENrdDFZbVZ5Ym1WMFpYTXhHakFZQmdOVkJBc1RFVXQxWW1WeWJtVjBaWE10

CmJXRnVkV0ZzTVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1DQVhEVEl4TURjd09UQTNNelV3TUZvWUR6SXgKTWpFd05qRTFNRGN6TlRBd1dqQjNNUXN3Q1FZRFZRUUdFd0pEVGpFUU1BNEdBMVVFQ0JNSFFtVnBh

bWx1WnpFUQpNQTRHQTFVRUJ4TUhRbVZwYW1sdVp6RVRNQkVHQTFVRUNoTUtTM1ZpWlhKdVpYUmxjekVhTUJnR0ExVUVDeE1SClMzVmlaWEp1WlhSbGN5MXRZVzUxWVd3eEV6QVJCZ05WQkFNVENtdDFZbVZ5Ym1WMFpY

TXdnZ0VpTUEwR0NTcUcKU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQzhaMUgya2d2QTltZDVnQ0Z3TjNSeGFOck9KQ3F3ampJOApNRG5TZkk2ZldzcjlDR2dvb0VpQTZRdDk0c2szcDNkMkZ0c3hNaWNkdHRO

QVJZQWJQU3JSdkZBdGhkeGpvNWVCCjFTYVFXcXh5ckp3ZFN4UW5hUkMyaXZPRm55NUNmU0VOekYyVnBOQVIwVTVLUjRLWko2MzQyQk1yYzF3aEE5VjkKd05aaXVySi95emZPSTY5dzJaWUFwQTVYaldDSXczOXQzdjBr

WVM3clZkQkhkMWFnUzJMcWNMb0dlZnJvT1BzZgp1ZEZCa0tkbnE5T0tLejRSVHpsQ2hnMXFTSk1wK0xmT1hiYTUzRmQ5c01WRFhvS3lDQVk4N1E3RHltWmlsZ2F5Cjl3YTFGZnNXajRKQzJzVy9lSWtWczQzazV4QVR2

ems0ZmF6QVhZZDdvUXUrbStKZUZvL1JBZ01CQUFHalpqQmsKTUE0R0ExVWREd0VCL3dRRUF3SUJCakFTQmdOVkhSTUJBZjhFQ0RBR0FRSC9BZ0VDTUIwR0ExVWREZ1FXQkJTLwpnN2xDaXBuZlkycGhVbWY4RXdXdE5o

ZE9LREFmQmdOVkhTTUVHREFXZ0JTL2c3bENpcG5mWTJwaFVtZjhFd1d0Ck5oZE9LREFOQmdrcWhraUc5dzBCQVFzRkFBT0NBUUVBamdRcE1hbndsaHN3dkNRWkd4NitwTXdqVm00LzNQbnYKcWZNU1F0YmVFWEpPRndY

WGN5T0h0clphRVNrbTV2eHZiWkRwMXZFcFc0aWN0V3U4emhGS3JKQWtRN0NKNHZDaAo3THBnVUdFSFFzTkJEU09reUxPcUhEN1RNOHFZV3hvWUZPbWFhdm1KelMwbFhZRkx1VmNYTm1GTTBPWlRsL01OCjVpdE5vMEVz

bWxBaUVYTFpJRkdpL0dZQkZXRHBQNVB2SlJIdUptd2JVQTdiMkNoVmhreFBXa0FYaDgvZjNzMjUKOWZSTC9ya0VTMGJqQVlHQy9lTGRDL09uUng2VFRyMHVRTUFSVjBqZGs3Qkg1dElXQjFtakthNjViR1Z6SlBZYQox

anNIRVhvOUo0MnBaQ3lWNnQxTklzRkFrL3pyazJzSFZHbFZzUzB4ZHpPaDhZaEhsKzVwVEE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

    server: https://192.168.232.236:8443

  name: kubernetes

contexts:

- context:

    cluster: kubernetes

    user: tls-bootstrap-token-user

  name: tls-bootstrap-token-user@kubernetes

current-context: tls-bootstrap-token-user@kubernetes

kind: Config

preferences: {}

users:

- name: tls-bootstrap-token-user

  user:

    token: c8ad9c.2e4d610cf3e7426e

根据后缀c8ad9c找到token

[root@k8s-master01 kubernetes]# kubectl get secret -n kube-system

NAME                                             TYPE                                  DATA   AGE

bootstrap-token-c8ad9c                           bootstrap.kubernetes.io/token         6      3d18h

读取secret中的内容，得到token-id，token-secret

[root@k8s-master01 kubernetes]# kubectl get secret -n kube-system bootstrap-token-c8ad9c -n kube-system -oyaml

apiVersion: v1

data:

  auth-extra-groups: c3lzdGVtOmJvb3RzdHJhcHBlcnM6ZGVmYXVsdC1ub2RlLXRva2VuLHN5c3RlbTpib290c3RyYXBwZXJzOndvcmtlcixzeXN0ZW06Ym9vdHN0cmFwcGVyczppbmdyZXNz

  description: VGhlIGRlZmF1bHQgYm9vdHN0cmFwIHRva2VuIGdlbmVyYXRlZCBieSAna3ViZWxldCAnLg==

  token-id: YzhhZDlj

  token-secret: MmU0ZDYxMGNmM2U3NDI2ZQ==

  usage-bootstrap-authentication: dHJ1ZQ==

  usage-bootstrap-signing: dHJ1ZQ==

kind: Secret

metadata:

  creationTimestamp: "2021-07-09T09:54:36Z"

  name: bootstrap-token-c8ad9c

  namespace: kube-system

  resourceVersion: "1408"

  uid: 9b77f873-d449-4ab2-aed7-fce9c32bdb21

type: bootstrap.kubernetes.io/token

解密token-id，与配置文件kubectl config中的token=c8ad9c.2e4d610cf3e7426e的前缀一致

[root@k8s-master01 kubernetes]# echo "YzhhZDlj" | base64 -d

c8ad9c

解密token-secret，与配置文件kubectl config中的token=c8ad9c.2e4d610cf3e7426e的后缀一致

[root@k8s-master01 kubernetes]# echo "MmU0ZDYxMGNmM2U3NDI2ZQ==" | base64 -d

2e4d610cf3e7426e

验证通过之后才会进入下一步

b) 找个这个secret中的一个字段auth-extra-groups，apiserver把这个token识别成一个username，名称是system:bootstrap:，属于system:bootstrappers这个组，这个组具有申请csr的权限，该组的权限绑定在一个叫system:node-bootstrapper的clusterrole

解密auth-extra-groups查询组名

[root@k8s-master01 kubernetes]# echo "c3lzdGVtOmJvb3RzdHJhcHBlcnM6ZGVmYXVsdC1ub2RlLXRva2VuLHN5c3RlbTpib290c3RyYXBwZXJzOndvcmtlcixzeXN0ZW06Ym9vdHN0cmFwcGVyczppbmdyZXNz" | base64 -d

system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress
  • clusterrole是k8s集群级别的权限控制，它作用整个k8s集群
  • clusterrolebinding是集群权限的绑定，它可以帮某个clusterrole绑定到一个用户、组或者seviceaccount

查看clusterrole

[root@k8s-master01 kubernetes]# kubectl get clusterrole

NAME                                                                   CREATED AT

system:node-bootstrapper                                               2021-07-09T09:34:55Z

查看system:node-bootstrapper

[root@k8s-master01 kubernetes]# kubectl get clusterrole system:node-bootstrapper -oyaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

  annotations:

    rbac.authorization.kubernetes.io/autoupdate: "true"

  creationTimestamp: "2021-07-09T09:34:55Z"

  labels:

    kubernetes.io/bootstrapping: rbac-defaults

  name: system:node-bootstrapper

  resourceVersion: "96"

  uid: b44bed52-fce4-4cf3-b3b1-38f887749d70

rules:

- apiGroups:

  - certificates.k8s.io

  resources:

  - certificatesigningrequests # 定义了一个csr权限

  verbs:

  - create

  - get

  - list

  - watch

查看clusterrolebinding

[root@k8s-master01 kubernetes]# kubectl get clusterrolebinding kubelet-bootstrap -oyaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

  creationTimestamp: "2021-07-09T09:54:36Z"

  name: kubelet-bootstrap

  resourceVersion: "1409"

  uid: 40de9025-a8f6-41c2-be85-763daca80bb2

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: system:node-bootstrapper

subjects:

- apiGroup: rbac.authorization.k8s.io

  kind: Group

  name: system:bootstrappers:default-node-token

可以看到clusterrolebinding把一个名为system:node-bootstrapper的ClusterRole绑定到一个名为system:bootstrappers:default-node-token的Group

查看Group是否一致，解密auth-extra-groups查询组名

[root@k8s-master01 kubernetes]# echo "c3lzdGVtOmJvb3RzdHJhcHBlcnM6ZGVmYXVsdC1ub2RlLXRva2VuLHN5c3RlbTpib290c3RyYXBwZXJzOndvcmtlcixzeXN0ZW06Ym9vdHN0cmFwcGVyczppbmdyZXNz" | base64 -d

system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress

c) CSR：相当于一个申请表，可以拿着这个申请表去申请我们的证书。

6、 经过上面的认证，kubelet就有了一个创建和检索CSR的权限

7、 Kubelet为自己创建一个CSR，名称为kubernetes.io/kube-apiserver-client-kubelet

8、 CSR被允许有两种方式：

a) K8s管理员使用kubectl手动的颁发证书

b) 如果配置了相关权限，kube-controller-manager会自动同意。

i. Controller-manager有一个CSRApprovingController。他会校验kubelet发来的csr的username和group是否有创建csr的权限，而且还要验证签发着是否是kubernetes.io/kube-apiserver-client-kubelet

ii. Controller-manager同意CSR请求

Bootstrapping: 证书自动续期原理

9、 CSR被同意后，controller-manager创建kubelet的证书文件

10、 Controller-manager将证书更新至csr的status字段

11、 Kubelet从apiserver获取证书

12、 Kubelet从获取到的key和证书文件创建kubelet.kubeconfig

13、 Kubelet启动完成并正常工作

14、 可选：如果配置了自动续期，kubelet会在证书文件过期的时候利用之前的kubeconfig文件去申请一个新的证书，相当于续约。

15、 新的证书被同意或签发，取决于我们的配置

查看node-autoapprove-certificate-rotation

[root@k8s-master01 kubernetes]# kubectl get clusterrolebinding node-autoapprove-certificate-rotation -oyaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

  creationTimestamp: "2021-07-09T09:54:36Z"

  name: node-autoapprove-certificate-rotation

  resourceVersion: "1411"

  uid: d7c618c8-0860-4d03-949f-4e2bbd33659a

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient

subjects:

- apiGroup: rbac.authorization.k8s.io

  kind: Group

  name: system:nodes

查看名为system:certificates.k8s.io:certificatesigningrequests:selfnodeclient的ClusterRole

[root@k8s-master01 kubernetes]# kubectl get clusterrole system:certificates.k8s.io:certificatesigningrequests:selfnodeclient -oyaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

  annotations:

    rbac.authorization.kubernetes.io/autoupdate: "true"

  creationTimestamp: "2021-07-09T09:34:55Z"

  labels:

    kubernetes.io/bootstrapping: rbac-defaults

  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient

  resourceVersion: "103"

  uid: d5215b54-8dd4-47ba-9e6e-bf664eb56dce

rules:

- apiGroups:

  - certificates.k8s.io

  resources:

  - certificatesigningrequests/selfnodeclient # 自动续期权限

  verbs:

  - create

a) Kubelet创建的CSR是属于一个组织：system:nodes

b) CN（比如域名）：system:nodes:主机名

课程链接（私信领取福利）

http://www.kubeasy.com/

Kubernetes全栈架构师（二进制高可用安装k8s集群扩展篇）--学习笔记

本作品采用知识共享署名-非商业性使用-相同方式共享 4.0 国际许可协议进行许可。

欢迎转载、使用、重新发布，但务必保留文章署名 郑子铭 （包含链接： http://www.cnblogs.com/MingsonZheng/ ），不得用于商业目的，基于本文修改后的作品务必以相同的许可发布。

上一篇：HTML中Meta标签中http-equiv属性小结


下一篇：Kubernetes全栈架构师（资源调度下）--学习笔记