Kubernetes K8S之Pod生命周期与探针检测

 

K8S中Pod的生命周期与ExecAction、TCPSocketAction和HTTPGetAction探针检测

 

主机配置规划

服务器名称(hostname) 系统版本 配置 内网IP 外网IP(模拟)
k8s-master CentOS7.7 2C/4G/20G 172.16.1.110 10.0.0.110
k8s-node01 CentOS7.7 2C/4G/20G 172.16.1.111 10.0.0.111
k8s-node02 CentOS7.7 2C/4G/20G 172.16.1.112 10.0.0.112

 

Pod容器生命周期

Kubernetes K8S之Pod生命周期与探针检测

 

Pause容器说明

每个Pod里运行着一个特殊的被称之为Pause的容器,其他容器则为业务容器,这些业务容器共享Pause容器的网络栈和Volume挂载卷,因此他们之间通信和数据交换更为高效。在设计时可以充分利用这一特性,将一组密切相关的服务进程放入同一个Pod中;同一个Pod里的容器之间仅需通过localhost就能互相通信。

kubernetes中的pause容器主要为每个业务容器提供以下功能:

PID命名空间:Pod中的不同应用程序可以看到其他应用程序的进程ID。

网络命名空间:Pod中的多个容器能够访问同一个IP和端口范围。

IPC命名空间:Pod中的多个容器能够使用SystemV IPC或POSIX消息队列进行通信。

UTS命名空间:Pod中的多个容器共享一个主机名;Volumes(共享存储卷)。

Pod中的各个容器可以访问在Pod级别定义的Volumes。

 

容器探针

探针是由 kubelet 对容器执行的定期诊断。要执行诊断,则需kubelet 调用由容器实现的 Handler。探针有三种类型的处理程序:

  • ExecAction:在容器内执行指定命令。如果命令退出时返回码为 0 则认为诊断成功。
  • CPSocketAction:对指定端口上的容器的 IP 地址进行 TCP 检查。如果端口打开,则诊断被认为是成功的。
  • HTTPGetAction:对指定的端口和路径上的容器的 IP 地址执行 HTTP Get 请求。如果响应的状态码大于等于200 且小于 400,则诊断被认为是成功的。

每次探测都将获得以下三种结果之一:

  • 成功:容器通过了诊断。
  • 失败:容器未通过诊断。
  • 未知:诊断失败,因此不会采取任何行动。

Kubelet 可以选择是否在容器上运行三种探针执行和做出反应:

  • livenessProbe:指示容器是否正在运行。如果存活探测失败,则 kubelet 会杀死容器,并且容器将受到其重启策略的影响。如果容器不提供存活探针,则默认状态为 Success。
  • readinessProbe:指示容器是否准备好服务请求【对外接受请求访问】。如果就绪探测失败,端点控制器将从与 Pod 匹配的所有 Service 的端点中删除该 Pod 的 IP 地址。初始延迟之前的就绪状态默认为 Failure。如果容器不提供就绪探针,则默认状态为 Success。
  • startupProbe: 指示容器中的应用是否已经启动。如果提供了启动探测(startup probe),则禁用所有其他探测,直到它成功为止。如果启动探测失败,kubelet 将杀死容器,容器服从其重启策略进行重启。如果容器没有提供启动探测,则默认状态为成功Success。

备注:可以以Tomcat web服务为例。

 

容器重启策略

PodSpec 中有一个 restartPolicy 字段,可能的值为 Always、OnFailure 和 Never。默认为 Always。

Always表示一旦不管以何种方式终止运行,kubelet都将重启;OnFailure表示只有Pod以非0退出码退出才重启;Nerver表示不再重启该Pod。

restartPolicy 适用于 Pod 中的所有容器。restartPolicy 仅指通过同一节点上的 kubelet 重新启动容器。失败的容器由 kubelet 以五分钟为上限的指数退避延迟(10秒,20秒,40秒…)重新启动,并在成功执行十分钟后重置。如 Pod 文档中所述,一旦pod绑定到一个节点,Pod 将永远不会重新绑定到另一个节点。

 

存活(liveness)和就绪(readiness)探针的使用场景

如果容器中的进程能够在遇到问题或不健康的情况下自行崩溃,则不一定需要存活探针;kubelet 将根据 Pod 的restartPolicy 自动执行正确的操作。

如果你希望容器在探测失败时被杀死并重新启动,那么请指定一个存活探针,并指定restartPolicy 为 Always 或 OnFailure。

如果要仅在探测成功时才开始向 Pod 发送流量,请指定就绪探针。在这种情况下,就绪探针可能与存活探针相同,但是 spec 中的就绪探针的存在意味着 Pod 将在没有接收到任何流量的情况下启动,并且只有在探针探测成功后才开始接收流量。

 

Pod phase(阶段)

Pod 的 status 定义在 PodStatus 对象中,其中有一个 phase 字段。

Pod 的运行阶段(phase)是 Pod 在其生命周期中的简单宏观概述。该阶段并不是对容器或 Pod 的综合汇总,也不是为了做为综合状态机。

Pod 相位的数量和含义是严格指定的。除了本文档中列举的内容外,不应该再假定 Pod 有其他的 phase 值。

 

Kubernetes K8S之Pod生命周期与探针检测

下面是 phase 可能的值:

  • 挂起(Pending):Pod 已被 Kubernetes 系统接受,但有一个或者多个容器镜像尚未创建。等待时间包括调度 Pod 的时间和通过网络下载镜像的时间,这可能需要花点时间。
  • 运行中(Running):该 Pod 已经绑定到了一个节点上,Pod 中所有的容器都已被创建。至少有一个容器正在运行,或者正处于启动或重启状态。
  • 成功(Succeeded):Pod 中的所有容器都被成功终止,并且不会再重启。
  • 失败(Failed):Pod 中的所有容器都已终止了,并且至少有一个容器是因为失败终止。也就是说,容器以非0状态退出或者被系统终止。
  • 未知(Unknown):因为某些原因无法取得 Pod 的状态,通常是因为与 Pod 所在主机通信失败。

 

检测探针-就绪检测

pod yaml脚本

 1 [root@k8s-master lifecycle]# pwd
 2 /root/k8s_practice/lifecycle
 3 [root@k8s-master lifecycle]# cat readinessProbe-httpget.yaml 
 4 apiVersion: v1
 5 kind: Pod
 6 metadata:
 7   name: readiness-httpdget-pod
 8   namespace: default
 9   labels:
10     test: readiness-httpdget
11 spec:
12   containers:
13   - name: readiness-httpget
14     image: registry.cn-beijing.aliyuncs.com/google_registry/nginx:1.17
15     imagePullPolicy: IfNotPresent
16     readinessProbe:
17       httpGet:
18         path: /index1.html
19         port: 80
20       initialDelaySeconds: 5  #容器启动完成后,kubelet在执行第一次探测前应该等待 5 秒。默认是 0 秒,最小值是 0。
21       periodSeconds: 3  #指定 kubelet 每隔 3 秒执行一次存活探测。默认是 10 秒。最小值是 1

 

创建 Pod,并查看pod状态

1 [root@k8s-master lifecycle]# kubectl apply -f readinessProbe-httpget.yaml 
2 pod/readiness-httpdget-pod created
3 [root@k8s-master lifecycle]# kubectl get pod -n default -o wide
4 NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
5 readiness-httpdget-pod   0/1     Running   0          5s    10.244.2.25   k8s-node02   <none>           <none>

 

查看pod详情

 1 [root@k8s-master lifecycle]# kubectl describe pod readiness-httpdget-pod
 2 Name:         readiness-httpdget-pod
 3 Namespace:    default
 4 Priority:     0
 5 Node:         k8s-node02/172.16.1.112
 6 Start Time:   Sat, 23 May 2020 16:10:04 +0800
 7 Labels:       test=readiness-httpdget
 8 Annotations:  kubectl.kubernetes.io/last-applied-configuration:
 9                 {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"test":"readiness-httpdget"},"name":"readiness-httpdget-pod","names...
10 Status:       Running
11 IP:           10.244.2.25
12 IPs:
13   IP:  10.244.2.25
14 Containers:
15   readiness-httpget:
16     Container ID:   docker://066d66aaef191b1db08e1b3efba6a9be75378d2fe70e99400fc513b91242089c
17 ………………
18     Port:           <none>
19     Host Port:      <none>
20     State:          Running
21       Started:      Sat, 23 May 2020 16:10:05 +0800
22     Ready:          False   ##### 状态为False
23     Restart Count:  0
24     Readiness:      http-get http://:80/index1.html delay=5s timeout=1s period=3s #success=1 #failure=3
25     Environment:    <none>
26     Mounts:
27       /var/run/secrets/kubernetes.io/serviceaccount from default-token-v48g4 (ro)
28 Conditions:
29   Type              Status
30   Initialized       True 
31   Ready             False   ##### 为False
32   ContainersReady   False   ##### 为False
33   PodScheduled      True 
34 Volumes:
35   default-token-v48g4:
36     Type:        Secret (a volume populated by a Secret)
37     SecretName:  default-token-v48g4
38     Optional:    false
39 QoS Class:       BestEffort
40 Node-Selectors:  <none>
41 Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
42                  node.kubernetes.io/unreachable:NoExecute for 300s
43 Events:
44   Type     Reason     Age                From                 Message
45   ----     ------     ----               ----                 -------
46   Normal   Scheduled  <unknown>          default-scheduler    Successfully assigned default/readiness-httpdget-pod to k8s-node02
47   Normal   Pulled     49s                kubelet, k8s-node02  Container image "registry.cn-beijing.aliyuncs.com/google_registry/nginx:1.17" already present on machine
48   Normal   Created    49s                kubelet, k8s-node02  Created container readiness-httpget
49   Normal   Started    49s                kubelet, k8s-node02  Started container readiness-httpget
50   Warning  Unhealthy  2s (x15 over 44s)  kubelet, k8s-node02  Readiness probe failed: HTTP probe failed with statuscode: 404

由上可见,容器未就绪。

我们进入pod的第一个容器,然后创建对应的文件

1 [root@k8s-master lifecycle]# kubectl exec -it readiness-httpdget-pod -c readiness-httpget bash
2 root@readiness-httpdget-pod:/# cd /usr/share/nginx/html
3 root@readiness-httpdget-pod:/usr/share/nginx/html# ls
4 50x.html  index.html
5 root@readiness-httpdget-pod:/usr/share/nginx/html# echo "readiness-httpdget info" > index1.html
6 root@readiness-httpdget-pod:/usr/share/nginx/html# ls
7 50x.html  index.html  index1.html

 

之后看pod状态与详情

 1 [root@k8s-master lifecycle]# kubectl get pod -n default -o wide
 2 NAME                     READY   STATUS    RESTARTS   AGE     IP            NODE         NOMINATED NODE   READINESS GATES
 3 readiness-httpdget-pod   1/1     Running   0          2m30s   10.244.2.25   k8s-node02   <none>           <none>
 4 [root@k8s-master lifecycle]# kubectl describe pod readiness-httpdget-pod
 5 Name:         readiness-httpdget-pod
 6 Namespace:    default
 7 Priority:     0
 8 Node:         k8s-node02/172.16.1.112
 9 Start Time:   Sat, 23 May 2020 16:10:04 +0800
10 Labels:       test=readiness-httpdget
11 Annotations:  kubectl.kubernetes.io/last-applied-configuration:
12                 {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"test":"readiness-httpdget"},"name":"readiness-httpdget-pod","names...
13 Status:       Running
14 IP:           10.244.2.25
15 IPs:
16   IP:  10.244.2.25
17 Containers:
18   readiness-httpget:
19     Container ID:   docker://066d66aaef191b1db08e1b3efba6a9be75378d2fe70e99400fc513b91242089c
20 ………………
21     Port:           <none>
22     Host Port:      <none>
23     State:          Running
24       Started:      Sat, 23 May 2020 16:10:05 +0800
25     Ready:          True     ##### 状态为True
26     Restart Count:  0
27     Readiness:      http-get http://:80/index1.html delay=5s timeout=1s period=3s #success=1 #failure=3
28     Environment:    <none>
29     Mounts:
30       /var/run/secrets/kubernetes.io/serviceaccount from default-token-v48g4 (ro)
31 Conditions:
32   Type              Status
33   Initialized       True 
34   Ready             True   ##### 为True
35   ContainersReady   True   ##### 为True
36   PodScheduled      True 
37 Volumes:
38   default-token-v48g4:
39     Type:        Secret (a volume populated by a Secret)
40     SecretName:  default-token-v48g4
41     Optional:    false
42 QoS Class:       BestEffort
43 Node-Selectors:  <none>
44 Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
45                  node.kubernetes.io/unreachable:NoExecute for 300s
46 Events:
47   Type     Reason     Age                   From                 Message
48   ----     ------     ----                  ----                 -------
49   Normal   Scheduled  <unknown>             default-scheduler    Successfully assigned default/readiness-httpdget-pod to k8s-node02
50   Normal   Pulled     2m33s                 kubelet, k8s-node02  Container image "registry.cn-beijing.aliyuncs.com/google_registry/nginx:1.17" already present on machine
51   Normal   Created    2m33s                 kubelet, k8s-node02  Created container readiness-httpget
52   Normal   Started    2m33s                 kubelet, k8s-node02  Started container readiness-httpget
53   Warning  Unhealthy  85s (x22 over 2m28s)  kubelet, k8s-node02  Readiness probe failed: HTTP probe failed with statuscode: 404

由上可见,容器已就绪。

 

检测探针-存活检测

存活检测-执行命令

pod yaml脚本

 1 [root@k8s-master lifecycle]# pwd
 2 /root/k8s_practice/lifecycle
 3 [root@k8s-master lifecycle]# cat livenessProbe-exec.yaml 
 4 apiVersion: v1
 5 kind: Pod
 6 metadata:
 7   name: liveness-exec-pod
 8   labels:
 9     test: liveness
10 spec:
11   containers:
12   - name: liveness-exec
13     image: registry.cn-beijing.aliyuncs.com/google_registry/busybox:1.24
14     imagePullPolicy: IfNotPresent
15     args:
16     - /bin/sh
17     - -c
18     - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
19     livenessProbe:
20       exec:
21         command:
22         - cat
23         - /tmp/healthy
24       initialDelaySeconds: 5   # 第一次检测前等待5秒
25       periodSeconds: 3   # 检测周期3秒一次

这个容器生命的前 30 秒,/tmp/healthy 文件是存在的。所以在这最开始的 30 秒内,执行命令 cat /tmp/healthy 会返回成功码。30 秒之后,执行命令 cat /tmp/healthy 就会返回失败状态码。

 

创建 Pod

1 [root@k8s-master lifecycle]# kubectl apply -f livenessProbe-exec.yaml 
2 pod/liveness-exec-pod created

 

在 30 秒内,查看 Pod 的描述:

 1 [root@k8s-master lifecycle]# kubectl get pod -o wide
 2 NAME                READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
 3 liveness-exec-pod   1/1     Running   0          17s   10.244.2.21   k8s-node02   <none>           <none>
 4 [root@k8s-master lifecycle]# kubectl describe pod liveness-exec-pod
 5 Name:         liveness-exec-pod
 6 Namespace:    default
 7 Priority:     0
 8 Node:         k8s-node02/172.16.1.112
 9 ………………
10 Events:
11   Type    Reason     Age   From                 Message
12   ----    ------     ----  ----                 -------
13   Normal  Scheduled  25s   default-scheduler    Successfully assigned default/liveness-exec-pod to k8s-node02
14   Normal  Pulled     24s   kubelet, k8s-node02  Container image "registry.cn-beijing.aliyuncs.com/google_registry/busybox:1.24" already present on machine
15   Normal  Created    24s   kubelet, k8s-node02  Created container liveness-exec
16   Normal  Started    24s   kubelet, k8s-node02  Started container liveness-exec

输出结果显示:存活探测器成功。

 

35 秒之后,再来看 Pod 的描述:

 1 [root@k8s-master lifecycle]# kubectl get pod -o wide   # 显示 RESTARTS 的值增加了 1
 2 NAME                READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
 3 liveness-exec-pod   1/1     Running   1          89s   10.244.2.22   k8s-node02   <none>           <none>
 4 [root@k8s-master lifecycle]# kubectl describe pod liveness-exec-pod
 5 ………………
 6 Events:
 7   Type     Reason     Age              From                 Message
 8   ----     ------     ----             ----                 -------
 9   Normal   Scheduled  42s              default-scheduler    Successfully assigned default/liveness-exec-pod to k8s-node02
10   Normal   Pulled     41s              kubelet, k8s-node02  Container image "registry.cn-beijing.aliyuncs.com/google_registry/busybox:1.24" already present on machine
11   Normal   Created    41s              kubelet, k8s-node02  Created container liveness-exec
12   Normal   Started    41s              kubelet, k8s-node02  Started container liveness-exec
13   Warning  Unhealthy  2s (x3 over 8s)  kubelet, k8s-node02  Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
14   Normal   Killing    2s               kubelet, k8s-node02  Container liveness-exec failed liveness probe, will be restarted

由上可见,在输出结果的最下面,有信息显示存活探测器失败了,因此这个容器被杀死并且被重建了。

 

存活检测-HTTP请求

pod yaml脚本

 1 [root@k8s-master lifecycle]# pwd
 2 /root/k8s_practice/lifecycle
 3 [root@k8s-master lifecycle]# cat livenessProbe-httpget.yaml 
 4 apiVersion: v1
 5 kind: Pod
 6 metadata:
 7   name: liveness-httpget-pod
 8   labels:
 9     test: liveness
10 spec:
11   containers:
12   - name: liveness-httpget
13     image: registry.cn-beijing.aliyuncs.com/google_registry/nginx:1.17
14     imagePullPolicy: IfNotPresent
15     ports:
16     - name: http
17       containerPort: 80
18     livenessProbe:
19       httpGet:   # 任何大于或等于 200 并且小于 400 的返回码表示成功,其它返回码都表示失败。
20         path: /index.html
21         port: 80
22         httpHeaders:  #请求中自定义的 HTTP 头。HTTP 头字段允许重复。
23         - name: Custom-Header
24           value: Awesome
25       initialDelaySeconds: 5
26       periodSeconds: 3

 

创建 Pod,查看pod状态

1 [root@k8s-master lifecycle]# kubectl apply -f livenessProbe-httpget.yaml 
2 pod/liveness-httpget-pod created
3 [root@k8s-master lifecycle]# kubectl get pod -n default -o wide
4 NAME                   READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
5 liveness-httpget-pod   1/1     Running   0          3s    10.244.2.27   k8s-node02   <none>           <none>

 

查看pod详情

 1 [root@k8s-master lifecycle]# kubectl describe pod liveness-httpget-pod
 2 Name:         liveness-httpget-pod
 3 Namespace:    default
 4 Priority:     0
 5 Node:         k8s-node02/172.16.1.112
 6 Start Time:   Sat, 23 May 2020 16:45:25 +0800
 7 Labels:       test=liveness
 8 Annotations:  kubectl.kubernetes.io/last-applied-configuration:
 9                 {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"test":"liveness"},"name":"liveness-httpget-pod","namespace":"defau...
10 Status:       Running
11 IP:           10.244.2.27
12 IPs:
13   IP:  10.244.2.27
14 Containers:
15   liveness-httpget:
16     Container ID:   docker://4b42a351414667000fe94d4f3166d75e72a3401e549fed723126d2297124ea1a
17 ………………
18     Port:           80/TCP
19     Host Port:      8080/TCP
20     State:          Running
21       Started:      Sat, 23 May 2020 16:45:26 +0800
22     Ready:          True
23     Restart Count:  0
24     Liveness:       http-get http://:80/index.html delay=5s timeout=1s period=3s #success=1 #failure=3
25     Environment:    <none>
26     Mounts:
27       /var/run/secrets/kubernetes.io/serviceaccount from default-token-v48g4 (ro)
28 Conditions:
29   Type              Status
30   Initialized       True 
31   Ready             True 
32   ContainersReady   True 
33   PodScheduled      True 
34 ………………
35 Events:
36   Type    Reason     Age        From                 Message
37   ----    ------     ----       ----                 -------
38   Normal  Scheduled  <unknown>  default-scheduler    Successfully assigned default/liveness-httpget-pod to k8s-node02
39   Normal  Pulled     5m52s      kubelet, k8s-node02  Container image "registry.cn-beijing.aliyuncs.com/google_registry/nginx:1.17" already present on machine
40   Normal  Created    5m52s      kubelet, k8s-node02  Created container liveness-httpget
41   Normal  Started    5m52s      kubelet, k8s-node02  Started container liveness-httpget

由上可见,pod存活检测正常

 

我们进入pod的第一个容器,然后删除对应的文件

1 [root@k8s-master lifecycle]# kubectl exec -it liveness-httpget-pod -c liveness-httpget bash
2 root@liveness-httpget-pod:/# cd /usr/share/nginx/html/
3 root@liveness-httpget-pod:/usr/share/nginx/html# ls
4 50x.html  index.html
5 root@liveness-httpget-pod:/usr/share/nginx/html# rm -f index.html 
6 root@liveness-httpget-pod:/usr/share/nginx/html# ls
7 50x.html

 

再次看pod状态和详情,可见Pod的RESTARTS从0变为了1。

 1 [root@k8s-master lifecycle]# kubectl get pod -n default -o wide   # RESTARTS 从0变为了1
 2 NAME                   READY   STATUS    RESTARTS   AGE     IP            NODE         NOMINATED NODE   READINESS GATES
 3 liveness-httpget-pod   1/1     Running   1          8m16s   10.244.2.27   k8s-node02   <none>           <none>
 4 [root@k8s-master lifecycle]# kubectl describe pod liveness-httpget-pod
 5 Name:         liveness-httpget-pod
 6 Namespace:    default
 7 Priority:     0
 8 Node:         k8s-node02/172.16.1.112
 9 Start Time:   Sat, 23 May 2020 16:45:25 +0800
10 Labels:       test=liveness
11 Annotations:  kubectl.kubernetes.io/last-applied-configuration:
12                 {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"test":"liveness"},"name":"liveness-httpget-pod","namespace":"defau...
13 Status:       Running
14 IP:           10.244.2.27
15 IPs:
16   IP:  10.244.2.27
17 Containers:
18   liveness-httpget:
19     Container ID:   docker://5d0962d383b1df5e59cd3d1100b259ff0415ac37c8293b17944034f530fb51c8
20 ………………
21     Port:           80/TCP
22     Host Port:      8080/TCP
23     State:          Running
24       Started:      Sat, 23 May 2020 16:53:38 +0800
25     Last State:     Terminated
26       Reason:       Completed
27       Exit Code:    0
28       Started:      Sat, 23 May 2020 16:45:26 +0800
29       Finished:     Sat, 23 May 2020 16:53:38 +0800
30     Ready:          True
31     Restart Count:  1
32     Liveness:       http-get http://:80/index.html delay=5s timeout=1s period=3s #success=1 #failure=3
33     Environment:    <none>
34     Mounts:
35       /var/run/secrets/kubernetes.io/serviceaccount from default-token-v48g4 (ro)
36 Conditions:
37   Type              Status
38   Initialized       True 
39   Ready             True 
40   ContainersReady   True 
41   PodScheduled      True 
42 Volumes:
43   default-token-v48g4:
44     Type:        Secret (a volume populated by a Secret)
45     SecretName:  default-token-v48g4
46     Optional:    false
47 QoS Class:       BestEffort
48 Node-Selectors:  <none>
49 Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
50                  node.kubernetes.io/unreachable:NoExecute for 300s
51 Events:
52   Type     Reason     Age                 From                 Message
53   ----     ------     ----                ----                 -------
54   Normal   Scheduled  <unknown>           default-scheduler    Successfully assigned default/liveness-httpget-pod to k8s-node02
55   Normal   Pulled     7s (x2 over 8m19s)  kubelet, k8s-node02  Container image "registry.cn-beijing.aliyuncs.com/google_registry/nginx:1.17" already present on machine
56   Normal   Created    7s (x2 over 8m19s)  kubelet, k8s-node02  Created container liveness-httpget
57   Normal   Started    7s (x2 over 8m19s)  kubelet, k8s-node02  Started container liveness-httpget
58   Warning  Unhealthy  7s (x3 over 13s)    kubelet, k8s-node02  Liveness probe failed: HTTP probe failed with statuscode: 404
59   Normal   Killing    7s                  kubelet, k8s-node02  Container liveness-httpget failed liveness probe, will be restarted

由上可见,当liveness-httpget检测失败,重建了Pod容器

 

存活检测-TCP端口

pod yaml脚本

 1 [root@k8s-master lifecycle]# pwd
 2 /root/k8s_practice/lifecycle
 3 [root@k8s-master lifecycle]# cat livenessProbe-tcp.yaml 
 4 apiVersion: v1
 5 kind: Pod
 6 metadata:
 7   name: liveness-tcp-pod
 8   labels:
 9     test: liveness
10 spec:
11   containers:
12   - name: liveness-tcp
13     image: registry.cn-beijing.aliyuncs.com/google_registry/nginx:1.17
14     imagePullPolicy: IfNotPresent
15     ports:
16     - name: http
17       containerPort: 80
18     livenessProbe:
19       tcpSocket:
20         port: 80
21       initialDelaySeconds: 5
22       periodSeconds: 3

 

TCP探测正常情况

创建 Pod,查看pod状态

1 [root@k8s-master lifecycle]# kubectl apply -f livenessProbe-tcp.yaml
2 pod/liveness-tcp-pod created
3 [root@k8s-master lifecycle]# kubectl get pod -o wide
4 NAME               READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
5 liveness-tcp-pod   1/1     Running   0          50s   10.244.4.23   k8s-node01   <none>           <none>

 

查看pod详情

 1 [root@k8s-master lifecycle]# kubectl describe pod liveness-tcp-pod
 2 Name:         liveness-tcp-pod
 3 Namespace:    default
 4 Priority:     0
 5 Node:         k8s-node01/172.16.1.111
 6 Start Time:   Sat, 23 May 2020 18:02:46 +0800
 7 Labels:       test=liveness
 8 Annotations:  kubectl.kubernetes.io/last-applied-configuration:
 9                 {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"test":"liveness"},"name":"liveness-tcp-pod","namespace":"default"}...
10 Status:       Running
11 IP:           10.244.4.23
12 IPs:
13   IP:  10.244.4.23
14 Containers:
15   liveness-tcp:
16     Container ID:   docker://4de13e7c2e36c028b2094bf9dcf8e2824bfd15b8c45a0b963e301b91ee1a926d
17 ………………
18     Port:           80/TCP
19     Host Port:      8080/TCP
20     State:          Running
21       Started:      Sat, 23 May 2020 18:03:04 +0800
22     Ready:          True
23     Restart Count:  0
24     Liveness:       tcp-socket :80 delay=5s timeout=1s period=3s #success=1 #failure=3
25     Environment:    <none>
26     Mounts:
27       /var/run/secrets/kubernetes.io/serviceaccount from default-token-v48g4 (ro)
28 Conditions:
29   Type              Status
30   Initialized       True 
31   Ready             True 
32   ContainersReady   True 
33   PodScheduled      True 
34 Volumes:
35   default-token-v48g4:
36     Type:        Secret (a volume populated by a Secret)
37     SecretName:  default-token-v48g4
38     Optional:    false
39 QoS Class:       BestEffort
40 Node-Selectors:  <none>
41 Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
42                  node.kubernetes.io/unreachable:NoExecute for 300s
43 Events:
44   Type    Reason     Age        From                 Message
45   ----    ------     ----       ----                 -------
46   Normal  Scheduled  <unknown>  default-scheduler    Successfully assigned default/liveness-tcp-pod to k8s-node01
47   Normal  Pulling    74s        kubelet, k8s-node01  Pulling image "registry.cn-beijing.aliyuncs.com/google_registry/nginx:1.17"
48   Normal  Pulled     58s        kubelet, k8s-node01  Successfully pulled image "registry.cn-beijing.aliyuncs.com/google_registry/nginx:1.17"
49   Normal  Created    57s        kubelet, k8s-node01  Created container liveness-tcp
50   Normal  Started    57s        kubelet, k8s-node01  Started container liveness-tcp

以上是正常情况,可见存活探测成功。

 

模拟TCP探测失败情况

将上面yaml文件中的探测TCP端口进行如下修改:

1 livenessProbe:
2   tcpSocket:
3     port: 8090  # 之前是80

 

删除之前的pod并重新创建,并过一会儿看pod状态

1 [root@k8s-master lifecycle]# kubectl apply -f livenessProbe-tcp.yaml 
2 pod/liveness-tcp-pod created
3 [root@k8s-master lifecycle]# kubectl get pod -o wide   # 可见RESTARTS变为了1,再过一会儿会变为2,之后依次叠加
4 NAME               READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
5 liveness-tcp-pod   1/1     Running   1          25s   10.244.2.28   k8s-node02   <none>           <none>

 

pod详情

 1 [root@k8s-master lifecycle]# kubectl describe pod liveness-tcp-pod
 2 Name:         liveness-tcp-pod
 3 Namespace:    default
 4 Priority:     0
 5 Node:         k8s-node02/172.16.1.112
 6 Start Time:   Sat, 23 May 2020 18:08:32 +0800
 7 Labels:       test=liveness
 8 ………………
 9 Events:
10   Type     Reason     Age                From                 Message
11   ----     ------     ----               ----                 -------
12   Normal   Scheduled  <unknown>          default-scheduler    Successfully assigned default/liveness-tcp-pod to k8s-node02
13   Normal   Pulled     12s (x2 over 29s)  kubelet, k8s-node02  Container image "registry.cn-beijing.aliyuncs.com/google_registry/nginx:1.17" already present on machine
14   Normal   Created    12s (x2 over 29s)  kubelet, k8s-node02  Created container liveness-tcp
15   Normal   Started    12s (x2 over 28s)  kubelet, k8s-node02  Started container liveness-tcp
16   Normal   Killing    12s                kubelet, k8s-node02  Container liveness-tcp failed liveness probe, will be restarted
17   Warning  Unhealthy  0s (x4 over 18s)   kubelet, k8s-node02  Liveness probe failed: dial tcp 10.244.2.28:8090: connect: connection refused

由上可见,liveness-tcp检测失败,重建了Pod容器。

 

检测探针-启动检测

有时候,会有一些现有的应用程序在启动时需要较多的初始化时间【如:Tomcat服务】。这种情况下,在不影响对触发这种探测的死锁的快速响应的情况下,设置存活探测参数是要有技巧的。

技巧就是使用一个命令来设置启动探测。针对HTTP 或者 TCP 检测,可以通过设置 failureThreshold * periodSeconds 参数来保证有足够长的时间应对糟糕情况下的启动时间。

示例如下:

pod yaml文件

 1 [root@k8s-master lifecycle]# pwd
 2 /root/k8s_practice/lifecycle
 3 [root@k8s-master lifecycle]# cat startupProbe-httpget.yaml
 4 apiVersion: v1
 5 kind: Pod
 6 metadata:
 7   name: startup-pod
 8   labels:
 9     test: startup
10 spec:
11   containers:
12   - name: startup
13     image: registry.cn-beijing.aliyuncs.com/google_registry/tomcat:7.0.94-jdk8-openjdk
14     imagePullPolicy: IfNotPresent
15     ports:
16     - name: web-port
17       containerPort: 8080
18       hostPort: 8080
19     livenessProbe:
20       httpGet:
21         path: /index.jsp
22         port: web-port
23       initialDelaySeconds: 5
24       periodSeconds: 10
25       failureThreshold: 1
26     startupProbe:
27       httpGet:
28         path: /index.jsp
29         port: web-port
30       periodSeconds: 10      #指定 kubelet 每隔 10 秒执行一次存活探测。默认是 10 秒。最小值是 1
31       failureThreshold: 30   #最大的失败次数

 

启动pod,并查看状态

1 [root@k8s-master lifecycle]# kubectl apply -f startupProbe-httpget.yaml 
2 pod/startup-pod created
3 [root@k8s-master lifecycle]# kubectl get pod -o wide
4 NAME          READY   STATUS    RESTARTS   AGE     IP            NODE         NOMINATED NODE   READINESS GATES
5 startup-pod   1/1     Running   0          8m46s   10.244.4.26   k8s-node01   <none>           <none>

 

查看pod详情

[root@k8s-master ~]# kubectl describe pod startup-pod

有启动探测,应用程序将会有最多 5 分钟(30 * 10 = 300s) 的时间来完成它的启动。一旦启动探测成功一次,存活探测任务就会接管对容器的探测,对容器死锁可以快速响应。 如果启动探测一直没有成功,容器会在 300 秒后被杀死,并且根据 restartPolicy 来设置 Pod 状态。

 

探测器配置详解

使用如下这些字段可以精确的控制存活和就绪检测行为:

  • initialDelaySeconds:容器启动后要等待多少秒后存活和就绪探测器才被初始化,默认是 0 秒,最小值是 0。
  • periodSeconds:执行探测的时间间隔(单位是秒)。默认是 10 秒。最小值是 1。
  • timeoutSeconds:探测的超时时间。默认值是 1 秒。最小值是 1。
  • successThreshold:探测器在失败后,被视为成功的最小连续成功数。默认值是 1。存活探测的这个值必须是 1。最小值是 1。
  • failureThreshold:当探测失败时,Kubernetes 的重试次数。存活探测情况下的放弃就意味着重新启动容器。就绪探测情况下的放弃 Pod 会被打上未就绪的标签。默认值是 3。最小值是 1。

HTTP 探测器可以在 httpGet 上配置额外的字段:

  • host:连接使用的主机名,默认是 Pod 的 IP。也可以在 HTTP 头中设置 “Host” 来代替。
  • scheme :用于设置连接主机的方式(HTTP 还是 HTTPS)。默认是 HTTP。
  • path:访问 HTTP 服务的路径。
  • httpHeaders:请求中自定义的 HTTP 头。HTTP 头字段允许重复。
  • port:访问容器的端口号或者端口名。如果数字必须在 1 ~ 65535 之间。

 

相关阅读

1、官网:配置存活、就绪和启动探测器

2、Kubernetes K8S之Pod 生命周期与init container初始化容器

完毕!

 


———END———
如果觉得不错就关注下呗 (-^O^-) !

Kubernetes K8S之Pod生命周期与探针检测

 

上一篇:CSAPP:Machine Level Programming part 1


下一篇:绑定和取消全局快捷键