5.调度策略
自定义调度策略
# 自定义一个demo-scheduler的资源策略配置文件
[14:47:54 root@master1 scheduler]#mkdir /etc/kubernetes/scheduler
[14:48:39 root@master1 scheduler]#cd /etc/kubernetes/scheduler
[14:49:20 root@master1 scheduler]#cat kubeschedulerconfiguration.yaml
apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
clientConnection:
kubeconfig: "/etc/kubernetes/scheduler.conf"
profiles:
- schedulerName: default-scheduler
- schedulerName: demo-scheduler
plugins:
filter:
disabled:
- name: NodeUnschedulable
score:
disabled:
- name: NodeResourcesBalancedAllocation
weight: 1
- name: NodeResourcesLeastAllocated
weight: 1
enabled:
- name: NodeResourcesMostAllocated
weight: 5
# 应用前面定义的配置文件,注意提前备份好原本的kube-scheduler.yaml
[15:02:09 root@master1 manifests]#diff kube-scheduler.yaml kube-scheduler.yaml-bk
16d15
< - --config=/etc/kubernetes/scheduler/kubeschedulerconfiguration.yaml
20c19
< #- --port=0
---
> - --port=0
51,53d49
< - mountPath: /etc/kubernetes/scheduler
< name: schedconf
< readOnly: true
64,67d59
< - hostPath:
< path: /etc/kubernetes/scheduler
< type: DirectoryOrCreate
< name: schedconf
# 查看结果
[15:03:56 root@master1 manifests]#kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-6fb865d84f-4lhbz 1/1 Running 15 (54m ago) 34d
calico-node-7hj44 1/1 Running 6 (5d20h ago) 34d
calico-node-hk2r2 1/1 Running 11 (5d20h ago) 34d
calico-node-kmmwm 1/1 Running 12 (5d20h ago) 34d
calico-node-ns2ff 1/1 Running 6 (5d20h ago) 34d
calico-node-qv7nn 1/1 Running 6 (5d20h ago) 34d
coredns-76b4d8bc8f-d69q9 1/1 Running 12 (5d20h ago) 34d
coredns-76b4d8bc8f-ndsg9 1/1 Running 12 (5d20h ago) 34d
etcd-master1 1/1 Running 13 (5d20h ago) 34d
etcd-master2.noisedu.cn 1/1 Running 14 (5d20h ago) 34d
etcd-master3.noisedu.cn 1/1 Running 14 (5d20h ago) 34d
kube-apiserver-master1 1/1 Running 14 (5d20h ago) 34d
kube-apiserver-master2.noisedu.cn 1/1 Running 15 (5d20h ago) 34d
kube-apiserver-master3.noisedu.cn 1/1 Running 15 (5d20h ago) 34d
kube-controller-manager-master1 1/1 Running 14 (5d20h ago) 34d
kube-controller-manager-master2.noisedu.cn 1/1 Running 6 (5d20h ago) 34d
kube-controller-manager-master3.noisedu.cn 1/1 Running 9 (5d20h ago) 34d
kube-proxy-6lw45 1/1 Running 6 (5d20h ago) 34d
kube-proxy-9bjch 1/1 Running 6 (5d20h ago) 34d
kube-proxy-b8g7m 1/1 Running 11 (5d20h ago) 34d
kube-proxy-bbrxh 1/1 Running 6 (5d20h ago) 34d
kube-proxy-pm6jk 1/1 Running 12 (5d20h ago) 34d
kube-scheduler-master1 1/1 Running 0 2m26s
kube-scheduler-master2.noisedu.cn 1/1 Running 7 (5d20h ago) 34d
kube-scheduler-master3.noisedu.cn 1/1 Running 7 (5d20h ago) 34d
# 测试,结果显示无法调度
10:33:53 root@master1 scheduler]#cat 01-scheduler-deployment-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-test
spec:
replicas: 5
selector:
matchLabels:
app: pod-test
template:
metadata:
labels:
app: pod-test
spec:
schedulerName: demo-scheduler
containers:
- name: nginxpod-test
image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
imagePullPolicy: IfNotPresent
[10:33:58 root@master1 scheduler]#kubectl apply -f 01-scheduler-deployment-test.yaml
deployment.apps/deployment-test created
[10:34:13 root@master1 scheduler]#kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
deployment-test-84444b586-68s7c 0/1 Pending 0 6s <none> <none> <none> <none>
deployment-test-84444b586-69xfs 0/1 Pending 0 6s <none> <none> <none> <none>
deployment-test-84444b586-j2vvt 0/1 Pending 0 6s <none> <none> <none> <none>
deployment-test-84444b586-k74zt 0/1 Pending 0 6s <none> <none> <none> <none>
deployment-test-84444b586-zb8vp 0/1 Pending 0 6s <none> <none> <none> <none>
[10:34:19 root@master1 scheduler]#kubectl describe pod deployment-test-84444b586-j2vvt
Name: deployment-test-84444b586-j2vvt
Namespace: default
Priority: 0
Node: <none>
Labels: app=pod-test
pod-template-hash=84444b586
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/deployment-test-84444b586
Containers:
nginxpod-test:
Image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
Port: <none>
Host Port: <none>
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bb2zj (ro)
Volumes:
kube-api-access-bb2zj:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
# 当不用这个自定义策略时，调度成功
[10:35:03 root@master1 scheduler]#cat 00-no-scheduler-deployment-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: deployment-test
spec:
replicas: 5
selector:
matchLabels:
app: pod-test
template:
metadata:
labels:
app: pod-test
spec:
containers:
- name: nginxpod-test
image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
imagePullPolicy: IfNotPresent
[10:35:08 root@master1 scheduler]#kubectl apply -f 00-no-scheduler-deployment-test.yaml
deployment.apps/deployment-test created
[10:35:21 root@master1 scheduler]#kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
deployment-test-7d8cb8c5d-2dkv9 1/1 Running 0 7s 10.244.4.3 node2.noisedu.cn <none> <none>
deployment-test-7d8cb8c5d-9hx8f 1/1 Running 0 7s 10.244.3.3 node1.noisedu.cn <none> <none>
deployment-test-7d8cb8c5d-f46p5 1/1 Running 0 7s 10.244.4.4 node2.noisedu.cn <none> <none>
deployment-test-7d8cb8c5d-sjctx 1/1 Running 0 7s 10.244.3.4 node1.noisedu.cn <none> <none>
deployment-test-7d8cb8c5d-vql7n 1/1 Running 0 7s 10.244.3.5 node1.noisedu.cn <none> <none>
节点调度 - 亲和nodeAffinity
# 调度到指定节点，注意节点的name一定要正确，下面先示范错误的节点名
[10:54:46 root@master1 scheduler]#cat 02-scheduler-pod-nodename.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-nodename
spec:
nodeName: node1
containers:
- name: demoapp
image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
imagePullPolicy: IfNotPresent
[13:49:01 root@master1 scheduler]#kubectl apply -f 02-scheduler-pod-nodename.yaml
pod/pod-nodename created
[13:49:28 root@master1 scheduler]#kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod-nodename 0/1 Pending 0 5s <none> node1 <none> <none>
[13:49:33 root@master1 scheduler]#kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod-nodename 0/1 Pending 0 9s <none> node1 <none> <none>
[13:49:37 root@master1 scheduler]#
[13:49:55 root@master1 scheduler]#
[13:49:55 root@master1 scheduler]#kubectl delete -f 02-scheduler-pod-nodename.yaml
pod "pod-nodename" deleted
# 改为正确的节点名
[13:50:26 root@master1 scheduler]#vim 02-scheduler-pod-nodename.yaml
[13:51:06 root@master1 scheduler]#cat 02-scheduler-pod-nodename.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-nodename
spec:
nodeName: node1.noisedu.cn
containers:
- name: demoapp
image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
imagePullPolicy: IfNotPresent
[13:51:10 root@master1 scheduler]#kubectl apply -f 02-scheduler-pod-nodename.yaml
pod/pod-nodename created
[13:51:15 root@master1 scheduler]#kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod-nodename 1/1 Running 0 6s 10.244.3.6 node1.noisedu.cn <none> <none>
#------------------------------------------------------------------------------
# 调度到标签为ssd的节点
[13:53:39 root@master1 scheduler]#cat 03-scheduler-pod-nodeselector.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-nodeselector
spec:
containers:
- name: demoapp
image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
imagePullPolicy: IfNotPresent
nodeSelector:
node: ssd
[13:54:05 root@master1 scheduler]#kubectl get node --show-labels
NAME STATUS ROLES AGE VERSION LABELS
master1 Ready control-plane,master 35d v1.22.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers=
master2.noisedu.cn Ready control-plane,master 35d v1.22.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master2.noisedu.cn,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers=
master3.noisedu.cn Ready control-plane,master 35d v1.22.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master3.noisedu.cn,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers=
node1.noisedu.cn Ready <none> 35d v1.22.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node1.noisedu.cn,kubernetes.io/os=linux
node2.noisedu.cn Ready <none> 35d v1.22.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=node2.noisedu.cn,kubernetes.io/os=linux
[13:54:19 root@master1 scheduler]#kubectl get node --show-labels | grep ssd
[13:54:24 root@master1 scheduler]#kubectl apply -f 03-scheduler-pod-nodeselector.yaml
pod/pod-nodeselector created
# 由于没有存在为ssd标签的节点，所以pod一直处于pending状态
[13:54:42 root@master1 scheduler]#kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod-nodeselector 0/1 Pending 0 5s <none> <none> <none> <none>
[13:54:47 root@master1 scheduler]#kubectl describe pod pod-nodeselector
Name: pod-nodeselector
Namespace: default
Priority: 0
Node: <none>
Labels: <none>
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Containers:
demoapp:
Image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
Port: <none>
Host Port: <none>
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mkmfp (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-mkmfp:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: node=ssd
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 11s default-scheduler 0/5 nodes are available: 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
# 一旦给某个节点加上这个标签，pod就会成功
[13:56:58 root@master1 scheduler]#kubectl label node node2.noisedu.cn node=ssd
node/node2.noisedu.cn labeled
[13:57:22 root@master1 scheduler]#kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod-nodeselector 1/1 Running 0 30s 10.244.4.5 node2.noisedu.cn <none> <none>
[13:57:28 root@master1 scheduler]#kubectl describe pod pod-nodeselector
Name: pod-nodeselector
Namespace: default
Priority: 0
Node: node2.noisedu.cn/10.0.0.54
Start Time: Sun, 16 Jan 2022 13:57:22 +0800
Labels: <none>
Annotations: cni.projectcalico.org/containerID: 74260ea9c42767d0f0b9fde2fdd74071c58e40fdd82be1af7afb69925ea438c5
cni.projectcalico.org/podIP: 10.244.4.5/32
cni.projectcalico.org/podIPs: 10.244.4.5/32
Status: Running
IP: 10.244.4.5
IPs:
IP: 10.244.4.5
Containers:
demoapp:
Container ID: docker://d663a8a36145e37c54a9987755f95311b250c1e0f9b8137b4836aae8ba89c0a4
Image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
Image ID: docker-pullable://10.0.0.55:80/mykubernetes/pod_test@sha256:54402cda2ef15f45e4aafe98a5e56d4de076e3d4100c2a1bf1b780c787372fed
Port: <none>
Host Port: <none>
State: Running
Started: Sun, 16 Jan 2022 13:57:24 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tktqv (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-tktqv:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: node=ssd
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 35s default-scheduler 0/5 nodes are available: 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
Normal Scheduled 11s default-scheduler Successfully assigned default/pod-nodeselector to node2.noisedu.cn
Normal Pulled 9s kubelet Container image "10.0.0.55:80/mykubernetes/pod_test:v0.1" already present on machine
Normal Created 9s kubelet Created container demoapp
Normal Started 9s kubelet Started container demoapp
节点表达式匹配matchExpressions
[13:59:01 root@master1 scheduler]#cat 04-scheduler-pod-node-required-affinity.yaml
apiVersion: v1
kind: Pod
metadata:
name: node-required-affinity
spec:
containers:
- name: demoapp
image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
imagePullPolicy: IfNotPresent
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: env
operator: In
values:
- dev
- test
[14:01:06 root@master1 scheduler]#kubectl apply -f 04-scheduler-pod-node-required-affinity.yaml
pod/node-required-affinity created
[14:01:33 root@master1 scheduler]#kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
node-required-affinity 0/1 Pending 0 6s <none> <none> <none> <none>
[14:01:39 root@master1 scheduler]#kubectl describe pod node-required-affinity
Name: node-required-affinity
Namespace: default
Priority: 0
Node: <none>
Labels: <none>
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Containers:
demoapp:
Image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
Port: <none>
Host Port: <none>
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-g52g2 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-g52g2:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 12s default-scheduler 0/5 nodes are available: 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
# 由于当前无任何的标签匹配到，所以pod一直处于pending
# 添加标签env=test到node1.noisedu.cn
[14:01:45 root@master1 scheduler]#kubectl label node node1.noisedu.cn env=test
node/node1.noisedu.cn labeled
[14:02:45 root@master1 scheduler]#kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
node-required-affinity 1/1 Running 0 75s 10.244.3.7 node1.noisedu.cn <none> <none>
[14:02:48 root@master1 scheduler]#kubectl describe pod node-required-affinity
Name: node-required-affinity
Namespace: default
Priority: 0
Node: node1.noisedu.cn/10.0.0.53
Start Time: Sun, 16 Jan 2022 14:02:45 +0800
Labels: <none>
Annotations: cni.projectcalico.org/containerID: 925088d019790f61bf46bf4e6949ff386bd145f53fcc328c669eaad2dcd130a1
cni.projectcalico.org/podIP: 10.244.3.7/32
cni.projectcalico.org/podIPs: 10.244.3.7/32
Status: Running
IP: 10.244.3.7
IPs:
IP: 10.244.3.7
Containers:
demoapp:
Container ID: docker://5df82eeef3c5d5d3ec1de8a36665f7f8ef5faac112c3f2fa2442fbf019d273bf
Image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
Image ID: docker-pullable://10.0.0.55:80/mykubernetes/pod_test@sha256:54402cda2ef15f45e4aafe98a5e56d4de076e3d4100c2a1bf1b780c787372fed
Port: <none>
Host Port: <none>
State: Running
Started: Sun, 16 Jan 2022 14:02:46 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-g52g2 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-g52g2:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 78s default-scheduler 0/5 nodes are available: 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
Normal Scheduled 6s default-scheduler Successfully assigned default/node-required-affinity to node1.noisedu.cn
Normal Pulled 5s kubelet Container image "10.0.0.55:80/mykubernetes/pod_test:v0.1" already present on machine
Normal Created 5s kubelet Created container demoapp
Normal Started 5s kubelet Started container demoapp
# 发现此时匹配成功，调度到node1.noisedu.cn
软亲和preferredDuringSchedulingIgnoredDuringExecution
[14:04:47 root@master1 scheduler]#cat 05-scheduler-pod-node-preferred-affinity.yaml
apiVersion: v1
kind: Pod
metadata:
name: node-preferred-affinity
spec:
containers:
- name: demoapp
image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
imagePullPolicy: IfNotPresent
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 50
preference:
matchExpressions:
- key: env
operator: In
values:
- test
- weight: 20
preference:
matchExpressions:
- key: env
operator: In
values:
- dev
[14:07:11 root@master1 scheduler]#kubectl apply -f 05-scheduler-pod-node-preferred-affinity.yaml
pod/node-preferred-affinity created
[14:07:26 root@master1 scheduler]#kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
node-preferred-affinity 1/1 Running 0 5s 10.244.3.8 node1.noisedu.cn <none> <none>
# 因为指定了preferredDuringSchedulingIgnoredDuringExecution这个属性，即使不满足任何条件也可以调度到任意节点
# 我们重新给node1.noisedu.cn和node2.noisedu.cn加上标签，然后重新创建这个pod查看效果
[14:09:19 root@master1 scheduler]#kubectl delete -f 05-scheduler-pod-node-preferred-affinity.yaml
pod "node-preferred-affinity" deleted
[14:12:46 root@master1 scheduler]#kubectl label node node1.noisedu.cn env=dev
node/node1.noisedu.cn labeled
[14:13:04 root@master1 scheduler]#kubectl label node node2.noisedu.cn env=test
node/node2.noisedu.cn labeled
[14:13:12 root@master1 scheduler]#kubectl apply -f 05-scheduler-pod-node-preferred-affinity.yaml
pod/node-preferred-affinity created
[14:13:22 root@master1 scheduler]#kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
node-preferred-affinity 1/1 Running 0 5s 10.244.4.6 node2.noisedu.cn <none> <none>
[14:13:27 root@master1 scheduler]#kubectl describe pod node-preferred-affinity
Name: node-preferred-affinity
Namespace: default
Priority: 0
Node: node2.noisedu.cn/10.0.0.54
Start Time: Sun, 16 Jan 2022 14:13:22 +0800
Labels: <none>
Annotations: cni.projectcalico.org/containerID: 113cd980d6dee836fcf307ae8214235636c7c35c4476b5a37e5a4c92693ea61b
cni.projectcalico.org/podIP: 10.244.4.6/32
cni.projectcalico.org/podIPs: 10.244.4.6/32
Status: Running
IP: 10.244.4.6
IPs:
IP: 10.244.4.6
Containers:
demoapp:
Container ID: docker://7a520abf227cdc993a2b6288ecc1b6a35da7179e1f4a9a4443e817a88baa96f9
Image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
Image ID: docker-pullable://10.0.0.55:80/mykubernetes/pod_test@sha256:54402cda2ef15f45e4aafe98a5e56d4de076e3d4100c2a1bf1b780c787372fed
Port: <none>
Host Port: <none>
State: Running
Started: Sun, 16 Jan 2022 14:13:24 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b7m2v (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-b7m2v:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11s default-scheduler Successfully assigned default/node-preferred-affinity to node2.noisedu.cn
Normal Pulled 10s kubelet Container image "10.0.0.55:80/mykubernetes/pod_test:v0.1" already present on machine
Normal Created 9s kubelet Created container demoapp
Normal Started 9s kubelet Started container demoapp
# 此时发现因为node2.noisedu.cn的权重比较大，所以调度到node2这个节点，权重比较也是软亲和和硬亲和的最重要区别。
# 资源调度实验，我们对CPU和内存做了资源限制，要求满足资源的node才给调度
[14:18:56 root@master1 scheduler]#cat 06-scheduler-pod-node-resourcefits-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: node-resourcefits-affinity
spec:
replicas: 2
selector:
matchLabels:
app: podtest
template:
metadata:
labels:
app: podtest
spec:
containers:
- name: podtest
image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
imagePullPolicy: IfNotPresent
resources:
requests:
cpu: 2
memory: 2Gi
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: env
operator: Exists
[14:19:14 root@master1 scheduler]#kubectl apply -f 06-scheduler-pod-node-resourcefits-affinity.yaml
deployment.apps/node-resourcefits-affinity created
[14:19:19 root@master1 scheduler]#kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
node-resourcefits-affinity-84fd5f6f9c-qmmlc 0/1 Pending 0 6s <none> <none> <none> <none>
node-resourcefits-affinity-84fd5f6f9c-wpqt6 0/1 Pending 0 6s <none> <none> <none> <none>
[14:19:25 root@master1 scheduler]#kubectl describe pod
poddisruptionbudgets.policy pods podsecuritypolicies.policy podtemplates
[14:19:25 root@master1 scheduler]#kubectl describe pod node-resourcefits-affinity-84fd5f6f9c-qmmlc
Name: node-resourcefits-affinity-84fd5f6f9c-qmmlc
Namespace: default
Priority: 0
Node: <none>
Labels: app=podtest
pod-template-hash=84fd5f6f9c
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/node-resourcefits-affinity-84fd5f6f9c
Containers:
podtest:
Image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
Port: <none>
Host Port: <none>
Requests:
cpu: 2
memory: 2Gi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ljtxk (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-ljtxk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 22s default-scheduler 0/5 nodes are available: 2 Insufficient cpu, 2 Insufficient memory, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
# 我们发现满足2 CPU和内存的node都没有，所以pod处于pending。
# 修改CPU为0.2个，内存和100Mi，重新创建pod
[14:23:17 root@master1 scheduler]#cat 06-scheduler-pod-node-resourcefits-affinity.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: node-resourcefits-affinity
spec:
replicas: 2
selector:
matchLabels:
app: podtest
template:
metadata:
labels:
app: podtest
spec:
containers:
- name: podtest
image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
imagePullPolicy: IfNotPresent
resources:
requests:
cpu: 0.2
memory: 100Mi
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: env
operator: Exists
[14:23:22 root@master1 scheduler]#kubectl apply -f 06-scheduler-pod-node-resourcefits-affinity.yaml
deployment.apps/node-resourcefits-affinity created
[14:23:26 root@master1 scheduler]#kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
node-resourcefits-affinity-778bdb685-76x5s 1/1 Running 0 6s 10.244.3.9 node1.noisedu.cn <none> <none>
node-resourcefits-affinity-778bdb685-h54vp 1/1 Running 0 6s 10.244.4.7 node2.noisedu.cn <none> <none>
[14:23:32 root@master1 scheduler]#kubectl describe pod node-resourcefits-affinity-778bdb685-76x5s
Name: node-resourcefits-affinity-778bdb685-76x5s
Namespace: default
Priority: 0
Node: node1.noisedu.cn/10.0.0.53
Start Time: Sun, 16 Jan 2022 14:23:26 +0800
Labels: app=podtest
pod-template-hash=778bdb685
Annotations: cni.projectcalico.org/containerID: f615b27cbf0fd22f4b08ba1675192b789c88646e38cf8db7c7df33add1409ae4
cni.projectcalico.org/podIP: 10.244.3.9/32
cni.projectcalico.org/podIPs: 10.244.3.9/32
Status: Running
IP: 10.244.3.9
IPs:
IP: 10.244.3.9
Controlled By: ReplicaSet/node-resourcefits-affinity-778bdb685
Containers:
podtest:
Container ID: docker://5ede69880785fe10fe4e12e6c0ef88a116fff69b40fe69aa0dc7f5ff4f8a8c44
Image: 10.0.0.55:80/mykubernetes/pod_test:v0.1
Image ID: docker-pullable://10.0.0.55:80/mykubernetes/pod_test@sha256:54402cda2ef15f45e4aafe98a5e56d4de076e3d4100c2a1bf1b780c787372fed
Port: <none>
Host Port: <none>
State: Running
Started: Sun, 16 Jan 2022 14:23:28 +0800
Ready: True
Restart Count: 0
Requests:
cpu: 200m
memory: 100Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n9rk2 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-n9rk2:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 16s default-scheduler Successfully assigned default/node-resourcefits-affinity-778bdb685-76x5s to node1.noisedu.cn
Normal Pulled 15s kubelet Container image "10.0.0.55:80/mykubernetes/pod_test:v0.1" already present on machine
Normal Created 15s kubelet Created container podtest
Normal Started 15s kubelet Started container podtest
# 发现成功，由于node1和node2都存在env这个label，所以随机调度。
[14:24:41 root@master1 scheduler]#kubectl get node --show-labels | grep env
node1.noisedu.cn Ready <none> 35d v1.22.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,env=dev,kubernetes.io/arch=amd64,kubernetes.io/hostname=node1.noisedu.cn,kubernetes.io/os=linux
node2.noisedu.cn Ready <none> 35d v1.22.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,env=test,kubernetes.io/arch=amd64,kubernetes.io/hostname=node2.noisedu.cn,kubernetes.io/os=linux