在StatefulSet中使用LocalVolume存储卷保持节点一致

2021-11-25 04:04:21

StatefulSet是一种有状态服务，其存储卷的使用有多种方式：

使用共享存储，这时在模板中定义一个volume卷，可以给多个pod共享；

每个pod配置独立的存储卷，使用非共享存储（块存储）时需要这样配置，通过配置volumeClaimTemplates实现；

对于StatefulSet使用localvolume的场景，和上述两种情况都不一样：

通过volumeClaimTemplates自动生成localvolume的controller没有提供；
localvolume本身定义了节点信息，会导致使用这个pv的pod调度到相应节点；
StatefulSet如果多个pod使用同一个localvolume，会导致多个pod调度到相同节点；

本文先通过手动创建pvc、localvolume的方式，并利用statefulset 自动生成pvc的规则，在statefulset中配置localvolume；

场景：

生成一个StatefulSet应用，每个pod定义到特定节点；StatefulSet中定义2个Pod，每个pod使用两个pvc，共需要创建4个pvc和4个localvolume；

如下图所示：

模板：

下面分别给出4个pv、pvc的模板，4个pv分别配置到cn-shenzhen.i-wz9gvy73m4qyk03xzg1y、cn-shenzhen.i-wz9c9m0m4oldr6mt89rd两个节点上；

pv1, pvc1:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-volume-a-1
  labels:
    alicloud-pvname: local-volume-a-1
spec:
  capacity:
    storage: 20Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-volume
  local:
    path: /local1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - cn-shenzhen.i-wz9c9m0m4oldr6mt89rd
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-volume-a-web-0
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: local-volume
  selector:
    matchLabels:
      alicloud-pvname: local-volume-a-1

pv2, pvc2:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-volume-a-2
  labels:
    alicloud-pvname: local-volume-a-2
spec:
  capacity:
    storage: 20Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-volume
  local:
    path: /local1
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - cn-shenzhen.i-wz9gvy73m4qyk03xzg1y
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-volume-a-web-1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: local-volume
  selector:
    matchLabels:
      alicloud-pvname: local-volume-a-2

pv3, pvc3:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-volume-b-1
  labels:
    alicloud-pvname: local-volume-b-1
spec:
  capacity:
    storage: 20Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-volume
  local:
    path: /local2
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - cn-shenzhen.i-wz9c9m0m4oldr6mt89rd
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-volume-b-web-0
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: local-volume
  selector:
    matchLabels:
      alicloud-pvname: local-volume-b-1

pv4, pvc4:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-volume-b-2
  labels:
    alicloud-pvname: local-volume-b-2
spec:
  capacity:
    storage: 20Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-volume
  local:
    path: /local2
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - cn-shenzhen.i-wz9gvy73m4qyk03xzg1y
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: local-volume-b-web-1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: local-volume
  selector:
    matchLabels:
      alicloud-pvname: local-volume-b-2

创建上述pv pvc后，创建应用：

创建应用：

创建的时候注意pvc的名字的构成：
pvc的名字 = statefulset-name + volume-name + 序号

local-volume-a-web-0 = "local-volume-a" + "-" + "web" + "-" + "0"

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx
  serviceName: "nginx"
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: local-volume-a
          mountPath: /data1
        - name: local-volume-b
          mountPath: /data2
  volumeClaimTemplates:
  - metadata:
      name: local-volume-a
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-volume"
      resources:
        requests:
          storage: 20Gi
  - metadata:
      name: local-volume-b
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-volume"
      resources:
        requests:
          storage: 20Gi

验证：

# kubectl get pod | grep web
web-0                              1/1     Running     0          15m
web-1                              1/1     Running     0          14m
# kubectl get pvc | grep local
local-volume-a-web-0                       Bound    local-volume-a-1         20Gi       RWO            local-volume        15m
local-volume-a-web-1                       Bound    local-volume-a-2         20Gi       RWO            local-volume        15m
local-volume-b-web-0                       Bound    local-volume-b-1         20Gi       RWO            local-volume        15m
local-volume-b-web-1                       Bound    local-volume-b-2         20Gi       RWO            local-volume        15m
# kubectl get pv | grep local
local-volume-a-1         20Gi       RWO            Retain           Bound      default/local-volume-a-web-0                       local-volume                 15m
local-volume-a-2         20Gi       RWO            Retain           Bound      default/local-volume-a-web-1                       local-volume                 17m
local-volume-b-1         20Gi       RWO            Retain           Bound      default/local-volume-b-web-0                       local-volume                 15m
local-volume-b-2         20Gi       RWO            Retain           Bound      default/local-volume-b-web-1                       local-volume                 17m

删除pod，验证新建pod调度到相同节点：

# kubectl describe pod web-0 | grep Node
Node:               cn-shenzhen.i-wz9c9m0m4oldr6mt89rd/192.168.0.12
Node-Selectors:  <none>
# kubectl delete pod web-0
pod "web-0" deleted
# kubectl get pod
NAME                               READY   STATUS      RESTARTS   AGE
web-0                              1/1     Running     0          10s
web-1                              1/1     Running     0          15m
# kubectl describe pod web-0 | grep Node
Node:               cn-shenzhen.i-wz9c9m0m4oldr6mt89rd/192.168.0.12
Node-Selectors:  <none>

码农公寓

场景：

模板：

创建应用：

验证：

相关文章