Kubernetes 环境中使用 NAS 动态存储卷

2022-06-05 17:47:11

介绍

K8S在应用中经常会用到nas共享存储卷，通常的使用方法是通过一个pv、pvc进行挂载，但当需要pv、pvc数量很大的时候，手动创建显得非常繁琐，这时动态数据卷的功能可以满足您的需求。
以下场景：

当为不同的用户提供不同的nas子目录进行挂载的时候，可以考虑使用nas动态存储卷；
当使用SatefulSet部署应用，每个pod配置使用不同nas数据卷的时候；

本方案中实现的动态nas卷，是在某个nas文件系统下通过创建子目录、并把子目录映射为一个动态pv提供给应用；

部署详解

1. 部署Controller

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: alicloud-nas-controller
  namespace: kube-system
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: alicloud-nas-controller
    spec:
      tolerations:
      - operator: "Exists"
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 1
            preference:
              matchExpressions:
              - key: node-role.kubernetes.io/master
                operator: Exists
      priorityClassName: system-node-critical
      serviceAccount: admin
      hostNetwork: true
      containers:
        - name: nfs-provisioner
          image: registry.cn-hangzhou.aliyuncs.com/acs/alicloud-nas-controller:v1.14.3.8-58bf821-aliyun
          env:
          - name: PROVISIONER_NAME
            value: alicloud/nas
          securityContext:
            privileged: true
          volumeMounts:
          - mountPath: /var/log
            name: log
      volumes:
      - hostPath:
          path: /var/log
        name: log

2. StorageClass配置：

Nas Controller支持两种挂载驱动，flexvolume、nfs方式；

flexvolume：表示使用阿里云提供的flexvolume(alicloud/nas)作为挂载驱动，集群中需要部署flexvolume；
nfs：表示用时Kubernetes提供的标准挂载驱动；

Flexvolume方式：

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-nas
mountOptions:
- nolock,tcp,noresvport
- vers=3
parameters:
  server: "23a9649583-iaq37.cn-shenzhen.nas.aliyuncs.com:/nasroot1/,23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com:/nasroot2/"
  driver: flexvolume
provisioner: alicloud/nas
reclaimPolicy: Delete

mountOptions：

表示生成的pv option 配置，挂载nas卷时使用这个option挂载；

server：

表示生成目标pv所使用nas挂载点列表；格式为 nfsurl1:/path1,nfsurl2:/path2；
当配置多个server的时候，通过此storageclass创建的pv会轮询使用上述server作为配置参数；

driver：

支持flexvolume、nfs，默认为nfs；

reclaimPolicy：

pv的回收策略，建议配置为Retain；当配置为Delete时，删除pv后nas文件系统中的对应目录会默认修改名字（path-name ==> archived-path-name）;
如果需要删除文件系统中对应的存储目录，可在storageclass中配置archiveOnDelete为false；

NFS 方式：

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-nas
mountOptions:
- noresvport
- vers=4.0
parameters:
  server: "23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com:/nasroot2/"
  driver: nfs
provisioner: alicloud/nas
reclaimPolicy: Delete

应用示例

1. 创建Flexvolume驱动数据卷

创建存储类：# kubectl create -f -

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-nas
mountOptions:
- nolock,tcp,noresvport
- vers=3
parameters:
  server: "23a9649583-iaq37.cn-shenzhen.nas.aliyuncs.com:/nasroot1/,23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com:/nasroot2/"
  driver: flexvolume
provisioner: alicloud/nas
reclaimPolicy: Delete

创建statefulset应用，配置动态存储卷：

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 5
  volumeClaimTemplates:
  - metadata:
      name: html
    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: alicloud-nas
      resources:
        requests:
          storage: 2Gi
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:alpine
        volumeMounts:
        - mountPath: "/data"
          name: html

检查生成的pvc、pv；

# kubectl get pvc | grep web
html-web-0      Bound         default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        96s
html-web-1      Bound         default-html-web-1-pvc-6b108a5b-b9e0-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        8m14s
html-web-2      Bound         default-html-web-2-pvc-661ebd73-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        73s
html-web-3      Bound         default-html-web-3-pvc-69569748-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        68s
html-web-4      Bound         default-html-web-4-pvc-6c52a277-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        63s
下面看到使用了flexvolume驱动创建的pv：
# kubectl get pv default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf -oyaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: alicloud/nas
  creationTimestamp: "2019-08-08T13:35:20Z"
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    createdby.aliyun.com: alicloud-nas-controller
    version.controller.aliyun.com: v1.14.3-1173ac8
  name: default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf
  resourceVersion: "2241971"
  selfLink: /api/v1/persistentvolumes/default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf
  uid: 5da7280c-b9e1-11e9-91a9-00163e10b7cf
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: html-web-0
    namespace: default
    resourceVersion: "2241962"
    uid: 58450866-b9e1-11e9-91a9-00163e10b7cf
  flexVolume:
    driver: alicloud/nas
    options:
      options: nolock,tcp,noresvport,vers=3
      path: /nasroot2/default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf
      server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
      vers: "3"
  persistentVolumeReclaimPolicy: Delete
  storageClassName: alicloud-nas
  volumeMode: Filesystem
status:
  phase: Bound

检查5个pv是否轮询使用server中的不同挂载点：

# for item in `kubectl get pv | grep web | awk '{print $1}'`; do  kubectl get pv $item -oyaml | grep server; done
      server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
      server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
      server: 23a9649583-iaq37.cn-shenzhen.nas.aliyuncs.com
      server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
      server: 23a9649583-iaq37.cn-shenzhen.nas.aliyuncs.com
      
# for item in `kubectl get pv | grep web | awk '{print $1}'`; do  kubectl get pv $item -oyaml | grep path; done
      path: /nasroot2/default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf
      path: /nasroot2/default-html-web-1-pvc-6b108a5b-b9e0-11e9-91a9-00163e10b7cf
      path: /nasroot1/default-html-web-2-pvc-661ebd73-b9e1-11e9-91a9-00163e10b7cf
      path: /nasroot2/default-html-web-3-pvc-69569748-b9e1-11e9-91a9-00163e10b7cf
      path: /nasroot1/default-html-web-4-pvc-6c52a277-b9e1-11e9-91a9-00163e10b7cf

nas文件系统中生成不同的子目录：

# ls nasroot1
default-html-web-2-pvc-661ebd73-b9e1-11e9-91a9-00163e10b7cf  default-html-web-4-pvc-6c52a277-b9e1-11e9-91a9-00163e10b7cf
# ls nasroot2
default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf  default-html-web-3-pvc-69569748-b9e1-11e9-91a9-00163e10b7cf
default-html-web-1-pvc-6b108a5b-b9e0-11e9-91a9-00163e10b7cf

删除应用pv、pvc，nas文件系统中目录名字改为archived-__

# ls nasroot1
archived-default-html-web-2-pvc-661ebd73-b9e1-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37
archived-default-html-web-4-pvc-6c52a277-b9e1-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37
# ls nasroot2
archived-default-html-web-0-pvc-58450866-b9e1-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37
archived-default-html-web-1-pvc-6b108a5b-b9e0-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37
archived-default-html-web-3-pvc-69569748-b9e1-11e9-91a9-00163e10b7cf.2019-08-08-13:38:37

2. 创建NFS驱动数据卷

创建存储类：# kubectl create -f -
archiveOnDelete: 配置为false，表示pv回收策略为Delete时，删除pv会把文件系统中的目录一起删除；

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: alicloud-nas
mountOptions:
- noresvport
- vers=4.0
parameters:
  server: "23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com:/nasroot2/"
  driver: nfs
  archiveOnDelete: "false"
provisioner: alicloud/nas
reclaimPolicy: Delete

创建statefulset，检查pv、pvc：

# kubectl get pvc | grep web
html-web-0      Bound         default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        2m4s
html-web-1      Bound         default-html-web-1-pvc-015441f5-b9e2-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        2m
html-web-2      Bound         default-html-web-2-pvc-3edcb47b-b9e2-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        17s
html-web-3      Bound         default-html-web-3-pvc-420f9bd8-b9e2-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        11s
html-web-4      Bound         default-html-web-4-pvc-4843579f-b9e2-11e9-91a9-00163e10b7cf   2Gi        RWO            alicloud-nas        1s
下面信息显示，pv使用的驱动为NFS：
# kubectl get pv default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf -oyaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: alicloud/nas
  creationTimestamp: "2019-08-08T13:39:50Z"
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    createdby.aliyun.com: alicloud-nas-controller
    version.controller.aliyun.com: v1.14.3-1173ac8
  name: default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf
  resourceVersion: "2243033"
  selfLink: /api/v1/persistentvolumes/default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf
  uid: fef7e4d1-b9e1-11e9-91a9-00163e10b7cf
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: html-web-0
    namespace: default
    resourceVersion: "2243017"
    uid: fed53a89-b9e1-11e9-91a9-00163e10b7cf
  mountOptions:
  - noresvport
  - vers=4.0
  nfs:
    path: /nasroot2/default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
  persistentVolumeReclaimPolicy: Delete
  storageClassName: alicloud-nas
  volumeMode: Filesystem
status:
  phase: Bound

检查5个pv的挂载点：

# for item in `kubectl get pv | grep web | awk '{print $1}'`; do  kubectl get pv $item -oyaml | grep server; done
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
    server: 23a9649583-ajv79.cn-shenzhen.nas.aliyuncs.com
# for item in `kubectl get pv | grep web | awk '{print $1}'`; do  kubectl get pv $item -oyaml | grep path; done
    path: /nasroot2/default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf
    path: /nasroot2/default-html-web-1-pvc-015441f5-b9e2-11e9-91a9-00163e10b7cf
    path: /nasroot2/default-html-web-2-pvc-3edcb47b-b9e2-11e9-91a9-00163e10b7cf
    path: /nasroot2/default-html-web-3-pvc-420f9bd8-b9e2-11e9-91a9-00163e10b7cf
    path: /nasroot2/default-html-web-4-pvc-4843579f-b9e2-11e9-91a9-00163e10b7cf

nas文件系统中生成不同的子目录：

# ls nasroot2/
default-html-web-0-pvc-fed53a89-b9e1-11e9-91a9-00163e10b7cf  default-html-web-3-pvc-420f9bd8-b9e2-11e9-91a9-00163e10b7cf
default-html-web-1-pvc-015441f5-b9e2-11e9-91a9-00163e10b7cf  default-html-web-4-pvc-4843579f-b9e2-11e9-91a9-00163e10b7cf
default-html-web-2-pvc-3edcb47b-b9e2-11e9-91a9-00163e10b7cf

删除应用pv、pvc，nas文件系统中目录同时被删除；（StorageClass中配置了archiveOnDelete: "false"）

“ 阿里巴巴云原生关注微服务、Serverless、容器、Service Mesh 等技术领域、聚焦云原生流行技术趋势、云原生大规模的落地实践，做最懂云原生开发者的技术圈。”

码农公寓

介绍