阿里云Kubernetes CSI实践 - 部署详解

本文以阿里云ACK环境为例介绍如何在Kubernetes系统中部署CSI插件;

CSI部署拓扑:

分为CSI Provisioner和CSI Plugin两部分:

Provisioner:目前只支持云盘,提供云盘数据卷动态创建的功能;

Plugin:云盘、NAS、OSS均需实现plugin;云盘实现了attach、mount的功能;NAS、OSS实现mount功能;

阿里云Kubernetes CSI实践 - 部署详解

OSS Plugin是通过fuse驱动实现的文件系统挂载,在为了解决升级问题,通过systemd方案 说明

环境准备:

需要有Kubernetes集群,可以通过阿里云ACK服务快速创建:使用文档

阿里云CSI插件推荐使用Node Attach方式参考,需要在kubelet中配置下面参数:

--enable-controller-attach-detach=false

下载部署模板:

https://github.com/AliyunContainerService/csi-plugin/blob/master/deploy/all/csi-plugin.yaml
https://github.com/AliyunContainerService/csi-plugin/blob/master/deploy/all/csi-provisioner.yaml

插件组成:

1. 依赖部分:

RBAC:
参考模板:https://github.com/AliyunContainerService/csi-plugin/blob/master/deploy/rbac.yaml
这里我们使用阿里云ACK中默认提供的serviceaccount:admin,可以满足我们的需求;

CSIDriver:
定义插件执行Node Attach方式,不需要部署attacher;分别为NAS,OSS,EBS创建CSIDriver;

2. Plugin 插件:

默认提供云盘、NAS、OSS三种存储的插件,每个插件由plugin容器 和 driver-registrar容器组成;

模板:https://github.com/AliyunContainerService/csi-plugin/blob/master/deploy/all/csi-plugin.yaml

通过配置- operator: "Exists",逃过所有node taint,保证在所有节点都部署;

通过配置priorityClassName: system-node-critical,保证插件在所有节点优先部署;

      tolerations:
        - operator: "Exists"
      priorityClassName: system-node-critical

网络、Pid、securityContext都配置为高权模式,以便插件具有挂载权限;

      hostNetwork: true
      hostPID: true

csi-diskplugin,配置以下环境变量:

    # 通过配置AK环境,为plugin提供挂载ak权限;也可用通过STS方式配置;
    #   文档:https://help.aliyun.com/document_detail/86485.html
    # 配置MAX_VOLUMES_PERNODE,声明单个节点最多过载的云盘数量;ecs限制最大值为16,这里取值应该小于等于16;
    
            - name: ACCESS_KEY_ID
              value: ""
            - name: ACCESS_KEY_SECRET
              value: ""
            - name: MAX_VOLUMES_PERNODE
              value: "15"

3. Provisioner插件:

默认提供以下5个StorageClass:

alicloud-disk-available:会通过efficiency、ssd、common的顺序依次尝试创建云盘;
alicloud-disk-essd:创建essd类型云盘;
alicloud-disk-common:创建普通云盘;
alicloud-disk-ssd:创建ssd类型云盘;
alicloud-disk-efficiency:创建高效云盘;

通过配置- operator: "Exists",逃过所有node taint,保证在所有节点都部署;

通过配置priorityClassName: system-node-critical,保证插件在所有节点优先部署;

通过配置nodeSelector,限制Provisioner只在master上执行,对于managed K8S场景可以去掉这个限制;

      tolerations:
      - operator: "Exists"
      nodeSelector:
         node-role.kubernetes.io/master: ""
      priorityClassName: system-node-critical

部署插件:

下载部署模板:

https://github.com/AliyunContainerService/csi-plugin/blob/master/deploy/all/csi-plugin.yaml
https://github.com/AliyunContainerService/csi-plugin/blob/master/deploy/all/csi-provisioner.yaml

执行部署命令:

# kubectl create -f csi-plugin.yaml
# kubectl create -f csi-provisioner.yaml

查看服务:

# kubectl get pod -nkube-system | grep csi
csi-plugin-2csjm                                    6/6     Running   0          75s
csi-plugin-52cs7                                    6/6     Running   0          75s
csi-plugin-g7kwr                                    6/6     Running   0          75s
csi-plugin-lqv2p                                    6/6     Running   0          75s
csi-plugin-r5d82                                    6/6     Running   0          75s
csi-plugin-vgkzp                                    6/6     Running   0          75s
csi-provisioner-0                                   2/2     Running   0          65s

# kubectl get sc
NAME                       PROVISIONER                       AGE
alicloud-disk-available    diskplugin.csi.alibabacloud.com   2m8s
alicloud-disk-common       diskplugin.csi.alibabacloud.com   2m8s
alicloud-disk-efficiency   diskplugin.csi.alibabacloud.com   2m8s
alicloud-disk-essd         diskplugin.csi.alibabacloud.com   2m8s
alicloud-disk-ssd          diskplugin.csi.alibabacloud.com   2m8s

# kubectl get CSIDriver
NAME                              CREATED AT
diskplugin.csi.alibabacloud.com   2019-07-11T08:36:09Z
nasplugin.csi.alibabacloud.com    2019-07-11T08:36:09Z
ossplugin.csi.alibabacloud.com    2019-07-11T08:36:09Z

插件测试:

部署下面模板,通过指定storageclass为alicloud-disk-ssd创建一个ssd类型云盘;

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: disk-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 25Gi
  storageClassName: alicloud-disk-ssd
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-disk
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        volumeMounts:
          - name: disk-pvc
            mountPath: "/data"
      volumes:
        - name: disk-pvc
          persistentVolumeClaim:
            claimName: disk-pvc

查看部署:

# kubectl get pvc
NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
disk-pvc   Bound    pvc-008eee21-a3db-11e9-8dec-00163e0a6ecc   25Gi       RWO            alicloud-disk-ssd   14s

# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM              STORAGECLASS        REASON   AGE
pvc-008eee21-a3db-11e9-8dec-00163e0a6ecc   25Gi       RWO            Retain           Bound    default/disk-pvc   alicloud-disk-ssd            23s

# kubectl get pod
NAME                          READY   STATUS    RESTARTS   AGE
nginx-disk-6d5659d745-pdjjl   1/1     Running   0          33s

# 云盘已经挂载成功
# kubectl exec nginx-disk-6d5659d745-pdjjl ls data
lost+found

# kubectl exec nginx-disk-6d5659d745-pdjjl mount | grep /data
/dev/vdb on /data type ext4 (rw,relatime,data=ordered)

欢迎加入阿里云CSI钉钉开发群

项目地址:https://github.com/AliyunContainerService/csi-plugin

阿里云Kubernetes CSI实践 - 部署详解

上一篇:config:invalid url domain 错误 微信jssdk


下一篇:K8S环境中NAS卷添加noresvport方法