K8S中手动扩容云盘数据卷

2021-06-27 22:12:41

K8S中手动扩容云盘数据卷

介绍：

云盘数据卷扩容包括以下几个部分：

云盘物理空间扩容，需要在云盘控制台操作；
文件系统扩容，需要挂载云盘到一个物理节点手动操作；
PV、PVC Size更新，需要更新StorageClass、PVC；

注意：扩容云盘前需要为云盘打快照，以预防升级失败导致数据丢失；

云盘目前无法做到在线扩容，需要重启应用才可以实现，可以通过两种方法实现云盘扩容。下面以集群中部署Zookeeper为例介绍两种扩容方式，Zookeeper集群如下：

# kubectl get pod
NAME                            READY   STATUS    RESTARTS   AGE
zookeeper-default-zookeeper-0   1/1     Running   0          2m55s
zookeeper-default-zookeeper-1   1/1     Running   0          2m14s
zookeeper-default-zookeeper-2   1/1     Running   0          94s

# kubectl get pvc| grep zoo
datadir-zookeeper-default-zookeeper-0   Bound    d-8vb5teafaoa80ia7affg   20Gi       RWO            alicloud-disk-efficiency   3m12s
datadir-zookeeper-default-zookeeper-1   Bound    d-8vb60faf6epslbctnzka   20Gi       RWO            alicloud-disk-efficiency   2m31s
datadir-zookeeper-default-zookeeper-2   Bound    d-8vbidmq57w4df6k84zem   20Gi       RWO            alicloud-disk-efficiency   111s

# kubectl get pv| grep zoo
d-8vb5teafaoa80ia7affg   20Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-0   alicloud-disk-efficiency            3m17s
d-8vb60faf6epslbctnzka   20Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-1   alicloud-disk-efficiency            2m33s
d-8vbidmq57w4df6k84zem   20Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-2   alicloud-disk-efficiency            107s

暂停应用方式：

如果应用可以暂停服务，可以先停掉应用（删除），手动对每个依赖数据盘进行扩容，然后再启动应用。缺点是：应用会暂停一定时间；

上面应用使用了3个20Gi的云盘，分别挂载在3个pod上。目标为把三个云盘扩容到30Gi，主要步骤：

  1. 删除应用负载；
  2. 云盘控制台在线扩容云盘；
  3. 挂载云盘到某节点进行文件系统扩容；
  4. 更新PV、PVC的Size参数；
  5. 重启应用；

1. 删除应用：

删除zookeeper statefulset对象；

# kubectl delete sts zookeeper-default-zookeeper

# kubectl get pvc | grep zoo
datadir-zookeeper-default-zookeeper-0   Bound    d-8vb6ie0kwtyynpf4gu4l   20Gi       RWO            alicloud-disk-efficiency   22m
datadir-zookeeper-default-zookeeper-1   Bound    d-8vbhscszlr47rbot0boc   20Gi       RWO            alicloud-disk-efficiency   21m
datadir-zookeeper-default-zookeeper-2   Bound    d-8vb444t0f8xnicj9c2ov   20Gi       RWO            alicloud-disk-efficiency   21m

# kubectl get pv | grep zoo
d-8vb444t0f8xnicj9c2ov   20Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-2   alicloud-disk-efficiency            21m
d-8vb6ie0kwtyynpf4gu4l   20Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-0   alicloud-disk-efficiency            22m
d-8vbhscszlr47rbot0boc   20Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-1   alicloud-disk-efficiency            21m

2.3. 云盘扩容：

根据云盘文档，分别对3块云盘进行扩容：

https://help.aliyun.com/document_detail/113316.html

https://help.aliyun.com/document_detail/25452.html

注意：

  1. 扩容云盘的同时也要完成对文件系统的扩容，否则容器内存储空间依然是扩容前的大小。
  2. 注意您的云盘：如果是使用的整个云盘（未分区）挂载，则不能执行分区操作；例如：您的云盘是通过/dev/vdc 挂载的，而没有/dev/vdc1这样的分区设备符，则操作时不能对云盘进行分区；

4. 扩容PV、PVC

编辑pvc、pv所使用的StorageClass，配置allowVolumeExpansion: true标签；

# kubectl edit sc alicloud-disk-efficiency

# kubectl get sc alicloud-disk-efficiency -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2019-09-05T12:30:27Z"
  name: alicloud-disk-efficiency
  resourceVersion: "1675896"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/alicloud-disk-efficiency
  uid: f1071bcc-cfd8-11e9-81cd-00163e0804c2
parameters:
  type: cloud_efficiency
provisioner: alicloud/disk
reclaimPolicy: Delete
volumeBindingMode: Immediate

更新PVC的Size为30Gi：

# kubectl patch pvc datadir-zookeeper-default-zookeeper-0 -p '{"spec":{"resources":{"requests":{"storage":"30Gi"}}}}'
persistentvolumeclaim/datadir-zookeeper-default-zookeeper-0 patched
# kubectl patch pvc datadir-zookeeper-default-zookeeper-1 -p '{"spec":{"resources":{"requests":{"storage":"30Gi"}}}}'
persistentvolumeclaim/datadir-zookeeper-default-zookeeper-1 patched
# kubectl patch pvc datadir-zookeeper-default-zookeeper-2 -p '{"spec":{"resources":{"requests":{"storage":"30Gi"}}}}'
persistentvolumeclaim/datadir-zookeeper-default-zookeeper-2 patched

# kubectl get pvc | grep zoo
datadir-zookeeper-default-zookeeper-0   Bound    d-8vb6ie0kwtyynpf4gu4l   20Gi       RWO            alicloud-disk-efficiency   51m
datadir-zookeeper-default-zookeeper-1   Bound    d-8vbhscszlr47rbot0boc   20Gi       RWO            alicloud-disk-efficiency   50m
datadir-zookeeper-default-zookeeper-2   Bound    d-8vb444t0f8xnicj9c2ov   20Gi       RWO            alicloud-disk-efficiency   49m

# kubectl get pv | grep zoo
d-8vb444t0f8xnicj9c2ov   30Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-2   alicloud-disk-efficiency            49m
d-8vb6ie0kwtyynpf4gu4l   30Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-0   alicloud-disk-efficiency            51m
d-8vbhscszlr47rbot0boc   30Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-1   alicloud-disk-efficiency            50m

可见pv的大小已经更新为30Gi，pvc的大小会在Pod启动后自动更新；

5. 重新创建应用：

# kubectl get pod
NAME                            READY   STATUS    RESTARTS   AGE
zookeeper-default-zookeeper-0   1/1     Running   0          94s
zookeeper-default-zookeeper-1   1/1     Running   0          64s
zookeeper-default-zookeeper-2   1/1     Running   0          39s

# kubectl exec -ti zookeeper-default-zookeeper-0 sh
# df -h |grep zoo
/dev/vdb         30G   45M   30G   1% /var/lib/zookeeper

# kubectl get pvc | grep zoo
datadir-zookeeper-default-zookeeper-0   Bound    d-8vb6ie0kwtyynpf4gu4l   30Gi       RWO            alicloud-disk-efficiency   56m
datadir-zookeeper-default-zookeeper-1   Bound    d-8vbhscszlr47rbot0boc   30Gi       RWO            alicloud-disk-efficiency   56m
datadir-zookeeper-default-zookeeper-2   Bound    d-8vb444t0f8xnicj9c2ov   30Gi       RWO            alicloud-disk-efficiency   55m

# kubectl get pv | grep zoo
d-8vb444t0f8xnicj9c2ov   30Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-2   alicloud-disk-efficiency            55m
d-8vb6ie0kwtyynpf4gu4l   30Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-0   alicloud-disk-efficiency            56m
d-8vbhscszlr47rbot0boc   30Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-1   alicloud-disk-efficiency            56m

上面输出日志可以看出，文件系统已经扩容到30G，pv pvc的size也扩容到30G；

逐个Pod升级方式：

由多个Pod运行的服务，可以通过一个一个pod升级的方式实现业务不中断，而只是Running pod数量暂时减少。

还是以zookeeper为例，使用3个20Gi的云盘，分别挂载在3个pod上。目标为把三个云盘扩容到40Gi：

主要步骤：

  1. 修改某个Pod所对应pv的Zoneid label，然后删除pod；
  2. 登陆云盘控制台在线扩容云盘（扩容前最好对云盘打快照，容灾）；
  3. 挂载云盘到某节点进行文件系统扩容；
  4. 更新PV、PVC 的Size参数，以及Pv的ZoneId Lable；
  5. 对其他Pod重复1-4步骤；

1. 修改PV的ZoneId

# kubectl get pod zookeeper-default-zookeeper-0 -oyaml | grep pers -C 1
  - name: datadir
    persistentVolumeClaim:
      claimName: datadir-zookeeper-default-zookeeper-0

# kubectl get pvc datadir-zookeeper-default-zookeeper-0
NAME                                    STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS               AGE
datadir-zookeeper-default-zookeeper-0   Bound    d-8vbhscszlr47tgn0eheb   20Gi       RWO            alicloud-disk-efficiency   21m

上述日志得出zookeeper-default-zookeeper-0 使用的pvc为：datadir-zookeeper-default-zookeeper-0，对应的PV为d-8vbhscszlr47tgn0eheb；

更新PV Label，

如果pv中已经有如下label，则把zone对应的值改为”原来value -none“，即配置一个不存在的zone，让pod无法调度成功；
如果pv中没有如下label，则添加这些lables到pv；
#kubectl edit pv d-8vbhscszlr47tgn0eheb

  labels:
    failure-domain.beta.kubernetes.io/region: cn-zhangjiakou
    failure-domain.beta.kubernetes.io/zone: cn-zhangjiakou-a-none

删除Pod：zookeeper-default-zookeeper-0

# kubectl delete pod zookeeper-default-zookeeper-0

这时删除的pod一直处于Pending状态：

# kubectl get pod
NAME                            READY   STATUS    RESTARTS   AGE
zookeeper-default-zookeeper-0   0/1     Pending   0          9s
zookeeper-default-zookeeper-1   1/1     Running   0          24m
zookeeper-default-zookeeper-2   1/1     Running   0          24m

2.3. 云盘扩容：

根据云盘文档，分别对3块云盘进行库容：

https://help.aliyun.com/document_detail/113316.html

https://help.aliyun.com/document_detail/25452.html

注意：扩容云盘的同时也要完成对文件系统的扩容。

4. 扩容PV、PVC

编辑pvc、pv所使用的StorageClass，配置allowVolumeExpansion: true标签；

# kubectl edit sc alicloud-disk-efficiency

# kubectl get sc alicloud-disk-efficiency -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2019-09-05T12:30:27Z"
  name: alicloud-disk-efficiency
  resourceVersion: "1675896"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/alicloud-disk-efficiency
  uid: f1071bcc-cfd8-11e9-81cd-00163e0804c2
parameters:
  type: cloud_efficiency
provisioner: alicloud/disk
reclaimPolicy: Delete
volumeBindingMode: Immediate

更新PVC的Size为40Gi：

# kubectl patch pvc datadir-zookeeper-default-zookeeper-0 -p '{"spec":{"resources":{"requests":{"storage":"40Gi"}}}}'
persistentvolumeclaim/datadir-zookeeper-default-zookeeper-0 patched

# kubectl get pvc | grep datadir-zookeeper-default-zookeeper-0
datadir-zookeeper-default-zookeeper-0   Bound    d-8vbhscszlr47tgn0eheb   20Gi       RWO            alicloud-disk-efficiency   29m

# kubectl get pv | grep datadir-zookeeper-default-zookeeper-0
d-8vbhscszlr47tgn0eheb   40Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-0   alicloud-disk-efficiency            29m

可见pv的大小已经更新为40Gi，pvc的大小会在Pod启动后自动更新；

恢复PV的Lable，把zoneId的值恢复之前的value，删除相应labels（若之前没有这些labels）：

labels:
    failure-domain.beta.kubernetes.io/region: cn-zhangjiakou
    failure-domain.beta.kubernetes.io/zone: cn-zhangjiakou-a

检查容器挂载文件系统大小，pv、pvc大小：

# kubectl exec -ti zookeeper-default-zookeeper-0 sh
# df -h | grep zoo
/dev/vdb         40G   48M   40G   1% /var/lib/zookeeper
# kubectl get pvc | grep datadir-zookeeper-default-zookeeper-0
datadir-zookeeper-default-zookeeper-0   Bound    d-8vbhscszlr47tgn0eheb   40Gi       RWO            alicloud-disk-efficiency   33m

# kubectl get pv | grep datadir-zookeeper-default-zookeeper-0
d-8vbhscszlr47tgn0eheb   40Gi       RWO            Delete           Bound      default/datadir-zookeeper-default-zookeeper-0   alicloud-disk-efficiency            33m

可见文件系统、pv、pvc都已经实现了扩容；

5. 对其他pod所对应的pvc、pv、云盘进行上述扩容

上一篇：Kubernetes集群中云盘在线扩容


下一篇：【面小易-面经03】阿里巴巴C++研发实习生面试经验总结