第四章 部署 Kubernetes 持久化存储 GlusterFS
文章目录
- 第四章 部署 Kubernetes 持久化存储 GlusterFS
- 前言
- 一、准备工作
- 二、Kubernetes 安装 Glusterfs 集群
- 三、安装配置 heketi
- 四、GlusterFS 集群查看验证
- 总结
前言
提示：这里要用到得有 Heketi 和 GlusterFS 存储系统，Heketi 是用来管理 GlusterFS 卷的生命周期的，并提供了一个RESTful API 接口供 Kubernetes 调用，因为 GlusterFS 没有提供 API 调用的方式，所以我们借助 heketi。这里两者得关系我不过多介绍，后期会专门出一期文章进行介绍。
一、准备工作
1、下载 gluster-kubernetes-master
https://github.com/gluster/gluster-kubernetes/archive/refs/heads/master.zip
2、GlusterFS 客户端安装
所有节点上安装的 GlusterFS 客户端版本应尽可能接近服务器的版本。必须至少有三个节点用于 glusterfs ； 每个节点必须至少连接一个裸磁盘设备，以供 heketi 使用。域名解析这里不再介绍，安装 Kubernetes 集群时已经配置完成。
安装方式
[root@k8snode01 ~]# yum -y install centos-release-gluster
[root@k8snode01 ~]# yum -y install glusterfs-fuse
3、加载相应模块
所有节点都要执行
modprobe dm_thin_pool
modprobe dm_snapshot
modprobe dm_mirror
4、K8S 集群 node 添加标签
### 节点添加标签命令
kubectl label node k8s-gfs-01 k8s-gfs-02 k8s-gfs-03 k8s-gfs-04 k8s-gfs-05 k8s-gfs-06 storagenode=glusterfs
### 给添加 glusterfs 得节点添加污点，作用是只在此节点运行glusterfs
kubectl taint nodes k8s-gfs-01 k8s-gfs-02 k8s-gfs-03 k8s-gfs-04 k8s-gfs-05 k8s-gfs-06 storagenode=glusterfs:NoSchedule
二、Kubernetes 安装 Glusterfs 集群
1、修改 gluster-kubernetes-master 得 yaml 模板文件。
[root@k8s-master-01 kube-templates]# pwd
/home/yaml/gluster-kubernetes-master/deploy/kube-templates
[root@k8s-master-01 kube-templates]# ls
deploy-heketi-deployment.yaml gluster-s3-storageclass.yaml heketi-deployment.yaml heketi-svc.yaml
glusterfs-daemonset.yaml gluster-s3-template.yaml heketi-deployment.yaml_bak
gluster-s3-pvcs.yaml heketi-bootstrap.json heketi-service-account.yaml
1.1 deploy-heketi-deployment.yaml
因为我使用的是本地 Harbor 镜像仓库，所有需要对默认的 image 进行更改
1.2 glusterfs-daemonset.yaml
因为我们设置了污点，并对节点增加了标签，所有我们需要通过选取节点，设置容忍，来让 pod 调度到指定得节点上。
1.3 heketi-deployment.yaml
其它的 yaml 文件只需要更改对应的镜像就可以，这里不再一一介绍。
2、修改 GlusterFS 拓扑
修改后如下：
[root@k8s-master-01 deploy]# cat topology.json
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"k8s-gfs-01"
],
"storage": [
"192.168.1.87"
]
},
"zone": 1
},
"devices": [
"/dev/sdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"k8s-gfs-02"
],
"storage": [
"192.168.1.88"
]
},
"zone": 1
},
"devices": [
"/dev/sdb"
]
},
{
"node": {
"hostnames": {
"manage": [
"k8s-gfs-03"
],
"storage": [
"192.168.1.89"
]
},
"zone": 1
},
"devices": [
"/dev/sdb"
]
}
]
}
]
}
注释：
k8s-gfs-01：主机名，这里时之前设置好的，需要在每个主机 hosts 文件加入解析。
192.168.1.87：这个为对应得主机 IP 。
/dev/sdb：这个为主机得裸盘，用来存储数据。
3、配置 heketi.json 文件
修改后如下：
{
"_port_comment": "Heketi Server Port Number",
"port": "8080",
"_use_auth": "Enable JWT authorization. Please enable for deployment",
"use_auth": true,
"_jwt": "Private keys for access",
"jwt": {
"_admin": "Admin has access to all APIs",
"admin": {
"key": "adminkey"
},
"_user": "User only has access to /volumes endpoint",
"user": {
"key": "userkey"
}
},
"_glusterfs_comment": "GlusterFS Configuration",
"glusterfs": {
"_executor_comment": "Execute plugin. Possible choices: mock, kubernetes, ssh",
"executor": "kubernetes",
"_db_comment": "Database file name",
"db": "/var/lib/heketi/heketi.db",
"kubeexec": {
"rebalance_on_expansion": true
},
"sshexec": {
"rebalance_on_expansion": true,
"keyfile": "/etc/heketi/private_key",
"fstab": "/etc/fstab",
"port": "22",
"user": "root",
"sudo": false
}
},
"_backup_db_to_kube_secret": "Backup the heketi database to a Kubernetes secret when running in Kubernetes. Default is off.",
"backup_db_to_kube_secret": false
}
主要修改截图如下：
4、修改 Glusterfs 集群创建脚本
按照截图中得修改，修改行数为 924 行
5、最后，启动部署脚本
### 创建独立得名称空间
kubectl create ns glusterfs
### 一键创建脚本运行
./gk-deploy -g -n glusterfs --admin-key adminkey --user-key userkey
[Y]es, [N]o? [Default: Y]: y
Using Kubernetes CLI.
Using namespace "glusterfs".
Checking for pre-existing resources...
GlusterFS pods ... not found.
deploy-heketi pod ... not found.
heketi pod ... not found.
gluster-s3 pod ... not found.
Creating initial resources ... serviceaccount/heketi-service-account created
clusterrolebinding.rbac.authorization.k8s.io/heketi-sa-view created
clusterrolebinding.rbac.authorization.k8s.io/heketi-sa-view labeled
OK
node/k8s-gfs-01 not labeled
node/k8s-gfs-02 not labeled
node/k8s-gfs-03 not labeled
daemonset.apps/glusterfs created
Waiting for GlusterFS pods to start ... OK
secret/heketi-config-secret created
secret/heketi-config-secret labeled
service/deploy-heketi created
deployment.apps/deploy-heketi created
Waiting for deploy-heketi pod to start ... OK
Creating cluster ... ID: 3f8822d3db35782c5b1057c8fd45a432
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node k8s-gfs-01 ... ID: fb115580c41b37d6d76f10bbd59a04db
Adding device /dev/sdb ... OK
Creating node k8s-gfs-02 ... ID: 3311fb1eb8665d9ac9faca87beb07e13
Adding device /dev/sdb ... OK
Creating node k8s-gfs-03 ... ID: ce797980ef5529efae5bab6c6293d550
Adding device /dev/sdb ... OK
heketi topology loaded.
Saving /tmp/heketi-storage.json
secret/heketi-storage-secret created
endpoints/heketi-storage-endpoints created
service/heketi-storage-endpoints created
job.batch/heketi-storage-copy-job created
service/heketi-storage-endpoints labeled
pod "deploy-heketi-6d8f67d659-7l6zr" deleted
service "deploy-heketi" deleted
deployment.apps "deploy-heketi" deleted
replicaset.apps "deploy-heketi-6d8f67d659" deleted
job.batch "heketi-storage-copy-job" deleted
secret "heketi-storage-secret" deleted
service/heketi created
deployment.apps/heketi created
Waiting for heketi pod to start ... OK
heketi is now running and accessible via http://10.100.254.93:8080 . To run
administrative commands you can install 'heketi-cli' and use it as follows:
# heketi-cli -s http://10.100.254.93:8080 --user admin --secret '<ADMIN_KEY>' cluster list
You can find it at https://github.com/heketi/heketi/releases . Alternatively,
use it from within the heketi pod:
# /usr/bin/kubectl -n glusterfs exec -i heketi-84bdf7d88f-qhtrb -- heketi-cli -s http://localhost:8080 --user admin --secret '<ADMIN_KEY>' cluster list
For dynamic provisioning, create a StorageClass similar to this:
---
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
name: glusterfs-storage
provisioner: kubernetes.io/glusterfs
parameters:
resturl: "http://10.100.254.93:8080"
restuser: "user"
restuserkey: "userkey"
Deployment complete!
以上说明部署完成
提示：部署脚本更多参数参考：https://github.com/gluster/gluster-kubernetes/blob/master/deploy/gk-deploy
注意：若部署失败，需要通过下方式彻底删除后重新部署：
[root@k8smaster01 deploy]# ./gk-deploy --abort --admin-key admin123 --user-key xianghy -y -n glusterfs
[root@k8smaster01 deploy]# kubectl delete -f kube-templates/ -n glusterfs
glusterfs node所有节点需要执行如下彻底清理：
[root@k8snode01 ~]# dmsetup ls
[root@k8snode01 ~]# dmsetup remove_all
[root@k8snode01 ~]# rm -rf /var/log/glusterfs/
[root@k8snode01 ~]# rm -rf /var/lib/heketi
[root@k8snode01 ~]# rm -rf /var/lib/glusterd/
[root@k8snode01 ~]# rm -rf /etc/glusterfs/
[root@k8snode01 ~]# dd if=/dev/zero of=/dev/sdb bs=512k count=1
[root@k8snode01 ~]# wipefs -af /dev/sdb
三、安装配置 heketi
由于在master节点管理heketi需要进入heketi容器或者使用kubectl exec -ti 方式，建议直接在master节点安装heketi客户端，直接管理。
我这里下载的是 v10 版本。直接解压，然后复制到 /usr/local/bin/heketi-cli 下即可使用
[root@k8s-master-01 ~]# heketi-cli -v
heketi-cli v10.0.0
配置 heketi
[root@k8s-master-01 profile.d]# echo "export HEKETI_CLI_SERVER=http://$(kubectl get svc heketi -n glusterfs -o go-template='{{.spec.clusterIP}}'):8080" >> /etc/profile.d/heketi.sh
[root@k8s-master-01 profile.d]# vim ~/.bashrc
[root@k8s-master-01 profile.d]# echo "alias heketi-cli='heketi-cli --user admin --secret adminkey'" >> ~/.bashrc
[root@k8s-master-01 profile.d]# vim ~/.bashrc
[root@k8s-master-01 profile.d]# source /etc/profile.d/heketi.sh
[root@k8s-master-01 profile.d]# source ~/.bashrc
[root@k8s-master-01 profile.d]# echo $HEKETI_CLI_SERVER
http://10.96.3.250:8080
四、GlusterFS 集群查看验证
1、查看集群列表
[root@k8s-master-01 ~]# heketi-cli cluster list
Clusters:
Id:3f8822d3db35782c5b1057c8fd45a432 [file][block]
2、查看集群详细信息
[root@k8s-master-01 ~]# heketi-cli cluster list
Clusters:
Id:3f8822d3db35782c5b1057c8fd45a432 [file][block]
[root@k8s-master-01 ~]# heketi-cli topology info 3f8822d3db35782c5b1057c8fd45a432
Cluster Id: 3f8822d3db35782c5b1057c8fd45a432
File: true
Block: true
Volumes:
Name: heketidbstorage
Size: 2
Id: d9e7541055a3bbccf64668885dd33bec
Cluster Id: 3f8822d3db35782c5b1057c8fd45a432
Mount: 192.168.1.88:heketidbstorage
Mount Options: backup-volfile-servers=192.168.1.89,192.168.1.87
Durability Type: replicate
Replica: 3
Snapshot: Disabled
Bricks:
Id: 2d481e2d977a4a10d56c95a178d20713
Path: /var/lib/heketi/mounts/vg_a287fa1cc1ddb5ee1823df641ed39a99/brick_2d481e2d977a4a10d56c95a178d20713/brick
Size (GiB): 2
Node: ce797980ef5529efae5bab6c6293d550
Device: a287fa1cc1ddb5ee1823df641ed39a99
Id: 43d4d2bb64ced333f5e8ed1832fcd3df
Path: /var/lib/heketi/mounts/vg_4aa603de16ffa6c9a900b56f0fb98a40/brick_43d4d2bb64ced333f5e8ed1832fcd3df/brick
Size (GiB): 2
Node: fb115580c41b37d6d76f10bbd59a04db
Device: 4aa603de16ffa6c9a900b56f0fb98a40
Id: 95afe23cb90845ce2cba3bfeda4e8192
Path: /var/lib/heketi/mounts/vg_b726931a77ccb1bae2a4359a0de710f8/brick_95afe23cb90845ce2cba3bfeda4e8192/brick
Size (GiB): 2
Node: 3311fb1eb8665d9ac9faca87beb07e13
Device: b726931a77ccb1bae2a4359a0de710f8
Nodes:
Node Id: 3311fb1eb8665d9ac9faca87beb07e13
State: online
Cluster Id: 3f8822d3db35782c5b1057c8fd45a432
Zone: 1
Management Hostnames: k8s-gfs-02
Storage Hostnames: 192.168.1.88
Devices:
Id:b726931a77ccb1bae2a4359a0de710f8 State:online Size (GiB):299 Used (GiB):2 Free (GiB):297
Known Paths: /dev/sdb
Bricks:
Id:95afe23cb90845ce2cba3bfeda4e8192 Size (GiB):2 Path: /var/lib/heketi/mounts/vg_b726931a77ccb1bae2a4359a0de710f8/brick_95afe23cb90845ce2cba3bfeda4e8192/brick
Node Id: ce797980ef5529efae5bab6c6293d550
State: online
Cluster Id: 3f8822d3db35782c5b1057c8fd45a432
Zone: 1
Management Hostnames: k8s-gfs-03
Storage Hostnames: 192.168.1.89
Devices:
Id:a287fa1cc1ddb5ee1823df641ed39a99 State:online Size (GiB):299 Used (GiB):2 Free (GiB):297
Known Paths: /dev/sdb
Bricks:
Id:2d481e2d977a4a10d56c95a178d20713 Size (GiB):2 Path: /var/lib/heketi/mounts/vg_a287fa1cc1ddb5ee1823df641ed39a99/brick_2d481e2d977a4a10d56c95a178d20713/brick
Node Id: fb115580c41b37d6d76f10bbd59a04db
State: online
Cluster Id: 3f8822d3db35782c5b1057c8fd45a432
Zone: 1
Management Hostnames: k8s-gfs-01
Storage Hostnames: 192.168.1.87
Devices:
Id:4aa603de16ffa6c9a900b56f0fb98a40 State:online Size (GiB):299 Used (GiB):2 Free (GiB):297
Known Paths: /dev/sdb
Bricks:
Id:43d4d2bb64ced333f5e8ed1832fcd3df Size (GiB):2 Path: /var/lib/heketi/mounts/vg_4aa603de16ffa6c9a900b56f0fb98a40/brick_43d4d2bb64ced333f5e8ed1832fcd3df/brick
查看所有 node
[root@k8s-master-01 ~]# heketi-cli node list
Id:3311fb1eb8665d9ac9faca87beb07e13 Cluster:3f8822d3db35782c5b1057c8fd45a432
Id:ce797980ef5529efae5bab6c6293d550 Cluster:3f8822d3db35782c5b1057c8fd45a432
Id:fb115580c41b37d6d76f10bbd59a04db Cluster:3f8822d3db35782c5b1057c8fd45a432
3、node 节点信息
heketi-cli node info <node ID>
4、列出所有卷
[root@k8s-master-01 ~]# heketi-cli volume list
Id:d9e7541055a3bbccf64668885dd33bec Cluster:3f8822d3db35782c5b1057c8fd45a432 Name:heketidbstorage
5、创建卷，默认为3副本的replica模式
[root@k8s-master-01 ~]# heketi-cli volume create --size=2 --replica=2
Name: vol_2c8de2e2203398b5a5767c6da0ecd210
Size: 2
Volume Id: 2c8de2e2203398b5a5767c6da0ecd210
Cluster Id: 3f8822d3db35782c5b1057c8fd45a432
Mount: 192.168.1.88:vol_2c8de2e2203398b5a5767c6da0ecd210
Mount Options: backup-volfile-servers=192.168.1.89,192.168.1.87
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distribute Count: 1
Replica Count: 2
6、删除卷
[root@k8s-master-01 ~]# heketi-cli volume delete 2c8de2e2203398b5a5767c6da0ecd210
Volume 2c8de2e2203398b5a5767c6da0ecd210 deleted
总结
以上就是 GlusterFS 集群得部署方案，后期文章会根据实际项目来演示 K8s 如何结合 Glusterfs 集群实现持久化存储，根据需求动态创建 PV，动态绑定、扩容、缩容 PVC。尽情期待！