本文记录使用Velero Restic快速完成云原生应用迁移至ACK集群的实践过程。
0. 实践步骤概览
（1）创建GKE集群（或自建Kubernetes集群）
（2）在GKE集群上部署示例应用Jenkins Application并执行一个构建任务
（3）创建ACK集群
（4）在ACK集群中部署Minio Service用于应用迁移时数据中转服务
（5）在GKE上部署Velero并备份整个Jenkins Application
（6）把Jenkins Application使用的容器镜像同步到阿里云容器镜像仓库
（7）在ACK上创建Jenkins Application所使用的StorageClass
（8）在ACK上部署Velero并恢复整个Jenkins Application
（9）在ACK上替换Jenkins Application所使用的容器镜像
（10）验证Jenkins Application能否正常提供服务及其构建任务是否存在
1. 环境物料清单
GKE集群（或自建Kubernetes集群）
ACK集群
2. 创建GKE集群并部署示例应用Jenkins Application
(1)在MarketPlace中找到Jenkins应用并配置部署到GKE集群的jenkins命名空间下：
(2)应用正在完成部署：
(3)查看Ingress并访问jenkins服务：
(4)Jenkins应用的初始化配置：
(5)Jenkin应用上创建一个名为gke-to-ack的任务并执行构建：
（6）Jenkins应用的k8s资源列表如下
$ kubectl -n jenkins get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
jenkins-jenkins-deployment 1 1 1 1 177m
$ kubectl -n jenkins get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins-jenkins-agents-connector ClusterIP 10.39.241.75 <none> 50000/TCP 177m
jenkins-jenkins-ui NodePort 10.39.253.141 <none> 8080:31759/TCP 177m
$ kubectl -n jenkins get ing
NAME HOSTS ADDRESS PORTS AGE
jenkins-jenkins-ui * xx.xx.xx.xx 80, 443 177m
$ kubectl -n jenkins get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
jenkins-jenkins-pvc Bound pvc-e64eacc4-a482-11e9-bfa9-42010a8000da 8Gi RWO standard 177m
$ kubectl -n jenkins get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-e64eacc4-a482-11e9-bfa9-42010a8000da 8Gi RWO Delete Bound jenkins/jenkins-jenkins-pvc standard 177m
3. 创建ACK集群并部署Minio对象存储应用
(1) 自定义替换minio-deploy.yaml中的 MINIO_ACCESS_KEY 和 MINIO_SECRET_KEY值并部署：
kubectl apply -f minio-deploy.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: minio
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
namespace: minio
name: minio
labels:
component: minio
spec:
strategy:
type: Recreate
template:
metadata:
labels:
component: minio
spec:
volumes:
- name: storage
emptyDir: {}
- name: config
emptyDir: {}
containers:
- name: minio
image: minio/minio:latest
imagePullPolicy: IfNotPresent
args:
- server
- /storage
- --config-dir=/config
env:
- name: MINIO_ACCESS_KEY
value: "<your MINIO_ACCESS_KEY>"
- name: MINIO_SECRET_KEY
value: "your MINIO_SECRET_KEY"
ports:
- containerPort: 9000
volumeMounts:
- name: storage
mountPath: "/storage"
- name: config
mountPath: "/config"
---
apiVersion: v1
kind: Service
metadata:
namespace: minio
name: minio
labels:
component: minio
spec:
# ClusterIP is recommended for production environments.
# Change to NodePort if needed per documentation,
# but only if you run Minio in a test/trial environment, for example with Minikube.
type: LoadBalancer
ports:
- port: 9000
targetPort: 9000
protocol: TCP
selector:
component: minio
(2) 查看minio LoadBalancer信息并登陆应用：
$ kubectl -n minio get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
minio LoadBalancer 172.21.2.216 xxx.xxx.xxx.xx 9000:30912/TCP 2d4h
(3) 替换minio-job.yaml中的minio_server_url minino_access_key minio_secret_key并运行job在minio中创建名为velero的bucket：
kubectl apply -f minio-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
namespace: minio1
name: minio1-setup
labels:
component: minio1
spec:
template:
metadata:
name: minio1-setup
spec:
restartPolicy: OnFailure
volumes:
- name: config
emptyDir: {}
containers:
- name: mc
image: minio/mc:latest
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- "mc --config-dir=/config config host add velero <your minio_server_url> <your minino_access_key> <your minio_secret_key> && mc --config-dir=/config mb -p velero/velero"
volumeMounts:
- name: config
mountPath: "/config"
(4) 查看bucket是否创建成功：
4. 在GKE和ACK上部署velero
(1) 安装velero client：
请从official release下载最新版本的velero客户端：
(2) 安装velero server：
创建credentials-velero文件并设置aws_access_key_id 与 aws_secret_access_key的值
[default]
aws_access_key_id = <your minio_access_key>
aws_secret_access_key = <your minio_secret_key>
ACK集群中请替换minio_server_url的值并指定image参数部署velero server
velero install --provider aws --image registry.cn-hangzhou.aliyuncs.com/acs/velero:latest --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=<your minio_server_url> --use-restic --wait
GKE中请替换minio_server_url的值并部署velero server
velero install --provider aws --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=<your minio_server_url> --use-restic --wait
5. 在GKE集群中备份Jenkins Application
(1)在备份带volume信息的pod之前，我们要通过给pod加annotation来告诉velero哪些pod需要包含volume数据， 查看jenkins应用的Deployment资源：
$ kubectl -n jenkins get deploy jenkins-jenkins-deployment -oyaml
(2)为pod加annotation:
$ kubectl -n jenkins get po
NAME READY STATUS RESTARTS AGE
jenkins-deployer-zh5p4 0/1 Completed 0 109m
jenkins-jenkins-deployment-7df86c64d4-tqqlr 1/1 Running 0 109m
$ kubectl -n jenkins annotate pod/jenkins-jenkins-deployment-7df86c64d4-tqqlr backup.velero.io/backup-volumes=jenkins-jenkins-pvc
pod/jenkins-jenkins-deployment-7df86c64d4-tqqlr annotated
(3)创建备份
$ velero backup create gcloud-jenkins-backup-restic --include-namespaces jenkins --wait
Backup request "gcloud-jenkins-backup-restic" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
.......................
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe gcloud-jenkins-backup-restic` and `velero backup logs gcloud-jenkins-backup-restic`.
(4) 查看备份
$ ./velero backup get
NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR
gcloud-jenkins-backup-restic Completed 2019-07-12 18:48:48 +0800 +08 29d default <none>
（5）登录Minio Server查看
5. 同步（或批量）迁移容器镜像
把Jenkins应用使用的容器镜像导入到阿里云容器镜像仓库，导入到的地址为：
registry.cn-hangzhou.aliyuncs.com/haoshuwei/jenkins:2.150.3
批量迁移可参考：https://github.com/AliyunContainerService/sync-repo.git
6. 在ACK集群中创建Jenkins应用所使用的StorageClass standard
$ kubectl apply -f storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
name: standard
provisioner: alicloud/disk
parameters:
type: cloud
reclaimPolicy: Delete
7. 在ACK集群中恢复Jenkins Application
查看备份信息：
$ velero backup get
NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR
gcloud-jenkins-backup-restic Completed 2019-07-12 18:48:48 +0800 CST 29d default <none>
恢复应用：
$ velero restore create --from-backup gcloud-jenkins-backup-restic
此处需要编辑deployment修改image为registry.cn-hangzhou.aliyuncs.com/haoshuwei/jenkins:2.150.3
查看restore的状态：
$ velero restore describe gcloud-jenkins-backup-restic-20190712190536
$ velero restore logs gcloud-jenkins-backup-restic-20190712190536
restore完成后的状态为：
$ velero restore get
NAME BACKUP STATUS WARNINGS ERRORS CREATED SELECTOR
gcloud-jenkins-backup-restic-20190712190536 gcloud-jenkins-backup-restic Completed 0 0 2019-07-12 19:05:36 +0800 CST <none>
8. 查看ACK集群上jenkins应用的ingress并访问服务进行验证
$ kubectl -n jenkins get ing
NAME HOSTS ADDRESS PORTS AGE
jenkins-jenkins-ui * xx.xx.xx.xx 80, 443 56m
至此，一个带pv存储的jenkins应用被完整地从GKE迁移到ACK上。
参考链接：
https://velero.io/docs/v1.0.0/get-started/
https://velero.io/docs/v1.0.0/restic/#troubleshooting
https://github.com/heptio/velero
https://github.com/AliyunContainerService/velero-plugin
本文作者：流生
本文为云栖社区原创内容，未经允许不得转载。