本文记录使用Velero Restic快速完成云原生应用及PV数据从GKE到至ACK的迁移的实践过程。 此过程也同样适用于自建Kubernetes集群内的应用及PV数据迁移至ACK。
实践步骤概览
（1）创建GKE集群（或自建Kubernetes集群）
（2）在GKE集群上部署示例应用Jenkins Application并执行一个构建任务
（3）创建ACK集群
（4）在ACK集群中部署Minio Service用于应用迁移时数据中转服务
（5）在GKE上部署Velero并备份整个Jenkins Application（或其他任意应用）
（6）把Jenkins Application使用的容器镜像同步到阿里云容器镜像仓库
（7）在ACK上创建Jenkins Application所使用的StorageClass
（8）在ACK上部署Velero并恢复整个Jenkins Application
（9）在ACK上替换Jenkins Application所使用的容器镜像
（10）验证Jenkins Application能否正常提供服务及其构建任务是否存在
环境物料清单
（1）GKE集群（或自建Kubernetes集群）
（2）ACK集群
创建GKE集群并部署示例应用Jenkins Application
(1) 在MarketPlace中找到Jenkins应用并配置部署到GKE集群的jenkins命名空间下：
(2)应用正在完成部署：
(3)查看Ingress并访问jenkins服务：
(4)Jenkins应用的初始化配置：
(5)Jenkin应用上创建一个名为gke-to-ack的任务并执行构建：
这个步骤相当于往PV存储卷中写数据。
（6）Jenkins应用的k8s资源列表如下
$ kubectl -n jenkins get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
jenkins-jenkins-deployment 1 1 1 1 177m
$ kubectl -n jenkins get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins-jenkins-agents-connector ClusterIP 10.39.241.75 <none> 50000/TCP 177m
jenkins-jenkins-ui NodePort 10.39.253.141 <none> 8080:31759/TCP 177m
$ kubectl -n jenkins get ing
NAME HOSTS ADDRESS PORTS AGE
jenkins-jenkins-ui * xx.xx.xx.xx 80, 443 177m
$ kubectl -n jenkins get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
jenkins-jenkins-pvc Bound pvc-e64eacc4-a482-11e9-bfa9-42010a8000da 8Gi RWO standard 177m
$ kubectl -n jenkins get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-e64eacc4-a482-11e9-bfa9-42010a8000da 8Gi RWO Delete Bound jenkins/jenkins-jenkins-pvc standard 177m创建ACK集群并部署Minio对象存储应用
Minio对象存储服务用于存储PV持久化数据， 我们即将支持阿里云OSS对象存储服务。
(1) 自定义替换minio-deploy.yaml中的 MINIO_ACCESS_KEY 和 MINIO_SECRET_KEY值并部署：
kubectl apply -f minio-deploy.yaml---
apiVersion: v1
kind: Namespace
metadata:
name: minio
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
namespace: minio
name: minio
labels:
component: minio
spec:
strategy:
type: Recreate
template:
metadata:
labels:
component: minio
spec:
volumes:
- name: storage
emptyDir: {}
- name: config
emptyDir: {}
containers:
- name: minio
image: minio/minio:latest
imagePullPolicy: IfNotPresent
args:
- server
- /storage
- --config-dir=/config
env:
- name: MINIO_ACCESS_KEY
value: "<your MINIO_ACCESS_KEY>"
- name: MINIO_SECRET_KEY
value: "your MINIO_SECRET_KEY"
ports:
- containerPort: 9000
volumeMounts:
- name: storage
mountPath: "/storage"
- name: config
mountPath: "/config"
---
apiVersion: v1
kind: Service
metadata:
namespace: minio
name: minio
labels:
component: minio
spec:
# ClusterIP is recommended for production environments.
# Change to NodePort if needed per documentation,
# but only if you run Minio in a test/trial environment, for example with Minikube.
type: LoadBalancer
ports:
- port: 9000
targetPort: 9000
protocol: TCP
selector:
component: minio(2) 查看minio LoadBalancer信息并登陆应用：
$ kubectl -n minio get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
minio LoadBalancer 172.21.2.216 xxx.xxx.xxx.xx 9000:30912/TCP 2d4h(3) 替换minio-job.yaml中的minio_server_url minino_access_key minio_secret_key并运行job在minio中创建名为velero的bucket：
kubectl apply -f minio-job.yamlapiVersion: batch/v1
kind: Job
metadata:
namespace: minio
name: minio-setup
labels:
component: minio
spec:
template:
metadata:
name: minio-setup
spec:
restartPolicy: OnFailure
volumes:
- name: config
emptyDir: {}
containers:
- name: mc
image: minio/mc:latest
imagePullPolicy: IfNotPresent
command:
- /bin/sh
- -c
- "mc --config-dir=/config config host add velero <your minio_server_url> <your minino_access_key> <your minio_secret_key> && mc --config-dir=/config mb -p velero/velero"
volumeMounts:
- name: config
mountPath: "/config"(4) 查看bucket是否创建成功：
在GKE和ACK上部署velero
(1) 安装velero client：
请从official release下载最新版本的velero客户端：
(2) 安装velero server：
创建credentials-velero文件并设置aws_access_key_id 与 aws_secret_access_key的值
[default]
aws_access_key_id = <your minio_access_key>
aws_secret_access_key = <your minio_secret_key>ACK集群中请替换minio_server_url的值并指定image参数部署velero server
velero install --provider aws --image registry.cn-hangzhou.aliyuncs.com/acs/velero:latest --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=<your minio_server_url> --use-restic --waitGKE中请替换minio_server_url的值并部署velero server
velero install --provider aws --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=<your minio_server_url> --use-restic --wait在GKE集群中备份Jenkins Application
(1)在备份带volume信息的pod之前，我们要通过给pod加annotation来告诉velero哪些pod需要包含volume数据， 查看jenkins应用的Deployment资源：
$ kubectl -n jenkins get deploy jenkins-jenkins-deployment -oyaml(2)为pod加annotation:
$ kubectl -n jenkins get po
NAME READY STATUS RESTARTS AGE
jenkins-deployer-zh5p4 0/1 Completed 0 109m
jenkins-jenkins-deployment-7df86c64d4-tqqlr 1/1 Running 0 109m
$ kubectl -n jenkins annotate pod/jenkins-jenkins-deployment-7df86c64d4-tqqlr backup.velero.io/backup-volumes=jenkins-jenkins-pvc
pod/jenkins-jenkins-deployment-7df86c64d4-tqqlr annotated(3)创建备份
$ velero backup create gcloud-jenkins-backup-restic --include-namespaces jenkins --wait
Backup request "gcloud-jenkins-backup-restic" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting - your backup will continue in the background.
.......................
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe gcloud-jenkins-backup-restic` and `velero backup logs gcloud-jenkins-backup-restic`.(4) 查看备份
$ ./velero backup get
NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR
gcloud-jenkins-backup-restic Completed 2019-07-12 18:48:48 +0800 +08 29d default <none>（5）登录Minio Server查看
同步（或批量）迁移容器镜像
把Jenkins应用使用的容器镜像导入到阿里云容器镜像仓库，导入到的地址为：
registry.cn-hangzhou.aliyuncs.com/haoshuwei/jenkins:2.150.3批量迁移可参考：https://github.com/AliyunContainerService/sync-repo.git
在ACK集群中创建Jenkins应用所使用的StorageClass standard
我们知道不同的云厂商，后端的存储基础设施是不一样的， 我们需要先创建一个相同名称的StorageClass来屏蔽对底层存储基础设施差异的感知。
$ kubectl apply -f storageclass.yamlkind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
name: standard
provisioner: alicloud/disk
parameters:
type: cloud
reclaimPolicy: Delete在ACK集群中恢复Jenkins Application
查看备份信息：
$ velero backup get
NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR
gcloud-jenkins-backup-restic Completed 2019-07-12 18:48:48 +0800 CST 29d default <none>恢复应用：
$ velero restore create --from-backup gcloud-jenkins-backup-restic此处需要编辑deployment修改image为registry.cn-hangzhou.aliyuncs.com/haoshuwei/jenkins:2.150.3
查看restore的状态：
$ velero restore describe gcloud-jenkins-backup-restic-20190712190536
$ velero restore logs gcloud-jenkins-backup-restic-20190712190536restore完成后的状态为：
$ velero restore get
NAME BACKUP STATUS WARNINGS ERRORS CREATED SELECTOR
gcloud-jenkins-backup-restic-20190712190536 gcloud-jenkins-backup-restic Completed 0 0 2019-07-12 19:05:36 +0800 CST <none>查看ACK集群上jenkins应用的ingress并访问服务进行验证
$ kubectl -n jenkins get ing
NAME HOSTS ADDRESS PORTS AGE
jenkins-jenkins-ui * xx.xx.xx.xx 80, 443 56m至此，一个带pv存储的jenkins应用被完整地从GKE迁移到ACK上。
参考链接：
https://velero.io/docs/v1.0.0/get-started/
https://velero.io/docs/v1.0.0/restic/#troubleshooting
https://github.com/heptio/velero
https://github.com/AliyunContainerService/velero-plugin