本文记录使用Velero快速完成云原生应用及PV数据从自建Kubernetes迁移到ACK的实践过程。 此过程也同样适用于其他云厂商Kubernetes集群内的应用及PV数据迁移至ACK。
在本示例中， 我们将对自建Kubernetes集群中的一个wordpress应用整体迁移上云到ACK，其中数据备份采用阿里云OSS服务，安全稳定。
Kubernetes应用及PV数据迁移ACK概览
- 容器镜像迁移上云到ACR
- 自建Kubernetes集群中部署Velero并备份应用wordpress到OSS
- ACK部署Velero并恢复应用wordpress到ACK集群
- 调整wordpress应用使之充分使用ACK的优势
- 访问和验证wordpress应用服务是否正常
Kubernetes应用及PV数据迁移
1 容器镜像迁移上云到ACR
自建Kubernetes集群通常位于用户自己的IDC中，容器镜像的存储也会使用自建镜像仓库， 在Kubernetes应用迁移上云之前， 首先要做的就是容器镜像迁移上云到ACR。 本示例中wordpress应用涉及的容器镜像有：
registry.api.paas.com:5000/admin/wordpress:latest
registry.api.paas.com:5000/admin/mysql:8迁移上云后为：
registry.cn-hangzhou.aliyuncs.com/ack-migration/wordpress:latest
registry.cn-hangzhou.aliyuncs.com/ack-migration/mysql:8如果需要大批量迁移容器镜像， 请参考容器镜像迁移 按步骤操作。
2 ACK及自建Kubernetes集群中部署Velero
请按照以下步骤，分别在ACK和自建Kubernetes集群中部署Velero。
2.1 安装 Velero 客户端
由于使用 velero 备份Kubernetes PV数据的功能还未正式合并到社区项目的主分支， velero的客户端请从以下链接下载并安装：
$ curl -o /usr/bin/velero https://public-bucket-1.oss-cn-hangzhou.aliyuncs.com/velero && chmod +x /usr/bin/velero2.2 创建OSS Bucket
velero 要求预先 创建一个 OSS Bucket(此处放链接转向相关帮助文档) 来存储 Kubernetes 应用数据及其PV数据， 推荐每个Kubernetes集群单独使用各自的OSS Bucket。 此文档示例中的OSS Bucket为 cn-hangzhou 区域下的 ls-velero bucket。
2.3 创建RAM账号并生成AK
如果您使用主账号AK，可以跳过此步骤。 创建子账号并授予以下权限(此处放链接转向相关帮助文档) ：
{
"Version": "1",
"Statement": [
{
"Action": [
"ecs:DescribeSnapshots",
"ecs:CreateSnapshot",
"ecs:DeleteSnapshot",
"ecs:DescribeDisks",
"ecs:CreateDisk",
"ecs:Addtags",
"oss:PutObject",
"oss:GetObject",
"oss:DeleteObject",
"oss:GetBucket",
"oss:ListObjects"
],
"Resource": [
"*"
],
"Effect": "Allow"
}
]
}最后生成AK，记录AK信息并在velero安装部署时使用。
2.4 自建Kubernetes集群部署Velero
编辑credentials-velero文件， 内容为上一步骤中生成的AK信息：
ALIBABA_CLOUD_ACCESS_KEY_ID=<access_key_id>
ALIBABA_CLOUD_ACCESS_KEY_SECRET=<access_key_secret>使用以下命令部署velero：
velero install --provider alibabacloud --image registry.cn-hangzhou.aliyuncs.com/haoshuwei24/velero:v1.2.0 --bucket ls-velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config region=cn-hangzhou --use-restic --plugins registry.cn-hangzhou.aliyuncs.com/acs/velero-plugin-alibabacloud:v1.2 --wait可以查看pod的运行状态：
kubectl -n velero get po
NAME READY STATUS RESTARTS AGE
restic-fqwsc 1/1 Running 0 41s
restic-kfzqt 1/1 Running 0 41s
restic-klxhc 1/1 Running 0 41s
restic-ql2kr 1/1 Running 0 41s
restic-qrsrn 1/1 Running 0 41s
restic-srjmm 1/1 Running 0 41s
velero-67b975f5cb-68nj4 1/1 Running 0 41s3 自建Kubernetes集群中备份wordpress应用
若只需要备份wordpress应用而不备份pv数据， 则使用以下命令备份：
$ velero backup create wordpress-backup-without-pv --include-namespaces wordpress
Backup request "wordpress-backup-without-pv" submitted successfully.
Run `velero backup describe wordpress-backup-without-pv` or `velero backup logs wordpress-backup-without-pv` for more details.
$ velero backup get
NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR
wordpress-backup-without-pv Completed 2019-12-12 14:08:24 +0800 CST 29d default <none>本文着重演示带pv数据的wordpress应用备份：
# 首先需要为挂载pv数据卷的pod添加annotation， 例如wordpress应用运行了2个pod， 分别为wordpress-7cf5849f47-mbvx4 mysql-74dddbdcc8-h2tls， wordpress-7cf5849f47-mbvx4
# 挂载的volume名为mysql-persistent-storage， mysql-74dddbdcc8-h2tls挂载的volume名为wordpress-persistent-storage, 则添加annotation的命令为
$ kubectl -n wordpress annotate pod/wordpress-7cf5849f47-mbvx4 backup.velero.io/backup-volumes=wordpress-persistent-storage
pod/wordpress-7cf5849f47-mbvx4 annotated
$ kubectl -n wordpress annotate pod/mysql-74dddbdcc8-h2tls backup.velero.io/backup-volumes=mysql-persistent-storage
pod/mysql-74dddbdcc8-h2tls annotated
# 备份wordpress
$ velero backup create wordpress-backup-with-pv --include-namespaces wordpress
Backup request "wordpress-backup-with-pv" submitted successfully.
Run `velero backup describe wordpress-backup-with-pv` or `velero backup logs wordpress-backup-with-pv` for more details.
$ velero backup get
NAME STATUS CREATED EXPIRES STORAGE LOCATION SELECTOR
wordpress-backup-with-pv Completed 2019-12-12 14:23:40 +0800 CST 29d default <none>
wordpress-backup-without-pv Completed 2019-12-12 14:08:24 +0800 CST 29d default <none>查看OSS Bucket可以看到备份的文件。
4 恢复应用wordpress到ACK集群
4.1 创建StorageClass
wordpress应用使用nfs类型持久化数据卷，PV/PVC使用的StorageClass名称为nfs，相应的，在ACK中我们也需要创建一个相同名字的StorageClass， 但StorageClass后端使用了什么存储介质我们可以根据业务需求来定义，比如本示例中我们就使用了SSD云盘块存储，而非必须使用Nas共享存储：（本示例使用ACK集群使用CSI plugin， 参考https://help.aliyun.com/document_detail/134859.html）
$ cat nfs.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs
provisioner: diskplugin.csi.alibabacloud.com
parameters:
type: cloud_ssd
reclaimPolicy: Retain
$ kubectl apply -f nfs.yaml
storageclass.storage.k8s.io/nfs created4.2 恢复wordpress应用
使用velero恢复wordpress应用到ACK， 完成wordpress从自建Kubernetes集群到ACK的迁移。
$ velero restore create --from-backup wordpress-backup-with-pv
$ velero restore get
NAME BACKUP STATUS WARNINGS ERRORS CREATED SELECTOR
wordpress-backup-with-pv-20191212152745 wordpress-backup-with-pv InProgress 0 0 2019-12-12 15:27:45 +0800 CST <none>此时查看wordpress应用运行情况，可能会有镜像拉取失败的问题：
$ kubectl -n wordpress get po
NAME READY STATUS RESTARTS AGE
mysql-669b4666cd-trsnz 0/1 ErrImagePull 0 19m
mysql-74dddbdcc8-h2tls 0/1 Init:0/1 0 19m
wordpress-7cf5849f47-mbvx4 0/1 Init:0/1 0 19m
wordpress-bb5d74d95-xcjxw 0/1 ErrImagePull 0 19m我们需要编辑deployment把image字段替换成2.1中迁移后的镜像地址：
# edit 编辑deployment并修改image url
$ kubectl -n wordpress edit deployment mysql
$ kubectl -n wordpress edit deployment wordpress再次查看wordpress应用运行情况：
$ kubectl -n wordpress get po
NAME READY STATUS RESTARTS AGE
mysql-678b5d8499-vckfd 1/1 Running 0 100s
wordpress-8566f5f7d8-7shk6 1/1 Running 0 3m18s测试环境重新绑定hosts后，访问置wordpress应用 http://wordpress.myk8s.paas.com:31570
4. 其他参考：wordpress示例应用的部署
wordpress示例应用分wordpress和mysql两个组件， 分别绑定两个不同的nfs volume用于应用数据的持久化存储，最后通过NodePort暴露服务。部署yaml文件内容如下：
# 1. 创建nfs storageclass
$ cat nfs-sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs
provisioner: helm.default/nfs
reclaimPolicy: Delete
$ kubectl apply -f nfs-sc.yaml
# 2. 创建mysql password的secret, echo -n "mysql" |base64
$ cat secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: mysql
type: Opaque
data:
password: bXlzcWw=
$ kubectl apply -f secret.yaml
# 3. 创建mysql的pvc deployment service
$ cat mysql.yaml
apiVersion: v1
kind: Service
metadata:
name: mysql
labels:
app: mysql
spec:
type: ClusterIP
ports:
- port: 3306
selector:
app: mysql
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-volumeclaim
annotations:
volume.beta.kubernetes.io/storage-class: "nfs"
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql
labels:
app: mysql
spec:
replicas: 1
selector:
matchLabels:
app: mysql
template:
metadata:
labels:
app: mysql
spec:
securityContext:
runAsUser: 999
runAsGroup: 999
fsGroup: 999
containers:
- image: registry.api.paas.com:5000/admin/mysql:8
name: mysql
args:
- "--default-authentication-plugin=mysql_native_password"
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql
key: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-volumeclaim
$ kubectl apply -f mysql.yaml
# 4. 创建wordpress的pvc deployment service
$ cat wordpress.yaml
apiVersion: v1
kind: Service
metadata:
labels:
app: wordpress
name: wordpress
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
nodePort: 31570
selector:
app: wordpress
type: NodePort
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wordpress-volumeclaim
annotations:
volume.beta.kubernetes.io/storage-class: "nfs"
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress
labels:
app: wordpress
spec:
replicas: 1
selector:
matchLabels:
app: wordpress
template:
metadata:
labels:
app: wordpress
spec:
containers:
- image: registry.api.paas.com:5000/admin/wordpress
name: wordpress
env:
- name: WORDPRESS_DB_HOST
value: mysql:3306
- name: WORDPRESS_DB_PASSWORD
valueFrom:
secretKeyRef:
name: mysql
key: password
ports:
- containerPort: 80
name: wordpress
volumeMounts:
- name: wordpress-persistent-storage
mountPath: /var/www/html
volumes:
- name: wordpress-persistent-storage
persistentVolumeClaim:
claimName: wordpress-volumeclaim
$ kubectl apply -f wordpress.yaml测试环境绑定hosts后，访问并安装配置wordpress应用 http://wordpress.myk8s.paas.com:31570