一、简单介绍
GitLab-CI
- GitLab CI/CD是GitLab的一部分，支持从计划到部署具有出色的用户体验。CI/CD是开源GitLab社区版和专有GitLab企业版的一部分。可以根据需要添加任意数量的计算节点，每个构建可以拆分为多个作业，这些作业可以在多台计算机上并行运行。
- GitLab-CI轻量级，不需要复杂的安装手段。配置简单，与gitlab可直接适配。实时构建日志十分清晰，UI交互体验很好。使用 YAML 进行配置，任何人都可以很方便的使用。GitLabCI 有助于DevOps人员，例如敏捷开发中，开发与运维是同一个人，最便捷的开发方式。
- 在大多数情况，构建项目都会占用大量的系统资源，如果让gitlab本身来运行构建任务的话，显然Gitlab的性能会大幅度下降。GitLab-CI最大的作用就是管理各个项目的构建状态。因此，运行构建任务这种浪费资源的事情交给一个独立的Gitlab Runner来做就会好很多，更重要的是Gitlab Runner 可以安装到不同的机器上，甚至是我们本机，这样完全就不会影响Gitlab本身了。
- 从GitLab8.0开始，GitLab-CI就已经集成在GitLab中，我们只需要在项目中添加一个.gitlab-ci.yaml文件，然后运行一个Runner，即可进行持续集成。
GitLab-CI：集成、开源、无缝、可扩展、更快的结果、针对交付进行了优化：
GItLab Runner
- Gitlab Runner是一个开源项目，用于运行您的作业并将结果发送给gitlab。它与Gitlab CI结合使用，gitlab ci是Gitlab随附的用于协调作用的开源持续集成服务。
- Gitlab Runner是用Go编写的，可以作为一个二进制文件运行，不需要特定于语言的要求
- 它皆在GNU/Linux，MacOS和Windows操作系统上运行。另外注意：如果要使用Docker，Gitlab Runner要求Docker 至少是v1.13.0版本才可以。
Kubernetes Gitlab CICD 演示图：
二、基于Kubernetes Gitlab CICD 容器化部署记录
- Gitlab官方提供了Helm的方式在Kubernetes集群中来快速安装，但是在使用的过程中发现Helm提供的Chart包中有很多其他额外的配置。所以这里我采用K8S自定义的方式来安装。
- Gitlab主要涉及3个应用：Redis、Postgresql、Gitlab核心程序。
- 本案例中使用的Gitlab-ce镜像部署，镜像中的Gitlab版本是13.7.4。
- 本案例中使用NFS作为持久化存储方式。除此之外，还可以选择HostPath本地持久化存储、NAS云端持久化存储、Ceph分布式持久化存储等。
注意：本示例部署所涉及到的image镜像均导入到Harbor私有私仓（172.16.60.230） 。
1）使用NFS作为持久化存储
在NFS服务器端（172.16.60.238）创建Redis、Postgresql、Gitlab核心程序容器的持久化挂载目录
[root@k8s-harbor01 ~]# mkdir -p /data/storage/k8s/gitlab/{postgresql,redis,gitlab}
[root@k8s-harbor01 ~]# ll /data/storage/k8s/gitlab/
total 0
drwxr-xr-x 2 root root 6 Mar 25 14:03 gitlab
drwxr-xr-x 2 root root 6 Mar 25 14:03 postgresql
drwxr-xr-x 2 root root 6 Mar 25 14:03 redis
2）部署Gitlab
可以先创建一个命名空间
[root@k8s-master01 gitlab]# kubectl create ns kube-ops [root@k8s-master01 gitlab]# kubectl get ns|grep kube-ops kube-ops Active 7d18h
配置三个核心程序的容器化部署的yaml文件
[root@k8s-master01 gitlab]# pwd /opt/k8s/k8s_project/gitlab [root@k8s-master01 gitlab]# ll total 12 -rw-r--r-- 1 root root 1629 Mar 25 14:05 gitlab-postgresql.yaml -rw-r--r-- 1 root root 1207 Mar 25 14:05 gitlab-redis.yaml -rw-r--r-- 1 root root 2691 Mar 25 14:05 gitlab.yaml
gitlab-postgresql.yaml 文件内容：
[root@k8s-master01 gitlab]# cat gitlab-postgresql.yaml apiVersion: apps/v1 kind: Deployment metadata: name: postgresql namespace: kube-ops labels: name: postgresql spec: replicas: 1 selector: matchLabels: name: postgresql template: metadata: name: postgresql labels: name: postgresql spec: containers: - name: postgresql image: 172.16.60.230/gitlab/postgresql:v1 imagePullPolicy: IfNotPresent env: - name: DB_USER value: gitlab - name: DB_PASS value: passw0rd - name: DB_NAME value: gitlab_production - name: DB_EXTENSION value: pg_trgm ports: - name: postgres containerPort: 5432 volumeMounts: - mountPath: /var/lib/postgresql name: data livenessProbe: exec: command: - pg_isready - -h - localhost - -U - postgres initialDelaySeconds: 30 timeoutSeconds: 5 readinessProbe: exec: command: - pg_isready - -h - localhost - -U - postgres initialDelaySeconds: 5 timeoutSeconds: 1 volumes: - name: data nfs: server: 172.16.60.238 path: /data/storage/k8s/gitlab/postgresql readOnly: false --- apiVersion: v1 kind: Service metadata: name: postgresql namespace: kube-ops labels: name: postgresql spec: ports: - name: postgres port: 5432 targetPort: postgres selector: name: postgresql
gitlab-redis.yaml 文件内容：
[root@k8s-master01 gitlab]# cat gitlab-redis.yaml apiVersion: apps/v1 kind: Deployment metadata: name: redis namespace: kube-ops labels: name: redis spec: replicas: 1 selector: matchLabels: name: redis template: metadata: name: redis labels: name: redis spec: containers: - name: redis image: 172.16.60.230/gitlab/redis:latest imagePullPolicy: IfNotPresent ports: - name: redis containerPort: 6379 volumeMounts: - mountPath: /var/lib/redis name: data livenessProbe: exec: command: - redis-cli - ping initialDelaySeconds: 30 timeoutSeconds: 5 readinessProbe: exec: command: - redis-cli - ping initialDelaySeconds: 5 timeoutSeconds: 1 volumes: - name: data nfs: server: 172.16.60.238 path: /data/storage/k8s/gitlab/redis readOnly: false --- apiVersion: v1 kind: Service metadata: name: redis namespace: kube-ops labels: name: redis spec: ports: - name: redis port: 6379 targetPort: redis selector: name: redis
gitlab.yaml 文件内容：
今天的极致任务： 把所有的面试题收集完成 全力补到Python [root@k8s-master01 gitlab]# cat gitlab.yaml apiVersion: apps/v1 kind: Deployment metadata: name: gitlab namespace: kube-ops labels: name: gitlab spec: replicas: 1 selector: matchLabels: name: gitlab template: metadata: name: gitlab labels: name: gitlab spec: containers: - name: gitlab image: 172.16.60.230/gitlab/gitlab-ce:latest imagePullPolicy: IfNotPresent env: - name: TZ value: Asia/Shanghai - name: GITLAB_TIMEZONE value: Beijing - name: GITLAB_SECRETS_DB_KEY_BASE value: long-and-random-alpha-numeric-string - name: GITLAB_SECRETS_SECRET_KEY_BASE value: long-and-random-alpha-numeric-string - name: GITLAB_SECRETS_OTP_KEY_BASE value: long-and-random-alpha-numeric-string - name: GITLAB_ROOT_PASSWORD value: admin321 - name: GITLAB_ROOT_EMAIL value: 1025337607@qq.com - name: GITLAB_HOST value: 0.0.0.0:30004 - name: GITLAB_PORT value: "80" - name: GITLAB_SSH_PORT value: "22" - name: GITLAB_NOTIFY_ON_BROKEN_BUILDS value: "true" - name: GITLAB_NOTIFY_PUSHER value: "false" - name: GITLAB_BACKUP_SCHEDULE value: daily - name: GITLAB_BACKUP_TIME value: 01:00 - name: DB_TYPE value: postgres - name: DB_HOST value: postgresql - name: DB_PORT value: "5432" - name: DB_USER value: gitlab - name: DB_PASS value: passw0rd - name: DB_NAME value: gitlab_production - name: REDIS_HOST value: redis - name: REDIS_PORT value: "6379" ports: - name: http containerPort: 80 - name: ssh containerPort: 22 volumeMounts: - mountPath: /home/git/data name: data livenessProbe: httpGet: path: / port: 80 initialDelaySeconds: 180 timeoutSeconds: 5 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 timeoutSeconds: 1 volumes: - name: data nfs: server: 172.16.60.238 path: /data/storage/k8s/gitlab/gitlab readOnly: false --- apiVersion: v1 kind: Service metadata: name: gitlab namespace: kube-ops labels: name: gitlab spec: type: NodePort ports: - name: http port: 80 targetPort: http nodePort: 30004 - name: ssh port: 22 targetPort: ssh selector: name: gitlab
创建并启动gitlab相关容器进程：
[root@k8s-master01 gitlab]# ll total 12 -rw-r--r-- 1 root root 1629 Mar 25 14:05 gitlab-postgresql.yaml -rw-r--r-- 1 root root 1207 Mar 25 14:05 gitlab-redis.yaml -rw-r--r-- 1 root root 2691 Mar 25 14:05 gitlab.yaml [root@k8s-master01 gitlab]# kubectl apply -f . deployment.apps/postgresql created service/postgresql created deployment.apps/redis created service/redis created deployment.apps/gitlab created service/gitlab created
稍微等一会儿（由于程序启动顺序原因，pod可能会出现重启次数，不过最终都会启动成功），
查看pod状态：
[root@k8s-master01 gitlab]# kubectl get pods -n kube-ops NAME READY STATUS RESTARTS AGE gitlab-5b887894d5-ntxzj 1/1 Running 1 38m postgresql-57bf98cdf8-7mdh9 1/1 Running 1 38m redis-56769dc6b6-c4rnq 1/1 Running 0 38m
查看svc：
[root@k8s-master01 gitlab]# kubectl get svc -n kube-ops NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE gitlab NodePort 10.254.48.72 <none> 80:30004/TCP,22:32280/TCP 14m postgresql ClusterIP 10.254.88.39 <none> 5432/TCP 14m redis ClusterIP 10.254.198.0 <none> 6379/TCP
3）访问 Gitlab
- 这里采用NodePort的方式，通过 http://任意node节点ip:30004 地址访问Gitlab
- Gitlab登录用户名：root，密码：admin321
- Gitlab登录密码可以在yaml文件里修改
这里容器化部署后的Gitlab版本是13.7.4
4）创建演示项目
接下来顺便创建一个项目，用于演示：
接下来在服务器上git clone，进行代码提交演示：
git clone地址是 http://gitlab-5b887894d5-ntxzj/root/kevin-test.git
地址中的gitlab-5b887894d5-ntxzj是pod名称，在容器外部访问不了，需要修改为对应的nodeport地址，故git clone地址可以是：http://172.16.60.234:30004/root/kevin-test.git
在其中一个node节点上进行代码提交演示：
[root@k8s-node02 mnt]# mkdir /mnt/haha [root@k8s-node02 mnt]# cd /mnt/haha [root@k8s-node02 haha]# git config --global user.email "1025337607@qq.com" [root@k8s-node02 haha]# git config --global user.name "Administrator" [root@k8s-node02 haha]# [root@k8s-node02 haha]# git clone http://172.16.60.234:30004/root/kevin-test.git Cloning into 'kevin-test'... Username for 'http://172.16.60.234:30004': root #输入账号root Password for 'http://root@172.16.60.234:30004': #输入账号root的密码 warning: You appear to have cloned an empty repository. [root@k8s-node02 haha]# ls kevin-test [root@k8s-node02 haha]# cd kevin-test/ [root@k8s-node02 kevin-test]# ll total 0 [root@k8s-node02 kevin-test]# [root@k8s-node02 kevin-test]# touch test.md [root@k8s-node02 kevin-test]# echo "come on" > test.md [root@k8s-node02 kevin-test]# git add test.md [root@k8s-node02 kevin-test]# git commit -m "add test.md" [master (root-commit) 8ccda29] add test.md 1 file changed, 1 insertion(+) create mode 100644 test.md [root@k8s-node02 kevin-test]# git commit -m "add test.md" [master (root-commit) 8ccda29] add test.md 1 file changed, 1 insertion(+) create mode 100644 test.md [root@k8s-node02 kevin-test]# git push -u origin master Username for 'http://172.16.60.234:30004': root Password for 'http://root@172.16.60.234:30004': Counting objects: 3, done. Writing objects: 100% (3/3), 216 bytes | 0 bytes/s, done. Total 3 (delta 0), reused 0 (delta 0) To http://172.16.60.234:30004/root/kevin-test.git * [new branch] master -> master Branch master set up to track remote branch master from origin.
如何解决 "每次输入用户名和密码" 的问题？
在代码目录.git/config文件内[remote "origin"]的url的gitlab域名前添加gitlab注册时的"用户名:密码@"
[root@k8s-node02 kevin-test]# cat .git/config [core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true [remote "origin"] url = http://root:admin321@172.16.60.234:30004/root/kevin-test.git fetch = +refs/heads/*:refs/remotes/origin/* [branch "master"] remote = origin merge = refs/heads/master
接着再次尝试提交内容，就不需要手动输入用户名和密码了：
[root@k8s-node02 kevin-test]# git pull Already up-to-date. [root@k8s-node02 kevin-test]# echo "this is gitlab test" > test.md [root@k8s-node02 kevin-test]# git add test.md [root@k8s-node02 kevin-test]# git commit -m "modified test.md" [master f2fbb27] modified test.md 1 file changed, 1 insertion(+), 1 deletion(-) [root@k8s-node02 kevin-test]# git push -u origin master Counting objects: 12, done. Delta compression using up to 4 threads. Compressing objects: 100% (4/4), done. Writing objects: 100% (10/10), 845 bytes | 0 bytes/s, done. Total 10 (delta 1), reused 0 (delta 0) To http://root:admin321@172.16.60.234:30004/root/kevin-test.git fe40316..f2fbb27 master -> master Branch master set up to track remote branch master from origin.
代码上传后，gitlab上展示效果如下：
进入该项目下，左侧栏CICD里有三种方式：Pipelines、Jobs、Schedules
5）Gitlab Runner 安装和注册
gitlab runner支持多种方式安装，我这里就采取在k8s中安装。
官方文档地址: https://docs.gitlab.com/runner/install/
打开gitlab，如下图所示，左边代表runner状态，右边是配置runner信息。
注意右边栏的token信息，后面注册runner的时候会用到：
接下来进行配置gitlab runner资源清单 (runner-configmap.yaml)
[root@k8s-master01 gitlab]# cat runner-configmap.yaml apiVersion: v1 data: REGISTER_NON_INTERACTIVE: "true" REGISTER_LOCKED: "false" METRICS_SERVER: "0.0.0.0:9100" CI_SERVER_URL: "http://gitlab.kube-ops.svc.cluster.local/ci" RUNNER_REQUEST_CONCURRENCY: "4" RUNNER_EXECUTOR: "kubernetes" KUBERNETES_NAMESPACE: "kube-ops" KUBERNETES_PRIVILEGED: "true" KUBERNETES_CPU_LIMIT: "1" KUBERNETES_CPU_REQUEST: "500m" KUBERNETES_MEMORY_LIMIT: "1Gi" KUBERNETES_SERVICE_CPU_LIMIT: "1" KUBERNETES_SERVICE_MEMORY_LIMIT: "1Gi" KUBERNETES_HELPER_CPU_LIMIT: "500m" KUBERNETES_HELPER_MEMORY_LIMIT: "100Mi" KUBERNETES_PULL_POLICY: "if-not-present" KUBERNETES_TERMINATIONGRACEPERIODSECONDS: "10" KUBERNETES_POLL_INTERVAL: "5" KUBERNETES_POLL_TIMEOUT: "360" kind: ConfigMap metadata: labels: app: gitlab-ci-runner name: gitlab-ci-runner-cm namespace: kube-ops
需要注意：
- CI_SERVER_URL 这个地址是gitlab的地址，如果gitlab在宿主机直接写宿主机的ip即可，容器是格式为：svc名称.命名空间.svc.cluster.local (如果都按照我的文档来进行安装不需要修改别的配置了)。
- 如果定义的gitlab域名并不是通过外网DNS解析，而是通过/etc/hosts进行映射，那么我们需要在Runner的Pod中去添加对应的hosts，需要通过--pre-clone-script参数来指定一段脚本来添加hosts信息，也就是在ConfigMap中添加环境变量RUNNER_PRE_CLONE_SCRIPT的值：
本案例，这里gitlab地址我是使用node节点的ip+port方式。 如果使用gitlab域名方式，且不是外网DNS解析，比如域名地址是http://gitlab.kevin.com 则需要在上面的ConfigMap中添加环境变量RUNNER_PRE_CLONE_SCRIPT的值: RUNNER_PRE_CLONE_SCRIPT = "echo 'xx.xx.xxx.xx git.i4t.com' >> /etc/hosts" 其中xx.xx.xxx.xx 为node节点ip地址
另外记住：在ConfigMap添加新选项后，需要删除Gitlab ci Runner Pod
因为这里我是使用envFrom来注入上面的这些环境变量而不是直接使用env(envfrom 通过将环境变量放置到ConfigMaps或Secrets来帮助减小清单文件)
如果我们想添加其他选项，那么可以在等到后面的gitlab-ci-runner的Pod容器启动成功后，登录gitlab-ci-runner的pod容器内部运行gitlab-ci-multi-runner register --help 命令来查看所有可使用的选项，只需要为配置的标志添加env变量即可：
gitlab-runner@gitlab-ci-runner-0:/$ gitlab-ci-multi-runner register --help [...] --kubernetes-cpu-limit value The CPU allocation given to build containers (default: "1") [$KUBERNETES_CPU_LIMIT] --kubernetes-memory-limit value The amount of memory allocated to build containers (default: "4Gi") [$KUBERNETES_MEMORY_LIMIT] --kubernetes-service-cpu-limit value The CPU allocation given to build service containers (default: "1") [$KUBERNETES_SERVICE_CPU_LIMIT] --kubernetes-service-memory-limit value The amount of memory allocated to build service containers (default: "1Gi") [$KUBERNETES_SERVICE_MEMORY_LIMIT] --kubernetes-helper-cpu-limit value The CPU allocation given to build helper containers (default: "500m") [$KUBERNETES_HELPER_CPU_LIMIT] --kubernetes-helper-memory-limit value The amount of memory allocated to build helper containers (default: "3Gi") [$KUBERNETES_HELPER_MEMORY_LIMIT] --kubernetes-cpu-request value The CPU allocation requested for build containers [$KUBERNETES_CPU_REQUEST] ... --pre-clone-script value Runner-specific command script executed before code is pulled [$RUNNER_PRE_CLONE_SCRIPT] [...]
创建资源清单的configmap
[root@k8s-master01 gitlab]# kubectl apply -f runner-configmap.yaml configmap/gitlab-ci-runner-cm created [root@k8s-master01 gitlab]# kubectl get configmaps -n kube-ops NAME DATA AGE gitlab-ci-runner-cm 19 4s 可通过下面命令来查看此configmap内容： [root@k8s-master01 gitlab]# kubectl describe cm gitlab-ci-runner-cm -n kube-ops
此时，还需要配置一个用于注册、运行和取消gitlab ci runner的小脚本。只有当Pod正常通过K8S (TERM信号)的终止流程时，才会触发注销注册。如果强行终止Pod（SIGKILL信号），Runner将不会自己注销自身。必须手动完成对这种Runner的清理 (注意：只有如这里在k8s集群里安装GitLan Runner才这样操作，二进制安装非K8s上安装则不受这个影响)
[root@k8s-master01 gitlab]# cat runner-scripts-cm.yaml
apiVersion: v1
data:
run.sh: |
#!/bin/bash
unregister() {
kill %1
echo "Unregistering runner ${RUNNER_NAME} ..."
/usr/bin/gitlab-ci-multi-runner unregister -t "$(/usr/bin/gitlab-ci-multi-runner list 2>&1 | tail -n1 | awk '{print $4}' | cut -d'=' -f2)" -n ${RUNNER_NAME}
exit $?
}
trap 'unregister' EXIT HUP INT QUIT PIPE TERM
echo "Registering runner ${RUNNER_NAME} ..."
/usr/bin/gitlab-ci-multi-runner register -r ${GITLAB_CI_TOKEN}
sed -i 's/^concurrent.*/concurrent = '"${RUNNER_REQUEST_CONCURRENCY}"'/' /home/gitlab-runner/.gitlab-runner/config.toml
echo "Starting runner ${RUNNER_NAME} ..."
/usr/bin/gitlab-ci-multi-runner run -n ${RUNNER_NAME} &
wait
kind: ConfigMap
metadata:
labels:
app: gitlab-ci-runner
name: gitlab-ci-runner-scripts
namespace: kube-ops
创建此脚本的configmap
[root@k8s-master01 gitlab]# kubectl apply -f runner-scripts-cm.yaml configmap/gitlab-ci-runner-scripts created [root@k8s-master01 gitlab]# kubectl get configmaps -n kube-ops NAME DATA AGE gitlab-ci-runner-cm 19 4m12s gitlab-ci-runner-scripts 1 11s
接着需要创建一个GITLAB_CI_TOKEN，然后使用gitlab ci runner token来创建一个Kubernetes secret对象。需要提前对token进行base64转码：
[root@k8s-master01 gitlab]# echo xWgpgrP3rSXnxvZL9oRf|base64 -w0 eFdncGdyUDNyU1hueHZaTDlvUmYK
特意注意：这里的token就是我们gitlab runner上截图的地方,base64只有在k8s环境上需要！
登录Gitlab，Runner右边栏token信息 如下图：
使用上面的token创建一个Sercret对象 (gitlab-ci-token-secret.yaml)
[root@k8s-master01 gitlab]# cat gitlab-ci-token-secret.yaml apiVersion: v1 kind: Secret metadata: name: gitlab-ci-token namespace: kube-ops labels: app: gitlab-ci-runner data: GITLAB_CI_TOKEN: eFdncGdyUDNyU1hueHZaTDlvUmYK
创建这个Secret
[root@k8s-master01 gitlab]# kubectl apply -f gitlab-ci-token-secret.yaml secret/gitlab-ci-token created [root@k8s-master01 gitlab]# kubectl get secret -n kube-ops|grep gitlab gitlab-ci-token Opaque 1 19s
接下来创建真正运行Runner的控制器镜像，这里使用Statefulset，在开始运行的时候，尝试取消注册所有的同名Runner，当节点丢失时(即NodeLost事件)，这尤其有用，然后再尝试注册自己并开始运行。在正常停止Pod的时候，Runner将会运行unregister命令来尝试取消自己，所以gitlab就不能再使用这个Runner，这个则是通过kubernetes Pod生命周期中的hooks来完成的：
编译gitlab runner的pod部署的yaml文件内容：
[root@k8s-master01 gitlab]# cat runner-statefulset.yaml apiVersion: v1 kind: Service metadata: name: gitlab-ci-runner namespace: kube-ops labels: app: gitlab-ci-runner spec: ports: - port: 9100 targetPort: 9100 name: http-metrics clusterIP: None selector: app: gitlab-ci-runner --- apiVersion: apps/v1 kind: StatefulSet metadata: name: gitlab-ci-runner namespace: kube-ops labels: app: gitlab-ci-runner spec: updateStrategy: type: RollingUpdate replicas: 2 serviceName: gitlab-ci-runner selector: matchLabels: app: gitlab-ci-runner template: metadata: labels: app: gitlab-ci-runner spec: volumes: - name: gitlab-ci-runner-scripts projected: sources: - configMap: name: gitlab-ci-runner-scripts items: - key: run.sh path: run.sh mode: 0755 serviceAccountName: gitlab-ci securityContext: runAsNonRoot: true runAsUser: 999 supplementalGroups: [999] containers: - image: 172.16.60.230/gitlab/gitlab-runner:latest name: gitlab-ci-runner command: - /scripts/run.sh envFrom: - configMapRef: name: gitlab-ci-runner-cm - secretRef: name: gitlab-ci-token env: - name: RUNNER_NAME valueFrom: fieldRef: fieldPath: metadata.name ports: - containerPort: 9100 name: http-metrics protocol: TCP volumeMounts: - name: gitlab-ci-runner-scripts mountPath: "/scripts" readOnly: true restartPolicy: Always
上面我们命名了一个gitlab-ci的serviceAccount，这里要新建一个rbac文件 (runner-rbac.yaml)
[root@k8s-master01 gitlab]# cat runner-rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: gitlab-ci namespace: kube-ops --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: gitlab-ci namespace: kube-ops rules: - apiGroups: [""] resources: ["*"] verbs: ["*"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: gitlab-ci namespace: kube-ops subjects: - kind: ServiceAccount name: gitlab-ci namespace: kube-ops roleRef: kind: Role name: gitlab-ci apiGroup: rbac.authorization.k8s.io
创建gitlab runner
[root@k8s-master01 gitlab]# kubectl apply -f runner-rbac.yaml serviceaccount/gitlab-ci created role.rbac.authorization.k8s.io/gitlab-ci created rolebinding.rbac.authorization.k8s.io/gitlab-ci created [root@k8s-master01 gitlab]# kubectl apply -f runner-statefulset.yaml service/gitlab-ci-runner unchanged statefulset.apps/gitlab-ci-runner created
接下来我们检查我们创建的资源信息：
[root@k8s-master01 gitlab]# kubectl get pod,svc,cm -n kube-ops NAME READY STATUS RESTARTS AGE pod/gitlab-5b887894d5-ntxzj 1/1 Running 1 150m pod/gitlab-ci-runner-0 1/1 Running 0 12m pod/gitlab-ci-runner-1 1/1 Running 0 8m33s pod/postgresql-57bf98cdf8-7mdh9 1/1 Running 1 150m pod/redis-56769dc6b6-c4rnq 1/1 Running 0 150m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/gitlab NodePort 10.254.252.242 <none> 80:30004/TCP,22:31455/TCP 150m service/gitlab-ci-runner ClusterIP None <none> 9100/TCP 13m service/postgresql ClusterIP 10.254.173.69 <none> 5432/TCP 150m service/redis ClusterIP 10.254.11.213 <none> 6379/TCP 150m NAME DATA AGE configmap/gitlab-ci-runner-cm 19 36m configmap/gitlab-ci-runner-scripts 1 32m
此时，在登录Gitlab，查看Runner信息，发现就已经将这2个pod节点添加进来了：
这里我们也可以更改Runner的一些配置，比如添加tag标签等：
6）Gitlab Runner 配置使用
这里上传了一个测试项目gitlab-ci-k8s-demo-master，下面介绍：在gitlab项目里添加k8s集群（只要一个项目添加成功后，其他项目也都添加了）
另外注意：CI/CD的Auto DevOps里的"Default to Auto DevOps pipeline“ 如果有勾选，则需要去掉这个勾选，否则会对集成k8s产生影响。
接下来要配置一下kubectl，默认情况下kubectl需要使用证书才可以连接到k8s集群。在服务器上默认使用的是/root/.kube/config，gitlab 也有添加证书的位置，下面进行添加配置。
这里简单说明下：
- Kubernetes cluster name 集群名称，这个可以随便写
- API URL 这里实际上就是apiserver地址，通过kubectl cluster-info查看到
- CA Certificate、Service Token 可以通过/root/.kube/config 或下面方式得到
Kubernetes cluster 名称
这里顺便填写一个，比如：kevin-k8s-cluster
API URL地址
可以通过"kubectl cluster-info"命令获取
[root@k8s-master01 gitlab]# kubectl cluster-info Kubernetes master is running at https://127.0.0.1:8443 CoreDNS is running at https://127.0.0.1:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy Metrics-server is running at https://127.0.0.1:8443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
由于这里的k8s集群中的kube-apiserver高可用通过local nginx+haproxy实现的，所以需要将https://127.0.0.1:8443 改为其中的一个master地址，这里填写：https://172.16.60.231:8443
CA Certificate、Service Token 值
这里没有使用/root/.kube/config获取Service Token，而是采用下面的方式获取：
这里提醒一下：cluster-admin是内置的一个角色，拥有集群所有权限的一个角色
这里先创建一个命名空间 [root@k8s-master01 gitlab]# kubectl create ns gitlab namespace/gitlab created 因为在操作的时候会涉及rbac权限的问题，这里要创建一个rbac的文件。我们将集群的cluster的权限绑定到ServerAccount [root@k8s-master01 gitlab]# pwd /opt/k8s/k8s_project/gitlab [root@k8s-master01 gitlab]# cat gitlabdemo-sa.yaml apiVersion: v1 kind: ServiceAccount metadata: name: gitlab namespace: gitlab --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: gitlab subjects: - kind: ServiceAccount name: gitlab namespace: gitlab roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin 进行创建 [root@k8s-master01 gitlab]# kubectl apply -f gitlabdemo-sa.yaml serviceaccount/gitlab created clusterrolebinding.rbac.authorization.k8s.io/gitlab created 查看 [root@k8s-master01 gitlab]# kubectl get sa -n gitlab NAME SECRETS AGE default 1 2m55s gitlab 1 17s
上面创建的serviceaccount实际上就是一个secret，接下来我们进行获取Service Token和CA Certificate
[root@k8s-master01 gitlab]# kubectl get secret -n gitlab
NAME TYPE DATA AGE
default-token-v2r6h kubernetes.io/service-account-token 3 4m13s
gitlab-token-ddhs6 kubernetes.io/service-account-token 3 95s
[root@k8s-master01 gitlab]# kubectl get secrets gitlab-token-ddhs6 -n gitlab -o yaml
apiVersion: v1
data:
ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR5RENDQXJDZ0F3SUJBZ0lVTnI4WnAvRmxmc2xvUHJrRzdiaGNjbkYrYWVZd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2FURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVRTUE0R0ExVUVDeE1IYjNCemJuVnNiREVXTUJRR0ExVUVBeE1OCmEzVmlaWEp1WlhSbGN5MWpZVEFnRncweU1ERXhNVFV4TkRBek1EQmFHQTh5TVRJd01UQXlNakUwTURNd01Gb3cKYVRFTE1Ba0dBMVVFQmhNQ1EwNHhFREFPQmdOVkJBZ1RCMEpsYVVwcGJtY3hFREFPQmdOVkJBY1RCMEpsYVVwcApibWN4RERBS0JnTlZCQW9UQTJzNGN6RVFNQTRHQTFVRUN4TUhiM0J6Ym5Wc2JERVdNQlFHQTFVRUF4TU5hM1ZpClpYSnVaWFJsY3kxallUQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU5rUmFpQ2wKM2VDNmFRSVYvUkU1OHpnajhOQTJLcTFqMmZBdW5ZNU96Q2Uza0wyTWtPRDNGUXZhZndJeXlpanU4eGtVNlVRYwprYzZ6US92a3BBTFJ5NWFCRTBQeTdpdUJTdnJwMFZMK0htbUdQSzhKcXovUlZ0RjAwemxOK1A4by8rV0ZqeExFCmFCVGtMMkkxK2tyNEFubFZhT0dJaGl6WjBWQVhmR2JWNnRVeFpCMWVkWVkzZmdlbHo0ZDRBU3MvcnB5NEpxZFYKTGErZkRvN2Z5ZEZvQ0IwS3FKN1B1VWI3c1JvVU1WR1Jnd2o0amc3ckVwWEZ6bFJBUmN5RjZneUplcXY5OWhUdQpaUUlSRVJIK3BadzJJVVNZTzF4MzZ0UlVQRHlMN0FrVzhxNU1VYVBVRkJzQ0hGM0dYeWNzdjdUSllocFAyQk84CmVwa3NFN2kxTW1LeXIvc0NBd0VBQWFObU1HUXdEZ1lEVlIwUEFRSC9CQVFEQWdFR01CSUdBMVVkRXdFQi93UUkKTUFZQkFmOENBUUl3SFFZRFZSME9CQllFRk93YStVei9LR2hDZ2RNdGlxd01xSGxyNzUyK01COEdBMVVkSXdRWQpNQmFBRk93YStVei9LR2hDZ2RNdGlxd01xSGxyNzUyK01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQlh0SHJlCmNuangvNVJqYm0vbXhPNDJLWXZQTmY3NXNxNitST3JTb2pBQmRmSXlTVDlTSEpDTkFqWElvTlJHYndZWWJNSC8KMHp5WEVObGJFcDlBd2VETGNER2ZvT2ZwMTJ0UzFuQ3FWZlhhazNLWWlRRWNVQk9QMzI3bVYxU3lrOEpYWGtXawpNRUxrRDRNQWovTFlYbmh1dWdua1QxMk9VOXF1d1FURUZROTZUckNEeFpyNktzMXI4d1R1WHlqRlJudlAyZk9OCmpFQytXS3dXTm05Nk5GMHFIWWlSUHV4V2FCWWM3MXN2SWpsNW9vWWFjRnVpTlpGNEpBeFRPcVRUb0poK3crTUsKLzcrd0x4WmJHTHB3cndpYXhUZW5ZRTdpeGFyVnV0QkRTbWxlbmR3aU9FUldXV2NwWDZjdWpTSnlMTHFsRTBYNApXZHFnMmsrd3BVaEcrbGtYCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
namespace: Z2l0bGFi
token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklrWXliM0JIV0dVME9USXhjVE51VTFnd04zWm5NM2RpVEZsVFNucHlibVJ1TWtKbmMxUkhVMXB1VjFFaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpuYVhSc1lXSWlMQ0pyZFdKbGNtNWxkR1Z6TG1sdkwzTmxjblpwWTJWaFkyTnZkVzUwTDNObFkzSmxkQzV1WVcxbElqb2laMmwwYkdGaUxYUnZhMlZ1TFdSa2FITTJJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpYSjJhV05sTFdGalkyOTFiblF1Ym1GdFpTSTZJbWRwZEd4aFlpSXNJbXQxWW1WeWJtVjBaWE11YVc4dmMyVnlkbWxqWldGalkyOTFiblF2YzJWeWRtbGpaUzFoWTJOdmRXNTBMblZwWkNJNkltSXlNV00yTW1RMExUVXpNV1F0TkdFMU55MDRNek5qTFRGbU9ETm1OakF3TkRGa1lpSXNJbk4xWWlJNkluTjVjM1JsYlRwelpYSjJhV05sWVdOamIzVnVkRHBuYVhSc1lXSTZaMmwwYkdGaUluMC5vU1ZsWGpSMk5aU3hGWWNHX2RLclJKOXljdWY4dFdkUDI1cHNSRkRoSGlsYm0zV2ZkZ09EMkdGMjAwS1JsV1hySG9nNzNybmZVOUFtakNvTWFwQ0JSVmZGdFY4TGNxOTZGeEtRcEJYbGhrU2JWaW53V2JXUXpUWEtOYmN3M3pLU2JJWkV2X09kbF96d1U1NzNYYXBWeFVORERqMEszZF9qR2xxbGg3ZGpDN1ZGb1pjRHNjR3g5UjJMZW5TVmdXY1QxX1pSZ1VVYkVHSFREQVI5Q1VwNU5sQzNJUVFBSTljQmVJcmtLeUo2blVDRi1jNGpQZlViSkVqaThMMDdtSzRNRnhNY3k4QmUwbTdHSWZGcWNIcGExc1V5VjBJb2lRSmF3Q2pwUUhvSFR6TmJkVzF3ZC1oMjIxcXJuRVRwanRrV1pMMnRZTlhfNUNTOXA3OXRZR2hJMnc=
kind: Secret
metadata:
annotations:
kubernetes.io/service-account.name: gitlab
kubernetes.io/service-account.uid: b21c62d4-531d-4a57-833c-1f83f60041db
creationTimestamp: "2021-03-25T10:38:04Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:ca.crt: {}
f:namespace: {}
f:token: {}
f:metadata:
f:annotations:
.: {}
f:kubernetes.io/service-account.name: {}
f:kubernetes.io/service-account.uid: {}
f:type: {}
manager: kube-controller-manager
operation: Update
time: "2021-03-25T10:38:04Z"
name: gitlab-token-ddhs6
namespace: gitlab
resourceVersion: "31458293"
selfLink: /api/v1/namespaces/gitlab/secrets/gitlab-token-ddhs6
uid: b460030a-616c-4713-bd36-d3dea49d2ea5
type: kubernetes.io/service-account-token
上面结果中的ca.crt内容进行base64转码才能得到CA Certificate
需要通过命令"echo "xxxx"|base64 -d"使用base64进行转码
"xxxx"是ca.crt的内容
[root@k8s-master01 gitlab]# echo "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR5RENDQXJDZ0F3SUJBZ0lVTnI4WnAvRmxmc2xvUHJrRzdiaGNjbkYrYWVZd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2FURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVRTUE0R0ExVUVDeE1IYjNCemJuVnNiREVXTUJRR0ExVUVBeE1OCmEzVmlaWEp1WlhSbGN5MWpZVEFnRncweU1ERXhNVFV4TkRBek1EQmFHQTh5TVRJd01UQXlNakUwTURNd01Gb3cKYVRFTE1Ba0dBMVVFQmhNQ1EwNHhFREFPQmdOVkJBZ1RCMEpsYVVwcGJtY3hFREFPQmdOVkJBY1RCMEpsYVVwcApibWN4RERBS0JnTlZCQW9UQTJzNGN6RVFNQTRHQTFVRUN4TUhiM0J6Ym5Wc2JERVdNQlFHQTFVRUF4TU5hM1ZpClpYSnVaWFJsY3kxallUQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU5rUmFpQ2wKM2VDNmFRSVYvUkU1OHpnajhOQTJLcTFqMmZBdW5ZNU96Q2Uza0wyTWtPRDNGUXZhZndJeXlpanU4eGtVNlVRYwprYzZ6US92a3BBTFJ5NWFCRTBQeTdpdUJTdnJwMFZMK0htbUdQSzhKcXovUlZ0RjAwemxOK1A4by8rV0ZqeExFCmFCVGtMMkkxK2tyNEFubFZhT0dJaGl6WjBWQVhmR2JWNnRVeFpCMWVkWVkzZmdlbHo0ZDRBU3MvcnB5NEpxZFYKTGErZkRvN2Z5ZEZvQ0IwS3FKN1B1VWI3c1JvVU1WR1Jnd2o0amc3ckVwWEZ6bFJBUmN5RjZneUplcXY5OWhUdQpaUUlSRVJIK3BadzJJVVNZTzF4MzZ0UlVQRHlMN0FrVzhxNU1VYVBVRkJzQ0hGM0dYeWNzdjdUSllocFAyQk84CmVwa3NFN2kxTW1LeXIvc0NBd0VBQWFObU1HUXdEZ1lEVlIwUEFRSC9CQVFEQWdFR01CSUdBMVVkRXdFQi93UUkKTUFZQkFmOENBUUl3SFFZRFZSME9CQllFRk93YStVei9LR2hDZ2RNdGlxd01xSGxyNzUyK01COEdBMVVkSXdRWQpNQmFBRk93YStVei9LR2hDZ2RNdGlxd01xSGxyNzUyK01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQlh0SHJlCmNuangvNVJqYm0vbXhPNDJLWXZQTmY3NXNxNitST3JTb2pBQmRmSXlTVDlTSEpDTkFqWElvTlJHYndZWWJNSC8KMHp5WEVObGJFcDlBd2VETGNER2ZvT2ZwMTJ0UzFuQ3FWZlhhazNLWWlRRWNVQk9QMzI3bVYxU3lrOEpYWGtXawpNRUxrRDRNQWovTFlYbmh1dWdua1QxMk9VOXF1d1FURUZROTZUckNEeFpyNktzMXI4d1R1WHlqRlJudlAyZk9OCmpFQytXS3dXTm05Nk5GMHFIWWlSUHV4V2FCWWM3MXN2SWpsNW9vWWFjRnVpTlpGNEpBeFRPcVRUb0poK3crTUsKLzcrd0x4WmJHTHB3cndpYXhUZW5ZRTdpeGFyVnV0QkRTbWxlbmR3aU9FUldXV2NwWDZjdWpTSnlMTHFsRTBYNApXZHFnMmsrd3BVaEcrbGtYCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K" | base64 -d -----BEGIN CERTIFICATE----- MIIDyDCCArCgAwIBAgIUNr8Zp/FlfsloPrkG7bhccnF+aeYwDQYJKoZIhvcNAQEL BQAwaTELMAkGA1UEBhMCQ04xEDAOBgNVBAgTB0JlaUppbmcxEDAOBgNVBAcTB0Jl aUppbmcxDDAKBgNVBAoTA2s4czEQMA4GA1UECxMHb3BzbnVsbDEWMBQGA1UEAxMN a3ViZXJuZXRlcy1jYTAgFw0yMDExMTUxNDAzMDBaGA8yMTIwMTAyMjE0MDMwMFow aTELMAkGA1UEBhMCQ04xEDAOBgNVBAgTB0JlaUppbmcxEDAOBgNVBAcTB0JlaUpp bmcxDDAKBgNVBAoTA2s4czEQMA4GA1UECxMHb3BzbnVsbDEWMBQGA1UEAxMNa3Vi ZXJuZXRlcy1jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANkRaiCl 3eC6aQIV/RE58zgj8NA2Kq1j2fAunY5OzCe3kL2MkOD3FQvafwIyyiju8xkU6UQc kc6zQ/vkpALRy5aBE0Py7iuBSvrp0VL+HmmGPK8Jqz/RVtF00zlN+P8o/+WFjxLE aBTkL2I1+kr4AnlVaOGIhizZ0VAXfGbV6tUxZB1edYY3fgelz4d4ASs/rpy4JqdV La+fDo7fydFoCB0KqJ7PuUb7sRoUMVGRgwj4jg7rEpXFzlRARcyF6gyJeqv99hTu ZQIRERH+pZw2IUSYO1x36tRUPDyL7AkW8q5MUaPUFBsCHF3GXycsv7TJYhpP2BO8 epksE7i1MmKyr/sCAwEAAaNmMGQwDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQI MAYBAf8CAQIwHQYDVR0OBBYEFOwa+Uz/KGhCgdMtiqwMqHlr752+MB8GA1UdIwQY MBaAFOwa+Uz/KGhCgdMtiqwMqHlr752+MA0GCSqGSIb3DQEBCwUAA4IBAQBXtHre cnjx/5Rjbm/mxO42KY***f75sq6+ROrSojABdfIyST9SHJCNAjXIoNRGbwYYbMH/ 0zyXENlbEp9AweDLcDGfoOfp12tS1nCqVfXak3KYiQEcUBOP327mV1Syk8JXXkWk MELkD4MAj/LYXnhuugnkT12OU9quwQTEFQ96TrCDxZr6Ks1r8wTuXyjFRnvP2fON jEC+WKwWNm96NF0qHYiRPuxWaBYc71svIjl5ooYacFuiNZF4JAxTOqTToJh+w+MK /7+wLxZbGLpwrwiaxTenYE7ixarVutBDSmlendwiOERWWWcpX6cujSJyLLqlE0X4 Wdqg2k+wpUhG+lkX -----END CERTIFICATE-----
上面转码结果中的-----BEGIN CERTIFICATE-----和 -----END CERTIFICATE-----之间的内容就是CA Certificate
运行以下命令得到Service Token输出值：
[root@k8s-master01 gitlab]# kubectl -n gitlab describe secret gitlab-token-ddhs6 Name: gitlab-token-ddhs6 Namespace: gitlab Labels: <none> Annotations: kubernetes.io/service-account.name: gitlab kubernetes.io/service-account.uid: b21c62d4-531d-4a57-833c-1f83f60041db Type: kubernetes.io/service-account-token Data ==== ca.crt: 1371 bytes namespace: 6 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6IkYyb3BHWGU0OTIxcTNuU1gwN3ZnM3diTFlTSnpybmRuMkJnc1RHU1puV1EifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJnaXRsYWIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiZ2l0bGFiLXRva2VuLWRkaHM2Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImdpdGxhYiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImIyMWM2MmQ0LTUzMWQtNGE1Ny04MzNjLTFmODNmNjAwNDFkYiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpnaXRsYWI6Z2l0bGFiIn0.oSVlXjR2NZSxFYcG_dKrRJ9ycuf8tWdP25psRFDhHilbm3WfdgOD2GF200KRlWXrHog73rnfU9AmjCoMapCBRVfFtV8Lcq96FxKQpBXlhkSbVinwWbWQzTXKNbcw3zKSbIZEv_Odl_zwU573XapVxUNDDj0K3d_jGlqlh7djC7VFoZcDscGx9R2LenSVgWcT1_ZRgUUbEGHTDAR9CUp5NlC3IQQAI9cBeIrkKyJ6nUCF-c4jPfUbJEji8L07mK4MFxMcy8Be0m7GIfFqcHpa1sUyV0IoiQJawCjpQHoHTzNbdW1wd-h221qrnETpjtkWZL2tYNX_5CS9p79tYGhI2w
接着，就可以按照上面得出的几个信息，进行添加k8s集群的操作：
点击下面的 "Add Kubernetes cluster"按钮，就能成功添加已有的kubernetes集成了。
如果出现下面报错：
解决方法：
Settings => Network ，然后点击 Outbound requests 右边 的“expand”按钮，勾选 "Allow requests to the local network from web hooks and services"。