在K8S中部署gilab-runner
目录开始写yml之前的思考：
写一个gitlab-runner的yaml文件进行在K8S上运行runner
方案：
1、daemonset 每个node上运行一个runner
缺点： 如果集群中的node并不都是来做runner时 此路就不通了（注册了不用 也是可以的）
优点：yaml文件简单，操作步骤少
2、deployment
缺点：操作步骤较多，需要对node进行打标签以及选择标签的方式来选择机器进行部署runner
优点：灵活安排那些node进行运行runner，解决“集群中并不是所有node都是来进行运行runner的”情况
这两种方式都是需要进行配合configmap以及serct进行关键词以及配置文件的传入
借鉴博主思路：
1、configmap 来存储runner注册时的url地址以及限制pod的资源
2、configmap资源类型来定义一个脚本 “用于注册、运行和取消注册”
3、sercet来存储runner注册的token
4、rbac的一个授权
5、StatefulSet来运行runner并使用以上定义的configmap以及sercet等资源
最终成品
apiVersion: v1
kind: Namespace
metadata:
name: gitlab-ci
---
apiVersion: v1
kind: Secret
metadata:
name: gitlab-ci-token
namespace: gitlab-ci
labels:
app: gitlab-ci-runner
data:
GITLAB_CI_TOKEN: WmF0Y3JNbUxUbmNncnpKem5CX1UK
---
apiVersion: v1
data:
run.sh: |
#!/bin/bash
unregister() {
kill %1
echo "Unregistering runner ${RUNNER_NAME} ..."
/usr/bin/gitlab-ci-multi-runner unregister -t "$(/usr/bin/gitlab-ci-multi-runner list 2>&1 | tail -n1 | awk '{print $4}' | cut -d'=' -f2)" -n ${RUNNER_NAME}
exit $?
}
trap 'unregister' EXIT HUP INT QUIT PIPE TERM
echo "Registering runner ${RUNNER_NAME} ..."
/usr/bin/gitlab-ci-multi-runner register -r ${GITLAB_CI_TOKEN}
sed -i 's/^concurrent.*/concurrent = '"${RUNNER_REQUEST_CONCURRENCY}"'/' /etc/gitlab-runner/config.toml
echo "Starting runner ${RUNNER_NAME} ..."
/usr/bin/gitlab-ci-multi-runner run -n ${RUNNER_NAME} &
wait
kind: ConfigMap
metadata:
labels:
app: gitlab-ci-runner
name: gitlab-ci-runner-scripts
namespace: gitlab-ci
---
apiVersion: v1
data:
KUBERNETES_NODE_TOLERATIONS: "node-role.kubernetes.io/master:NoSchedule"
KUBERNETES_IMAGE: "python"
RUNNER_TAG_LIST: "training"
REGISTER_NON_INTERACTIVE: "true"
REGISTER_LOCKED: "false"
METRICS_SERVER: "0.0.0.0:9100"
CI_SERVER_URL: "http://git.enflame.cn/"
RUNNER_REQUEST_CONCURRENCY: "4"
RUNNER_EXECUTOR: "kubernetes"
KUBERNETES_NAMESPACE: "gitlab-ci"
KUBERNETES_PRIVILEGED: "true"
KUBERNETES_CPU_LIMIT: "1"
KUBERNETES_CPU_REQUEST: "500m"
KUBERNETES_MEMORY_LIMIT: "1Gi"
KUBERNETES_SERVICE_CPU_LIMIT: "1"
KUBERNETES_SERVICE_MEMORY_LIMIT: "1Gi"
KUBERNETES_HELPER_CPU_LIMIT: "500m"
KUBERNETES_HELPER_MEMORY_LIMIT: "100Mi"
KUBERNETES_PULL_POLICY: "if-not-present"
KUBERNETES_TERMINATIONGRACEPERIODSECONDS: "10"
KUBERNETES_POLL_INTERVAL: "5"
KUBERNETES_POLL_TIMEOUT: "360"
kind: ConfigMap
metadata:
labels:
app: gitlab-ci-runner
name: gitlab-ci-runner-cm
namespace: gitlab-ci
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: gitlab-ci
namespace: gitlab-ci
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: gitlab-ci
namespace: gitlab-ci
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: gitlab-ci
namespace: gitlab-ci
subjects:
- kind: ServiceAccount
name: gitlab-ci
namespace: gitlab-ci
roleRef:
kind: Role
name: gitlab-ci
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: gitlab-ci-runner
namespace: gitlab-ci
labels:
app: gitlab-ci-runner
spec:
updateStrategy:
type: RollingUpdate
replicas: 1
selector:
matchLabels:
app: gitlab-ci-runner
serviceName: gitlab-ci-runner
template:
metadata:
labels:
app: gitlab-ci-runner
spec:
volumes:
- name: gitlab-ci-runner-scripts
projected:
sources:
- configMap:
name: gitlab-ci-runner-scripts
items:
- key: run.sh
path: run.sh
mode: 0755
serviceAccountName: gitlab-ci
containers:
- image: gitlab/gitlab-runner:alpine-v14.2.0
name: gitlab-ci-runner
command:
- /scripts/run.sh
envFrom:
- configMapRef:
name: gitlab-ci-runner-cm
- secretRef:
name: gitlab-ci-token
env:
- name: RUNNER_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
ports:
- containerPort: 9100
name: http-metrics
protocol: TCP
volumeMounts:
- name: gitlab-ci-runner-scripts
mountPath: "/scripts"
readOnly: true
restartPolicy: Always
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
注释版-StatefulSet(此版本只是参考作用)
#定义namespace
apiVersion: v1 #K8S的api接口版本
kind: Namespace #定义命名空间
metadata: #属性
name: kube-ops #命名空间的名字
---
#定义注册的token
apiVersion: v1 #K8S的api接口版本
kind: Secret #资源类型
metadata: #属性
name: gitlab-ci-token #secret的名字
namespace: kube-ops #选择命名空间
labels: #给资源打上标签
app: gitlab-ci-runner #标签内容
data: #定义键值对数据 存储runner注册时的token
GITLAB_CI_TOKEN: WURnUG14aUFBZFE0dFRrZ21RSkgK
---
#脚本用来注册和取消注册的命令，会在runner中进行执行
apiVersion: v1 #K8S的api接口版本
kind: ConfigMap #资源类型
data: #定义configmap中的内容 键值对方式定义
run.sh: |
#!/bin/bash
unregister() {
kill %1 #杀死后台运行程序为1的job
echo "Unregistering runner ${RUNNER_NAME} ..."
/usr/bin/gitlab-ci-multi-runner unregister -t "$(/usr/bin/gitlab-ci-multi-runner list 2>&1 | tail -n1 | awk '{print $4}' | cut -d'=' -f2)" -n ${RUNNER_NAME} #根据runner的token注销注册runner
exit $?
}
trap 'unregister' EXIT HUP INT QUIT PIPE TERM #当接收到 关闭的信号后 进行取消注册runner 好处就是当pod删除时 gitlab上自动删除此runner，不会导致任务分配过来而不执行的情况。
echo "Registering runner ${RUNNER_NAME} ..."
/usr/bin/gitlab-ci-multi-runner register -r ${GITLAB_CI_TOKEN}
sed -i 's/^concurrent.*/concurrent = '"${RUNNER_REQUEST_CONCURRENCY}"'/' /etc/gitlab-runner/config.toml
echo "Starting runner ${RUNNER_NAME} ..."
/usr/bin/gitlab-ci-multi-runner run -n ${RUNNER_NAME} & #进行注册runner
wait
metadata: #属性
labels: #标签
app: gitlab-ci-runner #标签具体内容
name: gitlab-ci-runner-scripts #configmap的名字
namespace: kube-ops #选择命名空间
---
#定义runner运行时的环境变量(这些变量runner部分是借助/usr/bin/gitlab-ci-multi-runner进行使用的，查看有哪些变量 /usr/bin/gitlab-ci-multi-runner register --help)
apiVersion: v1
kind: ConfigMap
data:
REGISTER_NON_INTERACTIVE: "true" #非交互式注册
REGISTER_LOCKED: "false" #寄存器锁定
METRICS_SERVER: "0.0.0.0:9100" #度量服务器
CI_SERVER_URL: "http://10.0.0.6/" #gitlab的url 注册地址
RUNNER_REQUEST_CONCURRENCY: "4" #请求并发数量
RUNNER_EXECUTOR: "kubernetes" #runner的执行器类型
KUBERNETES_NAMESPACE: "kube-ops" #指定命名空间
KUBERNETES_PRIVILEGED: "true" #有特权的
KUBERNETES_CPU_LIMIT: "1" #限制CPU资源为 1
KUBERNETES_CPU_REQUEST: "500m" #要求500m
KUBERNETES_MEMORY_LIMIT: "1Gi" #内存限制为1G
KUBERNETES_SERVICE_CPU_LIMIT: "1" #限制CPU为1
KUBERNETES_SERVICE_MEMORY_LIMIT: "1Gi" #内存为1G
KUBERNETES_HELPER_CPU_LIMIT: "500m" #辅助程序CPU限制
KUBERNETES_HELPER_MEMORY_LIMIT: "100Mi" #
KUBERNETES_PULL_POLICY: "if-not-present"
KUBERNETES_TERMINATIONGRACEPERIODSECONDS: "10"
KUBERNETES_POLL_INTERVAL: "5"
KUBERNETES_POLL_TIMEOUT: "360"
metadata:
labels:
app: gitlab-ci-runner
name: gitlab-ci-runner-cm
namespace: kube-ops
---
#进行rbac授权
apiVersion: v1
kind: ServiceAccount
metadata:
name: gitlab-ci
namespace: kube-ops
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: gitlab-ci
namespace: kube-ops
rules:
- apiGroups: [""]
resources: ["*"]
verbs: ["*"]
---
apiVersion: v1
kind: RoleBinding
metadata:
name: gitlab-ci
namespace: kube-ops
subjects:
- kind: ServiceAccount
name: gitlab-ci
namespace: kube-ops
roleRef:
kind: Role
name: gitlab-ci
apiGroup: rbac.authorization.k8s.io
---
#运行runner
apiVersion: apps/v1 #api版本
kind: StatefulSet #资源类型
metadata: 元数据
name: gitlab-ci-runner 指定资源的名称
namespace: kube-ops 指定名称空间
labels: 标签
app: gitlab-ci-runner 标签具体内容
spec: 属性
updateStrategy: #指定更新策略
type: RollingUpdate
replicas: 1 指定副本数
selector: 指定选择
matchLabels:
app: gitlab-ci-runner
serviceName: gitlab-ci-runner
template:
metadata:
labels:
app: gitlab-ci-runner
spec: 进行数据卷挂在
volumes:
- name: gitlab-ci-runner-scripts #使用脚本
projected:
sources:
- configMap:
name: gitlab-ci-runner-scripts #引用configmap中定义的那个脚本
items: #引用key
- key: run.sh
path: run.sh
mode: 0755
serviceAccountName: gitlab-ci #指定serviceaccount 进行绑定
containers: #容器信息
- image: gitlab/gitlab-runner:alpine-v12.10.1 #指定容器运行时的镜像
name: gitlab-ci-runner #容器名字
command: #运行的命令
- /scripts/run.sh
envFrom:
- configMapRef:
name: gitlab-ci-runner-cm #引入环境变量
- secretRef:
name: gitlab-ci-token #引入token
env: #环境变量
- name: RUNNER_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name #runner-name的名称 pod的
ports: #容器端口
- containerPort: 9100
name: http-metrics
protocol: TCP
volumeMounts:
- name: gitlab-ci-runner-scripts
mountPath: "/scripts"
readOnly: true #只读的方式挂载
restartPolicy: Always
daemonset
#定义runner工作的namespace
apiVersion: v1
kind: Namespace
metadata:
name: kube-ops
---
#定义runner注册时用到的token
apiVersion: v1
kind: Secret
metadata:
name: gitlab-ci-token
namespace: kube-ops
labels:
app: gitlab-ci-runner
data:
GITLAB_CI_TOKEN: WURnUG14aUFBZFE0dFRrZ21RSkgK
---
#定义一个脚本（暂时不明白博主为何这么操作）
apiVersion: v1
data:
run.sh: |
#!/bin/bash
unregister() {
kill %1
echo "Unregistering runner ${RUNNER_NAME} ..."
/usr/bin/gitlab-ci-multi-runner unregister -t "$(/usr/bin/gitlab-ci-multi-runner list 2>&1 | tail -n1 | awk '{print $4}' | cut -d'=' -f2)" -n ${RUNNER_NAME}
exit $?
}
trap 'unregister' EXIT HUP INT QUIT PIPE TERM
echo "Registering runner ${RUNNER_NAME} ..."
/usr/bin/gitlab-ci-multi-runner register -r ${GITLAB_CI_TOKEN}
sed -i 's/^concurrent.*/concurrent = '"${RUNNER_REQUEST_CONCURRENCY}"'/' /home/gitlab-runner/.gitlab-runner/config.toml
echo "Starting runner ${RUNNER_NAME} ..."
/usr/bin/gitlab-ci-multi-runner run -n ${RUNNER_NAME} &
wait
kind: ConfigMap
metadata:
labels:
app: gitlab-ci-runner
name: gitlab-ci-runner-scripts
namespace: kube-ops
---
#定义runner运行时的环境变量
apiVersion: v1
data:
REGISTER_NON_INTERACTIVE: "true"
REGISTER_LOCKED: "false"
METRICS_SERVER: "0.0.0.0:9100"
CI_SERVER_URL: "http://10.0.0.6/"
RUNNER_REQUEST_CONCURRENCY: "4"
RUNNER_EXECUTOR: "kubernetes"
KUBERNETES_NAMESPACE: "kube-ops"
KUBERNETES_PRIVILEGED: "true"
KUBERNETES_CPU_LIMIT: "1"
KUBERNETES_CPU_REQUEST: "500m"
KUBERNETES_MEMORY_LIMIT: "1Gi"
KUBERNETES_SERVICE_CPU_LIMIT: "1"
KUBERNETES_SERVICE_MEMORY_LIMIT: "1Gi"
KUBERNETES_HELPER_CPU_LIMIT: "500m"
KUBERNETES_HELPER_MEMORY_LIMIT: "100Mi"
KUBERNETES_PULL_POLICY: "if-not-present"
KUBERNETES_TERMINATIONGRACEPERIODSECONDS: "10"
KUBERNETES_POLL_INTERVAL: "5"
KUBERNETES_POLL_TIMEOUT: "360"
kind: ConfigMap
metadata:
labels:
app: gitlab-ci-runner
name: gitlab-ci-runner-cm
namespace: kube-ops
---
#定义RBAC授权
apiVersion: v1
kind: ServiceAccount
metadata:
name: gitlab-ci
namespace: kube-ops
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: gitlab-ci
namespace: kube-ops
rules:
- apiGroups: [""]
resources: ["*"]
verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: gitlab-ci
namespace: kube-ops
subjects:
- kind: ServiceAccount
name: gitlab-ci
namespace: kube-ops
roleRef:
kind: Role
name: gitlab-ci
apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: gitlab-ci-runner
namespace: kube-ops
labels:
app: gitlab-ci-runner
spec:
updateStrategy:
type: RollingUpdate
selector:
matchLabels:
app: gitlab-ci-runner
template:
metadata:
labels:
app: gitlab-ci-runner
spec:
volumes:
- name: gitlab-ci-runner-scripts
projected:
sources:
- configMap:
name: gitlab-ci-runner-scripts
items:
- key: run.sh
path: run.sh
mode: 0755
serviceAccountName: gitlab-ci
containers:
- image: gitlab/gitlab-runner:alpine-v12.10.1
name: gitlab-ci-runner
command:
- /scripts/run.sh
envFrom:
- configMapRef:
name: gitlab-ci-runner-cm
- secretRef:
name: gitlab-ci-token
env:
- name: RUNNER_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
ports:
- containerPort: 9100
name: http-metrics
protocol: TCP
volumeMounts:
- name: gitlab-ci-runner-scripts
mountPath: "/scripts"
readOnly: true
restartPolicy: Always
遇到的问题
问题一:脚本中注册runner的是什么方式进行注册的？
问题描述：当我尝试以手动的方式执行脚本的内容时，发现脚本中用到了runner的token 并不是gitlab-server的注册runner时的token从而产生了疑惑：难道注册gitlab-runner还能用自己的token（gitlab-runner的token）
思考：应该不是使用token的方式 因为命令行方式获取得到的token和gitlab-ci获取的token值是不一样的
解决：
此yaml中 使用runner的token进行的注销操作
注册runner是根据Secret中的token进行注册的（也就是gitlab-server中的token）
提问问题时没搞明白脚本的具体含义，脚本中是 借助gitlab-runner的token来进行注销runner的一个操作 而注册的操作还是使用gitlab-server提供的token
问题二：configmap中的变量从哪里可以看到？
在configmap中gitlab-ci-runner-cm的其他变量怎么没看到使用？
这些变量是借助/usr/bin/gitlab-ci-multi-runner进行使用的，自带的内置变量参数
查看有哪些变量 /usr/bin/gitlab-ci-multi-runner register --help
问题四：RUNNER_NAME这个变量是自带的吗？
根据K8S中的pod自带获取名称方式获取的
metadata.name
问题五：kill %1 是什么意思？
kill %1这条命令表示杀死一个后台程序，这个后台程序的“工作号码（jobnumber）”是1号。job命令的作用就是将放在后台运行的程序（& nohup等方式运行的进程）给显示出来，kill %1 则是将后台程序的job号码为1的给杀掉。
问题六：设置容忍度
问题描述：将K8S自动设置的系统污点误认为是人为设置的，导致找不到对应的key:value值，陷入不知道该如何设置容忍度的问题
正常情况下 kubectl describe node work1 |grep Taint 命令是可以看到人为设置的key:value的污点的。
我误认为 磁盘故障而导致K8S系统自动给node节点打的污点 是人为设置的污点，导致使用kubectl describe node work1|grep Taint 而没有找到key:value，从而陷入不知道该如何设置容忍的困境
解决： 如果是人为设置的污点，kubectl describe node work1 其实是可以看到设置的污点的key:value的
work1因为磁盘的故障而自动打污点
Warning Evicted 1s kubelet The node had condition: [DiskPressure].
问题七：gitlab-runner创建的容器需要设置容忍度
问题描述：gitlab-runner在接收到任务时 自动创建执行具体内容的容器 无法容忍污点 一直处于pending的状态
因为在yml文件中可以设置gitlab-runner的容忍，但是不清楚gitlab-runner运行后自动创建的容器是怎么设置污点容忍的。
解决：
须知：①gitlab-runner创建的容器 是从configmap中定义的环境变量中来获取信息的；②configmap中设置的gitlab-runner的环境变量获取方式为:/usr/bin/gitlab-ci-multi-runner register --help|grep kubernetes-node-tolerations
root@c75cacba4598:/# /usr/bin/gitlab-ci-multi-runner register --help|grep kubernetes-node-tolerations
Runtime platform arch=amd64 os=linux pid=50 revision=8925d9a0 version=14.1.0
--kubernetes-node-tolerations value A toml table/json object of key=value:effect. Value and effect are expected to be strings. When set, pods will tolerate the given taints. Only one toleration is supported through environment variable configuration. (default: "{}") [$KUBERNETES_NODE_TOLERATIONS]
具体解决方法：在yml中的configmap中设置 gitlab-runner创建容器时的特性
KUBERNETES_NODE_TOLERATIONS: "node-role.kubernetes.io/master:NoSchedule"
补充gitlab-ci-multi-runner命令解释
NAME:
gitlab-ci-multi-runner - a GitLab Runner
USAGE:
gitlab-ci-multi-runner [global options] command [command options] [arguments...]
VERSION:
14.1.0 (8925d9a0)
AUTHOR:
GitLab Inc. <support@gitlab.com>
COMMANDS:
exec 在本地执行构建
list 列出所有配置的运行程序
run 运行多运行程序服务
register 登记新的runner
install 安装服务
uninstall 卸载服务
start 开始服务
stop 停止服务
restart 重启服务
status 获取服务状态
run-single 启动单线程runner
unregister 取消注册特定的运行程序
verify 验证所有已注册的runner
artifacts-downloader 下载并提取构建工件（内部）
artifacts-uploader 创建和上载构建工件（内部）
cache-archiver 创建和上载缓存工件（内部）
cache-extractor 下载并提取缓存工件（内部）
cache-init 已更改缓存路径的权限（内部）
health-check 检查特定地址的运行状况
read-logs 从kubernetes executor（内部）使用的文件读取作业日志
help, h Shows a list of commands or help for one command
GLOBAL OPTIONS:
--cpuprofile value write cpu profile to file [$CPU_PROFILE]
--debug debug mode [$DEBUG]
--log-format value Choose log format (options: runner, text, json) [$LOG_FORMAT]
--log-level value, -l value Log level (options: debug, info, warn, error, fatal, panic) [$LOG_LEVEL]
--help, -h show help
--version, -v print the version