一. 前言
趁假期空闲,把以前的学习笔记整理下,梳理下知识,本系列会有三篇,一篇基础,两篇自定义开发&部署。
1.1 CRD定义
Custom Resources Definition,即在Kubernetes 中添加一个和 Pod、service 类似的、新的 API 资源类型,用于统一部署/编排多个内置K8S资源(pod,service等),熟练掌握 CRD 是成为 Kubernetes 高级玩家的必备技能。
1.2 为什么需要CRD?
helm也可以做到统一部署/编排deployment,service,ingress,但它缺乏对资源的全生命周期的监控,CRD通过apiserver接口,在etcd中注册一种新的资源类型,此后就可以创建对应的资源对象&并监控它们的状态&执行相关动作,比如,你可以定义一个MYSQL的 CRD完成MYSQL集群项目的全部pod&service创建&监控功能。
1.3 CRD实现方法
CRD的实现工具&框架有kubebuilder & operator sdk,后者在向前者融合,建议使用kubebuilder。
1.4 网络相关资源
这篇理论讲的非常好:https://www.cnblogs.com/alisystemsoftware/p/11580202.html
这篇理论兼示例(场景比较实用,虽然有点复杂^_^):https://blog.upweto.top/gitbooks/kubebuilder/%E8%AE%A4%E8%AF%86Kubebuilder.html
官网资料:https://book.kubebuilder.io/
https://cloudnative.to/kubebuilder/(中文)
https://book-v1.book.kubebuilder.io/(老版本)
网络上通俗易懂,适合初学者的例子非常少,就像官网所说:
…太多的教程一开始都是以一些非常复杂的设置开始,或者是实现一些玩具应用,让你了解了基本的知识,然后又在更复杂的东西上停滞不前…
希望本系列能避免官网提到的问题,达到简单,实用的目标。
二. 必备知识
CRD开发涉及的知识点还是比较多的,最好具备以下知识:
2.1 golang基础及部分高级内容,熟悉常用基础包如cobra,flag,log,channel,context,exception,interface,reflect…
2.2 docker基础;
2.3 k8s基础;
三. 开发前准备
3.1 k8s集群环境搭建
参见另一篇博文https://blog.csdn.net/dustzhu/article/details/108527676或其它网络资料。
3.2 开发环境搭建
3.2.1 安装gcc
yum -y install gcc
3.2.2 go语言安装包
官网:https://golang.google.cn/dl/
tar -C /usr/local -xzf go1.xx.xx.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
export GO111MODULE=on
3.2.3 kubebuilder
os=$(go env GOOS)
arch=$(go env GOARCH)
curl -L https://go.kubebuilder.io/dl/2.3.1/${os}/${arch} | tar -xz -C /tmp/
mv /tmp/kubebuilder_2.3.1_${os}_${arch} /usr/local/kubebuilder
export PATH=$PATH:/usr/local/kubebuilder/bin
3.2.4 设定国内代理模块仓库
export GOPROXY=https://goproxy.cn/
3.2.5 安装kustomize
curl -s "https://raw.githubusercontent.com/\
k
kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash #或者把shell下载下来手动执行
四. CRD开发&部署步骤
4.1 项目初始化
[root@k8s01 home]# mkdir mydemo
[root@k8s01 mydemo]# go mod init mydemo
go: creating new go.mod: module mydemo
[root@k8s01 mydemo]# kubebuilder init --domain ips.com.cn
Writing scaffold for you to edit…
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.5.0
Update go.mod:
$ go mod tidy
Running make:
$ make
which: no controller-gen in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/go/bin:/usr/local/kubebuilder/bin)
go: creating new go.mod: module tmp
go: found sigs.k8s.io/controller-tools/cmd/controller-gen in sigs.k8s.io/controller-tools v0.2.5
/root/go/bin/controller-gen object:headerFile=“hack/boilerplate.go.txt” paths="./…"
go fmt ./…
go vet ./…
go build -o bin/manager main.go
Next: define a resource with:
$ kubebuilder create api
4.2 创建API
[root@k8s01 mydemo]# kubebuilder create api --group mygroup --version v1 --kind Mykind
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing scaffold for you to edit…
api/v1/mykind_types.go
controllers/mykind_controller.go
Running make:
$ make
which: no controller-gen in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/go/bin:/usr/local/kubebuilder/bin)
go: creating new go.mod: module tmp
go: found sigs.k8s.io/controller-tools/cmd/controller-gen in sigs.k8s.io/controller-tools v0.2.5
/root/go/bin/controller-gen object:headerFile=“hack/boilerplate.go.txt” paths="./…"
go fmt ./…
go vet ./…
go build -o bin/manager main.go
[root@k8s01 mydemo]#
[root@k8s01 mydemo]# vi main.go #optional,change 8080 to 8081,如果没有端口冲突,请跳过!
4.3 修改产生的API代码
这是CRD开发的重点,主要涉及控制器&TYPE的golang代码,本文仅是基础流程介绍,不做任何代码修改。
4.4 安装CRD
这步的主要作用是把CRD对象安装到ETCD数据库中,主要原理是根据api/v1/mykind_type.go的MykindSpec数据结构创建base下面的yaml文件,并据此创建CRD。
[root@k8s01 mydemo]# make install
which: no controller-gen in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/go/bin:/usr/local/kubebuilder/bin)
go: creating new go.mod: module tmp
go: found sigs.k8s.io/controller-tools/cmd/controller-gen in sigs.k8s.io/controller-tools v0.2.5
/root/go/bin/controller-gen “crd:trivialVersions=true” rbac:roleName=manager-role webhook paths="./…" output:crd:artifacts:config=config/crd/bases
kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/mykinds.mygroup.ips.com.cn created
[root@k8s01 mydemo]# make run #optional,适用于集群本地测试运行;如果部署到集群,请跳过!
…
查看安装的CRD对象
[root@k8s01 mydemo]# kubectl get crd
NAME CREATED AT
mykinds.mygroup.ips.com.cn 2020-11-09T05:43:15Z
[root@k8s01 mydemo]# kubectl describe crd
Name: mykinds.mygroup.ips.com.cn
Namespace:
Labels: <none>
Annotations: controller-gen.kubebuilder.io/version: v0.2.5
API Version: apiextensions.k8s.io/v1
Kind: CustomResourceDefinition
Metadata:
......忽略
Spec:
Conversion:
Strategy: None
Group: mygroup.ips.com.cn
Names:
Kind: Mykind
List Kind: MykindList
Plural: mykinds
Singular: mykind
Preserve Unknown Fields: true
Scope: Namespaced
Versions:
Name: v1
Schema:
openAPIV3Schema:
Description: Mykind is the Schema for the mykinds API
Properties:
API Version:
Description: APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
Type: string
Kind:
Description: Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
Type: string
Metadata:
Type: object
Spec:
Description: MykindSpec defines the desired state of Mykind
Properties:
Foo:
Description: Foo is an example field of Mykind. Edit Mykind_types.go to remove/update
Type: string
Type: object
Status:
Description: MykindStatus defines the observed state of Mykind
Type: object
Type: object
Served: true
Storage: true
Status:
Accepted Names:
Kind: Mykind
List Kind: MykindList
Plural: mykinds
Singular: mykind
Conditions:
......忽略
Stored Versions:
v1
Events: <none>
4.5 创建CRD的实例
[root@k8s01 mydemo]# kubectl apply -f config/samples/
查看创建的CRD实例
[root@k8s01 mydemo]# kubectl get mykind -A
NAME SPACE NAME AGE
default mykind-sample 10s
[root@k8s01 mydemo]# kubectl describe mykind
Name: mykind-sample
Namespace: default
Labels: <none>
Annotations: API Version: mygroup.ips.com.cn/v1
Kind: Mykind
Metadata:
......忽略
Spec:
Foo: bar
Events: <none>
4.6 部署CRD
4.6.1 修改配置文件
由于默认生成Dockerfile&yaml存在一些问题,需要针对具体环境做相应修改。
[root@k8s01 mydemo]# vi Dockerfile
# Build the manager binary
FROM golang:1.13 as builder
WORKDIR /workspace
ENV GOPROXY https://goproxy.cn/
ENV GO111MODULE on
…
#FROM gcr.io/distroless/static:nonroot
FROM registry.cn-hangzhou.aliyuncs.com/byteforce/distroless:nonroot
…
[root@k8s01 mydemo]# vi config/default/manager_auth_proxy_patch.yaml
…
#image: gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
image: registry.cn-hangzhou.aliyuncs.com/hsc/kube-rbac-proxy
…
4.6.2 部署
[root@k8s01 mydemo]# make docker-build docker-push IMG=registry.ips.com.cn/demo/mycontroller:v0.1.0
which: no controller-gen in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/go/bin:/usr/local/kubebuilder/bin)
go: creating new go.mod: module tmp
go: found sigs.k8s.io/controller-tools/cmd/controller-gen in sigs.k8s.io/controller-tools v0.2.5
/root/go/bin/controller-gen object:headerFile=“hack/boilerplate.go.txt” paths="./…"
go fmt ./…
go vet ./…
/root/go/bin/controller-gen “crd:trivialVersions=true” rbac:roleName=manager-role webhook paths="./…" output:crd:artifacts:config=config/crd/bases
go test ./… -coverprofile cover.out
? mydemo [no test files]
? mydemo/api/v1 [no test files]
ok mydemo/controllers 4.649s coverage: 0.0% of statements
docker build . -t registry.ips.com.cn/demo/mycontroller:v0.1.0
Sending build context to Docker daemon 40.34MB
Step 1/16 : FROM golang:1.13 as builder
…
Successfully built f691c20c5f70
Successfully tagged registry.ips.com.cn/demo/mycontroller:v0.1.0
docker push registry.ips.com.cn/demo/mycontroller:v0.1.0
The push refers to repository [registry.ips.com.cn/demo/mycontroller]
Get https://registry.ips.com.cn/v2/: dial tcp: lookup registry.ips.com.cn on 114.114.114.114:53: no such host
make: *** [docker-push] Error 1
解决方法:如果无法连接镜像仓库,可以导出导入镜像到对应节点
[root@k8s01 mydemo]# cd …
[root@k8s01 home]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.ips.com.cn/demo/mycontroller v0.1.0 f691c20c5f70 41 seconds ago 43.9MB
…
[root@k8s01 home]# docker save -o mycon.tar registry.ips.com.cn/demo/mycontroller:v0.1.0
[root@k8s01 home]# scp mycon.tar k8s02:/home/
[root@k8s01 home]# scp mycon.tar k8s03:/home/
[root@k8s02 home]# docker load -i mycon.tar
[root@k8s03 home]# docker load -i mycon.tar
[root@k8s01 mydemo]# make deploy IMG=registry.ips.com.cn/demo/mycontroller:v0.1.0
which: no controller-gen in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/go/bin:/usr/local/kubebuilder/bin)
go: creating new go.mod: module tmp
go: found sigs.k8s.io/controller-tools/cmd/controller-gen in sigs.k8s.io/controller-tools v0.2.5
/root/go/bin/controller-gen “crd:trivialVersions=true” rbac:roleName=manager-role webhook paths="./…" output:crd:artifacts:config=config/crd/bases
cd config/manager && kustomize edit set image controller=registry.ips.com.cn/demo/mycontroller:v0.1.0
kustomize build config/default | kubectl apply -f -
namespace/mydemo-system created
customresourcedefinition.apiextensions.k8s.io/mykinds.mygroup.ips.com.cn configured
role.rbac.authorization.k8s.io/mydemo-leader-election-role created
clusterrole.rbac.authorization.k8s.io/mydemo-manager-role created
clusterrole.rbac.authorization.k8s.io/mydemo-proxy-role created
clusterrole.rbac.authorization.k8s.io/mydemo-metrics-reader created
rolebinding.rbac.authorization.k8s.io/mydemo-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/mydemo-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/mydemo-proxy-rolebinding created
service/mydemo-controller-manager-metrics-service created
deployment.apps/mydemo-controller-manager created
查看部署的对象
[root@k8s01 mydemo]# kubectl get deploy -A
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
mydemo-system mydemo-controller-manager 1/1 1 1 31s
[root@k8s01 mydemo]# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-flannel-ds-rp4t6 1/1 Running 0 20d
kube-system kube-flannel-ds-vqmqs 1/1 Running 0 20d
mydemo-system mydemo-controller-manager-59cf997ccd-rtpkg 2/2 Running 1 40s
[root@k8s01 mydemo]# kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 196.169.0.1 443/TCP 20d
mydemo-system mydemo-controller-manager-metrics-service ClusterIP 196.169.36.71 8443/TCP 48s
五. 总结
本文主要阐述kubebuilder默认产生的CRD的整个流程,给初学者一个整体概念,方便后续深入学习,最后用一张图总结CRD开发部署流程,如下: