0. 前言
- 紧接上一篇,本篇文章我们尝试学习多节点部署 kubernetes 集群
- 并通过 haproxy+keepalived 实现 Master 节点的负载均衡
1. 实验环境
- 实验环境主要为 5 台虚拟机,IP 地址分别为:192.168.1.65、192.168.1.66、192.168.1.67、192.168.1.68、192.168.1.69
1.1 节点分配
-
LB 节点:
- lb1:192.168.1.65
- lb2:192.168.1.66
-
Master 节点:
- master1:192.168.1.67
- master2:192.168.1.68
- master3:192.168.1.69
-
Node 节点:
- node1:192.168.1.67
- node2:192.168.1.68
- node3:192.168.1.69
-
Etcd 节点:
- etcd01:192.168.1.67
- etcd02:192.168.1.68
- etcd03:192.168.1.69
- 为节约计算资源,kubernetes 集群中的 Master 节点、Node 节点和 Etcd 节点均各自部署在一个节点内
2. 部署流程
2.1 源码编译
- 安装 golang 环境
- kubernetes v1.18 要求使用的 golang 版本为 1.13
$ wget https://dl.google.com/go/go1.13.8.linux-amd64.tar.gz $ tar -zxvf go1.13.8.linux-amd64.tar.gz -C /usr/local/
- 添加如下环境变量至 ~/.bashrc 或者 ~/.zshrc
export GOROOT=/usr/local/go # GOPATH export GOPATH=$HOME/go # GOROOT bin export PATH=$PATH:$GOROOT/bin # GOPATH bin export PATH=$PATH:$GOPATH/bin
- 更新环境变量
$ source ~/.bashrc
- 从 github 上下载 kubernetes 最新源码
$ git clone https://github.com/kubernetes/kubernetes.git
- 编译形成二进制文件
$ make KUBE_BUILD_PLATFORMS=linux/amd64 +++ [0215 22:16:44] Building go targets for linux/amd64: ./vendor/k8s.io/code-generator/cmd/deepcopy-gen +++ [0215 22:16:52] Building go targets for linux/amd64: ./vendor/k8s.io/code-generator/cmd/defaulter-gen +++ [0215 22:17:00] Building go targets for linux/amd64: ./vendor/k8s.io/code-generator/cmd/conversion-gen +++ [0215 22:17:12] Building go targets for linux/amd64: ./vendor/k8s.io/kube-openapi/cmd/openapi-gen +++ [0215 22:17:25] Building go targets for linux/amd64: ./vendor/github.com/go-bindata/go-bindata/go-bindata +++ [0215 22:17:27] Building go targets for linux/amd64: cmd/kube-proxy cmd/kube-apiserver cmd/kube-controller-manager cmd/kubelet cmd/kubeadm cmd/kube-scheduler vendor/k8s.io/apiextensions-apiserver cluster/gce/gci/mounter cmd/kubectl cmd/gendocs cmd/genkubedocs cmd/genman cmd/genyaml cmd/genswaggertypedocs cmd/linkcheck vendor/github.com/onsi/ginkgo/ginkgo test/e2e/e2e.test cluster/images/conformance/go-runner cmd/kubemark vendor/github.com/onsi/ginkgo/ginkgo
- KUBE_BUILD_PLATFORMS 指定了编译生成的二进制文件的目标平台,包括 darwin/amd64、linux/amd64 和 windows/amd64 等
- 执行 make cross 会生成所有平台的二进制文件
- 本地编译然后上传至服务器
- 生成的 _output 目录即为编译生成文件,核心二进制文件在 _output/local/bin/linux/amd64 中
$ pwd /root/Coding/kubernetes/_output/local/bin/linux/amd64 $ ls apiextensions-apiserver genman go-runner kube-scheduler kubemark e2e.test genswaggertypedocs kube-apiserver kubeadm linkcheck gendocs genyaml kube-controller-manager kubectl mounter genkubedocs ginkgo kube-proxy kubelet
- 其中 kube-apiserver、kube-scheduler、kube-controller-manager、kubectl、kube-proxy 和 kubelet 为安装需要的二进制文件
2.2 安装 docker
- 在 kubernetes 集群的三个虚拟机上安装 docker:192.168.1.67、192.168.1.68、192.168.1.69
- 具体安装细节参见 官方文档
2.3 下载安装脚本
- 后续安装部署的所有脚本已经上传至 github 仓库 中,感兴趣的朋友可以下载
- 在 master1、master2 和 master3 上创建工作目录 k8s 和脚本目录 k8s/scripts,复制仓库中的所有脚本,到工作目录中的脚本文件夹中
$ git clone https://github.com/wangao1236/k8s_cluster_deploy.git $ cd k8s_cluster_deploy/scripts $ chmod +x *.sh $ mkdir -p k8s/scripts $ cp k8s_cluster_deploy/scripts/* k8s/scripts
2.4 安装 cfssl
- 在 master1、master2 和 master3 上安装 cfssl
- 在左右 kubernetes 节点上安装 cfssl,执行 k8s/scripts/cfssl.sh 脚本,或者执行如下命令:
$ curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl $ curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson $ curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo $ chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo
- k8s/scripts/cfssl.sh 脚本内容如下:
$ cat k8s_cluster_deploy/scripts/cfssl.sh curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo
2.5 安装 etcd
- 在其中一台机器上(如 etcd01)创建目标文件夹
$ mkdir -p /opt/etcd/{cfg,bin,ssl}
- 下载 etcd 最新版安装包
$ wget https://github.com/etcd-io/etcd/releases/download/v3.3.18/etcd-v3.3.18-linux-amd64.tar.gz $ tar -zxvf etcd-v3.3.18-linux-amd64.tar.gz $ cp etcd-v3.3.18-linux-amd64/etcdctl etcd-v3.3.18-linux-amd64/etcd /opt/etcd/bin
- 创建文件夹 k8s/etcd-cert,其中 k8s 部署相关文件和脚本的存储根目录,etcd-cert 暂存 etcd https 的证书
$ mkdir -p k8s/etcd-cert
- 复制 etcd-cert.sh 脚本执行 etcd-cert 目录中
$ cp k8s/scripts/etcd-cert.sh k8s/etcd-cert
- 脚本内容如下:
cat > ca-config.json <<EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "www": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF cat > ca-csr.json <<EOF { "CN": "etcd CA", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing" } ] } EOF cfssl gencert -initca ca-csr.json | cfssljson -bare ca - #----------------------- cat > server-csr.json <<EOF { "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.1.65", "192.168.1.66", "192.168.1.67", "192.168.1.68", "192.168.1.69" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
- 注意修改 server-csr.json 部分的 hosts 内容为 127.0.0.1 和虚拟机集群的所有 IP 地址
- 执行脚本
$ ./etcd-cert.sh 2020/02/20 17:18:09 [INFO] generating a new CA key and certificate from CSR 2020/02/20 17:18:09 [INFO] generate received request 2020/02/20 17:18:09 [INFO] received CSR 2020/02/20 17:18:09 [INFO] generating key: rsa-2048 2020/02/20 17:18:09 [INFO] encoded CSR 2020/02/20 17:18:09 [INFO] signed certificate with serial number 712703952401219579947544408367305212876133158662 2020/02/20 17:18:09 [INFO] generate received request 2020/02/20 17:18:09 [INFO] received CSR 2020/02/20 17:18:09 [INFO] generating key: rsa-2048 2020/02/20 17:18:09 [INFO] encoded CSR 2020/02/20 17:18:09 [INFO] signed certificate with serial number 59975233056205858127163767550140095337822886214 2020/02/20 17:18:09 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); specifically, section 10.2.3 ("Information Requirements").
- 拷贝证书
$ cp *.pem /opt/etcd/ssl
- 执行 k8s/scripts/etcd.sh 脚本,第一个参数为 etcd 节点名称,第二个为当前启动节点的 IP 地址,第三个参数为 Etcd 集群的所有地址
$ ./k8s/scripts/etcd.sh etcd01 192.168.1.67 etcd01=https://192.168.1.67:2380,etcd02=https://192.168.1.68:2380,etcd03=https://192.168.1.69:2380
- k8s/scripts/etcd.sh 脚本内容如下:
#!/bin/bash # example: ./etcd.sh etcd01 192.168.1.10 etcd01=https://192.168.1.10:2380,etcd02=https://192.168.1.11:2380,etcd03=https://192.168.1.12:2380 ETCD_NAME=$1 ETCD_IP=$2 ETCD_CLUSTER=$3 systemctl stop etcd systemctl disable etcd WORK_DIR=/opt/etcd cat <<EOF >$WORK_DIR/cfg/etcd #[Member] ETCD_NAME="${ETCD_NAME}" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380" ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380" ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379" ETCD_INITIAL_CLUSTER="${ETCD_CLUSTER}" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" EOF cat <<EOF >/usr/lib/systemd/system/etcd.service [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=${WORK_DIR}/cfg/etcd ExecStart=${WORK_DIR}/bin/etcd \ --name=\${ETCD_NAME} \ --data-dir=\${ETCD_DATA_DIR} \ --listen-peer-urls=\${ETCD_LISTEN_PEER_URLS} \ --listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \ --advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS} \ --initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS} \ --initial-cluster=\${ETCD_INITIAL_CLUSTER} \ --initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN} \ --initial-cluster-state=new \ --cert-file=${WORK_DIR}/ssl/server.pem \ --key-file=${WORK_DIR}/ssl/server-key.pem \ --peer-cert-file=${WORK_DIR}/ssl/server.pem \ --peer-key-file=${WORK_DIR}/ssl/server-key.pem \ --trusted-ca-file=${WORK_DIR}/ssl/ca.pem \ --peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable etcd systemctl restart etcd
- 接下来将 etcd 的工作目录和 etcd.service 文件复制给 etcd02 和 etcd03
$ scp -r /opt/etcd/ root@192.168.1.68:/opt/ $ scp -r /opt/etcd/ root@192.168.1.69:/opt/ $ scp /usr/lib/systemd/system/etcd.service root@192.168.1.68:/usr/lib/systemd/system/ $ scp /usr/lib/systemd/system/etcd.service root@192.168.1.69:/usr/lib/systemd/system/
- 分别在 etcd02 和 etcd03 上修改配置文件:/opt/etcd/cfg/etcd
[root@192.168.1.68] $ vim /opt/etcd/cfg/etcd #[Member] ETCD_NAME="etcd02" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.1.68:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.1.68:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.68:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.68:2379" ETCD_INITIAL_CLUSTER="etcd01=https://192.168.1.67:2380,etcd02=https://192.168.1.68:2380,etcd03=https://192.168.1.69 ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" [root@192.168.1.69] $ vim /opt/etcd/cfg/etcd #[Member] ETCD_NAME="etcd03" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.1.69:2380" ETCD_LISTEN_CLIENT_URLS="https://192.168.1.69:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.69:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.69:2379" ETCD_INITIAL_CLUSTER="etcd01=https://192.168.1.67:2380,etcd02=https://192.168.1.68:2380,etcd03=https://192.168.1.69 ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"
- 分别在 etcd02 和 etcd03 上启动 etcd 服务
$ sudo systemctl enable etcd.service Created symlink /etc/systemd/system/multi-user.target.wants/etcd.service → /usr/lib/systemd/system/etcd.service. $ sudo systemctl start etcd.service
- 检查安装是否成功,执行如下命令:
$ sudo etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.1.67:2379,https://192.168.1.68:2379,https://192.168.1.69:2379" cluster-health member 3143a1397990e241 is healthy: got healthy result from https://192.168.1.68:2379 member 469e7b2757c25086 is healthy: got healthy result from https://192.168.1.67:2379 member 5b1e32d0ab5e3e1b is healthy: got healthy result from https://192.168.1.69:2379 cluster is healthy
2.6 部署 flannel
- 在 node1、node2、node3 三个节点上分别部署 flannel
- 写入分配的子网段到 etcd 中,供 flannel 使用:
$ /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://127.0.0.1:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}' { "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}
- 查看写入的信息
$ /opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://127.0.0.1:2379" get /coreos.com/network/config { "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}
- 下载 flannel 最新安装包
$ wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz $ tar -zxvf flannel-v0.11.0-linux-amd64.tar.gz $ mkdir -p /opt/kubernetes/{cfg,bin,ssl} $ mv mk-docker-opts.sh flanneld /opt/kubernetes/bin/
- 执行脚本 k8s/scripts/flannel.sh,第一个参数为 etcd 地址
$ ./k8s/scripts/flannel.sh https://192.168.1.67:2379,https://192.168.1.68:2379,https://192.168.1.69:2379
- 脚本内容如下:
$ cat ./k8s/scripts/flannel.sh #!/bin/bash ETCD_ENDPOINTS=${1:-"http://127.0.0.1:2379"} systemctl stop flanneld systemctl disable flanneld cat <<EOF >/opt/kubernetes/cfg/flanneld FLANNEL_OPTIONS="--etcd-endpoints=${ETCD_ENDPOINTS} \\ -etcd-cafile=/opt/etcd/ssl/ca.pem \\ -etcd-certfile=/opt/etcd/ssl/server.pem \\ -etcd-keyfile=/opt/etcd/ssl/server-key.pem" EOF cat <<EOF >/usr/lib/systemd/system/flanneld.service [Unit] Description=Flanneld overlay address etcd agent After=network-online.target network.target Before=docker.service [Service] Type=notify EnvironmentFile=/opt/kubernetes/cfg/flanneld ExecStart=/opt/kubernetes/bin/flanneld --ip-masq \$FLANNEL_OPTIONS ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker -f /run/flannel/subnet.env Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable flanneld systemctl restart flanneld
- 查看启动时指定的子网
$ cat /run/flannel/subnet.envFLANNEL_NETWORK=172.17.0.0/16 FLANNEL_SUBNET=172.17.89.1/24 FLANNEL_MTU=1450 FLANNEL_IPMASQ=true $ cat /run/flannel/docker DOCKER_OPT_BIP="--bip=172.17.89.1/24" DOCKER_OPT_IPMASQ="--ip-masq=false" DOCKER_OPT_MTU="--mtu=1450" DOCKER_OPTS=" --bip=172.17.89.1/24 --ip-masq=false --mtu=1450"
- 执行
vim /usr/lib/systemd/system/docker.service
修改 docker 配置
[Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com BindsTo=containerd.service After=network-online.target firewalld.service containerd.service Wants=network-online.target Requires=docker.socket [Service] Type=notify # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker EnvironmentFile=/run/flannel/docker ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H unix:///var/run/docker.soc #ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.soc ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always ......
- 重启 docker 服务
$ systemctl daemon-reload $ systemctl restart docker
- 查看 flannel 网络,docker0 位于 flannel 分配的子网中
$ ifconfig docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 172.17.89.1 netmask 255.255.255.0 broadcast 172.17.89.255 ether 02:42:fb:16:3b:12 txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255 inet6 fe80::a00:27ff:feaf:b59f prefixlen 64 scopeid 0x20<link> ether 08:00:27:af:b5:9f txqueuelen 1000 (Ethernet) RX packets 517 bytes 247169 (247.1 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 361 bytes 44217 (44.2 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.1.67 netmask 255.255.255.0 broadcast 192.168.1.255 inet6 fe80::a00:27ff:fe9f:cb5c prefixlen 64 scopeid 0x20<link> inet6 2409:8a10:2e24:d130:a00:27ff:fe9f:cb5c prefixlen 64 scopeid 0x0<global> ether 08:00:27:9f:cb:5c txqueuelen 1000 (Ethernet) RX packets 9244 bytes 2349434 (2.3 MB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 7420 bytes 1047863 (1.0 MB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450 inet 172.17.89.0 netmask 255.255.255.255 broadcast 0.0.0.0 inet6 fe80::60c3:ecff:fe34:9d6c prefixlen 64 scopeid 0x20<link> ether 62:c3:ec:34:9d:6c txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 6 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 3722 bytes 904859 (904.8 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 3722 bytes 904859 (904.8 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
- 创建容器,查看容器网络
[root@adf9fc37d171 /]# yum install -y net-tools [root@adf9fc37d171 /]# ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.17.89.2 netmask 255.255.255.0 broadcast 172.17.89.255 ether 02:42:ac:11:59:02 txqueuelen 0 (Ethernet) RX packets 1538 bytes 14149689 (13.4 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1383 bytes 81403 (79.4 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 loop txqueuelen 1000 (Local Loopback) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [root@adf9fc37d171 /]# ping 172.17.89.1 PING 172.17.89.1 (172.17.89.1) 56(84) bytes of data. 64 bytes from 172.17.89.1: icmp_seq=1 ttl=64 time=0.045 ms 64 bytes from 172.17.89.1: icmp_seq=2 ttl=64 time=0.045 ms 64 bytes from 172.17.89.1: icmp_seq=3 ttl=64 time=0.050 ms 64 bytes from 172.17.89.1: icmp_seq=4 ttl=64 time=0.052 ms 64 bytes from 172.17.89.1: icmp_seq=5 ttl=64 time=0.049 ms
- 测试可以 ping 通 docker0 网卡 证明 flannel 起到路由作用
2.7 安装 haproxy+keepalieved
- 在 lb1 和 lb2 上安装 haproxy 和 keepalived
- 执行如下命令:
$ sudo apt-get -y install haproxy keepalived
- 两台机器上分别修改 /etc/haproxy/haproxy.cfg 文件,追加如下内容:
listen admin_stats bind 0.0.0.0:10080 mode http log 127.0.0.1 local0 err stats refresh 30s stats uri /status stats realm welcome login\ Haproxy stats auth admin:123456 stats hide-version stats admin if TRUE listen kube-master bind 0.0.0.0:8443 mode tcp option tcplog balance source server 192.168.1.67 192.168.1.67:6443 check inter 2000 fall 2 rise 2 weight 1 server 192.168.1.68 192.168.1.68:6443 check inter 2000 fall 2 rise 2 weight 1 server 192.168.1.69 192.168.1.69:6443 check inter 2000 fall 2 rise 2 weight 1
- 定义监听端口为 8443 防止和 kube-apiserver 的 6443 端口重复
- haproxy 的访问页面为 http://192.168.1.66:10080/status 或者 http://192.168.1.67:10080/status,需要输入定义的用户名密码:admin/123456
- 完整配置文件如下:
$ cat /etc/haproxy/haproxy.cfg global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners stats timeout 30s user haproxy group haproxy daemon # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # Default ciphers to use on SSL-enabled listening sockets. # For more information, see ciphers(1SSL). This list is from: # https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ # An alternative list with additional directives can be obtained from # https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS ssl-default-bind-options no-sslv3 defaults log global mode http option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 errorfile 400 /etc/haproxy/errors/400.http errorfile 403 /etc/haproxy/errors/403.http errorfile 408 /etc/haproxy/errors/408.http errorfile 500 /etc/haproxy/errors/500.http errorfile 502 /etc/haproxy/errors/502.http errorfile 503 /etc/haproxy/errors/503.http errorfile 504 /etc/haproxy/errors/504.http listen admin_stats bind 0.0.0.0:10080 mode http log 127.0.0.1 local0 err stats refresh 30s stats uri /status stats realm welcome login\ Haproxy stats auth admin:123456 stats hide-version stats admin if TRUE listen kube-master bind 0.0.0.0:8443 mode tcp option tcplog balance source server 192.168.1.67 192.168.1.67:6443 check inter 2000 fall 2 rise 2 weight 1 server 192.168.1.68 192.168.1.68:6443 check inter 2000 fall 2 rise 2 weight 1 server 192.168.1.69 192.168.1.69:6443 check inter 2000 fall 2 rise 2 weight 1View Code
- 修改配置文件后,执行如下命令,重启服务:
$ sudo systemctl enable haproxy $ sudo systemctl daemon-reload $ sudo systemctl restart haproxy.service
- 访问页面如下:
- 选择 lb1 作为 keepalived 的主节点,lb2 为备份节点
- 添加 lb1 的配置文件如下:
$ sudo vim /etc/keepalived/keepalived.conf global_defs { router_id lb-master } vrrp_script check-haproxy { script "killall -0 haproxy" interval 5 weight -30 } vrrp_instance VI-kube-master { state MASTER priority 120 dont_track_primary interface enp0s3 virtual_router_id 68 advert_int 3 track_script { check-haproxy } virtual_ipaddress { 192.168.1.99 } }View Code
- 添加 lb2 的配置文件如下:
$ sudo vim /etc/keepalived/keepalived.conf global_defs { router_id lb-backup } vrrp_script check-haproxy { script "killall -0 haproxy" interval 5 weight -30 } vrrp_instance VI-kube-master { state BACKUP priority 110 dont_track_primary interface enp0s3 virtual_router_id 68 advert_int 3 track_script { check-haproxy } virtual_ipaddress { 192.168.1.99 } }View Code
- 注意上述 vrrp_instance 部分的 interface 字段为 lb1 和 lb2 上对应的网卡名称
- 重启服务:
$ sudo systemctl enable keepalived $ sudo systemctl daemon-reload $ sudo systemctl restart keepalived.service
2.8 脚本生成各个组件的证书
- 在192.168.1.67、192.168.1.68 和 192.168.1.69 上分别执行如下命令:
$ sudo mkdir -p /opt/kubernetes/{ssl,cfg,bin,log}
- k8s/scripts/k8s-cert.sh 定义了各个组件 https 服务的 csr,并生成证书
- 修改 k8s/scripts/k8s-cert.sh 中 kube-apiserver-csr.json 部分的 hosts 字段为 127.0.0.1、虚拟 IP 地址、master 的 IP 地址和 kube-apiserver 定义的 --service-cluster-ip-range 参数指定的 IP 地址段(10.254.0.0/24)的第一个IP地址
cat > kube-apiserver-csr.json <<EOF { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.1.99", // 虚拟机 IP "192.168.1.67", // master1 IP "192.168.1.68", // master2 IP "192.168.1.69", // master3 IP "10.254.0.1", // kube-apiserver 定义的 --service-cluster-ip-range 参数指定的 IP 地址段(10.254.0.0/24)的第一个IP地址 "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF
- 修改 k8s/scripts/k8s-cert.sh 中 kube-controller-manager-csr.json 部分的 hosts 字段为 127.0.0.1 和 master 的 IP 地址
cat > kube-controller-manager-csr.json <<EOF { "CN": "system:kube-controller-manager", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "192.168.1.67", "192.168.1.68", "192.168.1.69" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:kube-controller-manager", "OU": "System" } ] } EOF
- CN 为 system:kube-controller-manager
- O 为 system:kube-controller-manager
- kube-apiserver预定义的 RBAC使用的 ClusterRoleBindings system:kube-controller-manager 将用户 system:kube-controller-manager 与 ClusterRole system:kube-controller-manager 绑定
- 修改 k8s/scripts/k8s-cert.sh 中 kube-scheduler-csr.json 部分的 hosts 字段为 127.0.0.1 和 master 的 IP 地址
cat > kube-scheduler-csr.json <<EOF { "CN": "system:kube-scheduler", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "192.168.1.67", "192.168.1.68", "192.168.1.69" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:kube-scheduler", "OU": "System" } ] } EOF
- 脚本内容如下:
rm -rf master rm -rf node mkdir master mkdir node cat > ca-config.json <<EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF cat > ca-csr.json <<EOF { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing", "O": "k8s", "OU": "System" } ] } EOF cfssl gencert -initca ca-csr.json | cfssljson -bare ca - #----------------------- kube-apiserver echo "generate kube-apiserver cert" cd master cat > kube-apiserver-csr.json <<EOF { "CN": "kubernetes", "hosts": [ "127.0.0.1", "192.168.1.99", "192.168.1.67", "192.168.1.68", "192.168.1.69", "10.254.0.1", "kubernetes", "kubernetes.default", "kubernetes.default.svc", "kubernetes.default.svc.cluster", "kubernetes.default.svc.cluster.local" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "k8s", "OU": "System" } ] } EOF cfssl gencert -ca=../ca.pem -ca-key=../ca-key.pem -config=../ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver cd .. #----------------------- kubectl echo "generate kubectl cert" cat > admin-csr.json <<EOF { "CN": "admin", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:masters", "OU": "System" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin #----------------------- kube-controller-manager echo "generate kube-controller-manager cert" cd master cat > kube-controller-manager-csr.json <<EOF { "CN": "system:kube-controller-manager", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "192.168.1.67", "192.168.1.68", "192.168.1.69" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:kube-controller-manager", "OU": "System" } ] } EOF cfssl gencert -ca=../ca.pem -ca-key=../ca-key.pem -config=../ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager cd .. #----------------------- kube-scheduler echo "generate kube-scheduler cert" cd master cat > kube-scheduler-csr.json <<EOF { "CN": "system:kube-scheduler", "key": { "algo": "rsa", "size": 2048 }, "hosts": [ "127.0.0.1", "192.168.1.67", "192.168.1.68", "192.168.1.69" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:kube-scheduler", "OU": "System" } ] } EOF cfssl gencert -ca=../ca.pem -ca-key=../ca-key.pem -config=../ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler cd .. #----------------------- kube-proxy echo "generate kube-proxy cert" cd node cat > kube-proxy-csr.json <<EOF { "CN": "system:kube-proxy", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing", "O": "system:kube-proxy", "OU": "System" } ] } EOF cfssl gencert -ca=../ca.pem -ca-key=../ca-key.pem -config=../ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy cd ..
- 在其中一台机器上(如 192.168.1.67)使用 k8s/scripts/k8s-cert.sh 脚本生成认证,并复制到 /opt/kubernetes/ssl/ 中
$ mkdir -p k8s/k8s-cert $ cp k8s/scripts/k8s-cert.sh k8s/k8s-cert $ cd k8s/k8s-cert $ ./k8s-cert.sh $ sudo cp -r ca* kube* master node /opt/kubernetes/ssl/
- 复制所有的证书到另外两个节点上
$ sudo scp -r /opt/kubernetes/ssl/* root@192.168.1.68:/opt/kubernetes/ssl/ $ sudo scp -r /opt/kubernetes/ssl/* root@192.168.1.69:/opt/kubernetes/ssl/
2.9 脚本生成各个组件的 kubeconfig
- 在 192.168.1.67、192.168.1.68 和 192.168.1.69 分别创建 .kube 文件夹
$ mkdir -p ~/.kube
- 在其中一台机器上(如 192.168.1.67)创建工作目录,复制 k8s/scripts/kubeconfig.sh 脚本
$ mkdir -p k8s/kubeconfig $ cp k8s/scripts/kubeconfig.sh k8s/kubeconfig $ cd k8s/kubeconfig
- 执行脚本生成各个组件的 kubeconfig,第一参数为虚拟 IP 地址,第二个参数为 kubernetes 安装目录的证书文件
$ sudo ./kubeconfig.sh 192.168.1.99 /opt/kubernetes/ssl 0524b3077444a437dfc662e5739bfa1a ===> generate kubectl config Cluster "kubernetes" set. User "admin" set. Context "admin@kubernetes" created. Switched to context "admin@kubernetes". ===> generate kube-controller-manager.kubeconfig Cluster "kubernetes" set. User "system:kube-controller-manager" set. Context "system:kube-controller-manager@kubernetes" created. Switched to context "system:kube-controller-manager@kubernetes". ===> generate kube-scheduler.kubeconfig Cluster "kubernetes" set. User "system:kube-scheduler" set. Context "system:kube-scheduler@kubernetes" created. Switched to context "system:kube-scheduler@kubernetes". ===> generate kubelet bootstrapping kubeconfig Cluster "kubernetes" set. User "kubelet-bootstrap" set. Context "default" created. Switched to context "default". ===> generate kube-proxy.kubeconfig Cluster "kubernetes" set. User "system:kube-proxy" set. Context "system:kube-proxy@kubernetes" created. Switched to context "system:kube-proxy@kubernetes".
- 脚本内容如下:
#----------------------创建 kube-apiserver TLS Bootstrapping Token BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ') echo ${BOOTSTRAP_TOKEN} cat > token.csv <<EOF ${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap" EOF #---------------------- APISERVER=$1 SSL_DIR=$2 export KUBE_APISERVER="https://$APISERVER:8443" #---------------------- echo "===> generate kubectl config" # 创建 kubectl config kubectl config set-cluster kubernetes \ --certificate-authority=$SSL_DIR/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=config kubectl config set-credentials admin \ --client-certificate=$SSL_DIR/admin.pem \ --client-key=$SSL_DIR/admin-key.pem \ --embed-certs=true \ --kubeconfig=config kubectl config set-context admin@kubernetes \ --cluster=kubernetes \ --user=admin \ --kubeconfig=config kubectl config use-context admin@kubernetes --kubeconfig=config #---------------------- echo "===> generate kube-controller-manager.kubeconfig" # 创建 kube-controller-manager.kubeconfig kubectl config set-cluster kubernetes \ --certificate-authority=$SSL_DIR/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-credentials system:kube-controller-manager \ --client-certificate=$SSL_DIR/master/kube-controller-manager.pem \ --client-key=$SSL_DIR/master/kube-controller-manager-key.pem \ --embed-certs=true \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-context system:kube-controller-manager@kubernetes \ --cluster=kubernetes \ --user=system:kube-controller-manager \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config use-context system:kube-controller-manager@kubernetes --kubeconfig=kube-controller-manager.kubeconfig #---------------------- echo "===> generate kube-scheduler.kubeconfig" # 创建 kube-scheduler.kubeconfig kubectl config set-cluster kubernetes \ --certificate-authority=$SSL_DIR/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-scheduler.kubeconfig kubectl config set-credentials system:kube-scheduler \ --client-certificate=$SSL_DIR/master/kube-scheduler.pem \ --client-key=$SSL_DIR/master/kube-scheduler-key.pem \ --embed-certs=true \ --kubeconfig=kube-scheduler.kubeconfig kubectl config set-context system:kube-scheduler@kubernetes \ --cluster=kubernetes \ --user=system:kube-scheduler \ --kubeconfig=kube-scheduler.kubeconfig kubectl config use-context system:kube-scheduler@kubernetes --kubeconfig=kube-scheduler.kubeconfig #---------------------- echo "===> generate kubelet bootstrapping kubeconfig" # 创建 kubelet bootstrapping kubeconfig # 设置集群参数 kubectl config set-cluster kubernetes \ --certificate-authority=$SSL_DIR/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=bootstrap.kubeconfig # 设置客户端认证参数 kubectl config set-credentials kubelet-bootstrap \ --token=${BOOTSTRAP_TOKEN} \ --kubeconfig=bootstrap.kubeconfig # 设置上下文参数 kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=bootstrap.kubeconfig # 设置默认上下文 kubectl config use-context default --kubeconfig=bootstrap.kubeconfig #---------------------- echo "===> generate kube-proxy.kubeconfig" # 创建 kube-proxy.kubeconfig kubectl config set-cluster kubernetes \ --certificate-authority=$SSL_DIR/ca.pem \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-credentials system:kube-proxy \ --client-certificate=$SSL_DIR/node/kube-proxy.pem \ --client-key=$SSL_DIR/node/kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=kube-proxy.kubeconfig kubectl config set-context system:kube-proxy@kubernetes \ --cluster=kubernetes \ --user=system:kube-proxy \ --kubeconfig=kube-proxy.kubeconfig kubectl config use-context system:kube-proxy@kubernetes --kubeconfig=kube-proxy.kubeconfig
- 复制 config 文件到 ~/.kube 文件夹中
$ sudo chown ao:ao config $ cp config ~/.kube
- 同时复制 config 文件到其他两台机器上
$ scp config ao@192.168.1.68:/home/ao/.kube $ scp config ao@192.168.1.69:/home/ao/.kube
- 复制 token.csv 文件到 /opt/kubernetes/cfg 中
$ sudo cp token.csv /opt/kubernetes/cfg
- 复制到 token.csv 到另外两台机器上
$ sudo scp token.csv root@192.168.1.68:/opt/kubernetes/cfg $ sudo scp token.csv root@192.168.1.69:/opt/kubernetes/cfg
- 其他文件也需要复制到对应机器上的对应目录,后面安装其他组件时继续介绍
2.10 安装 kube-apiserver
- 在 192.168.1.67、192.168.1.68 和 192.168.1.69 上复制上述提到的 kube-apiserver、kubectl、kube-controller-manager、kube-scheduler、kubelet 和 kube-proxy 到 /
opt/kubernetes/bin/ 中
$ cp kube-apiserver kubectl kube-controller-manager kube-scheduler kubelet kube-proxy /opt/kubernetes/bin/
- 在三台机器上分别执行 k8s/scripts/apiserver.sh 脚本,启动 kube-apiserver.service 服务,第一个参数为 Master 节点地址,第二个为 etcd 集群地址
$ sudo ./k8s/scripts/apiserver.sh 192.168.1.67 https://192.168.1.67:2379,https://192.168.1.68:2379,https://192.168.1.69:2379 $ sudo ./k8s/scripts/apiserver.sh 192.168.1.68 https://192.168.1.67:2379,https://192.168.1.68:2379,https://192.168.1.69:2379 $ sudo ./k8s/scripts/apiserver.sh 192.168.1.69 https://192.168.1.67:2379,https://192.168.1.68:2379,https://192.168.1.69:2379
- 脚本内容如下:
#!/bin/bash MASTER_ADDRESS=$1 ETCD_SERVERS=$2 systemctl stop kube-apiserver systemctl disable kube-apiserver cat <<EOF >/opt/kubernetes/cfg/kube-apiserver KUBE_APISERVER_OPTS="--logtostderr=true \\ --v=4 \\ --anonymous-auth=false \\ --etcd-servers=${ETCD_SERVERS} \\ --etcd-cafile=/opt/etcd/ssl/ca.pem \\ --etcd-certfile=/opt/etcd/ssl/server.pem \\ --etcd-keyfile=/opt/etcd/ssl/server-key.pem \\ --service-cluster-ip-range=10.254.0.0/16 \\ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\ --bind-address=${MASTER_ADDRESS} \\ --secure-port=6443 \\ --client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --token-auth-file=/opt/kubernetes/cfg/token.csv \\ --allow-privileged=true \\ --tls-cert-file=/opt/kubernetes/ssl/master/kube-apiserver.pem \\ --tls-private-key-file=/opt/kubernetes/ssl/master/kube-apiserver-key.pem \\ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --advertise-address=${MASTER_ADDRESS} \\ --authorization-mode=RBAC,Node \\ --kubelet-https=true \\ --enable-bootstrap-token-auth \\ --kubelet-certificate-authority=/opt/kubernetes/ssl/ca.pem \\ --kubelet-client-key=/opt/kubernetes/ssl/master/kube-apiserver-key.pem \\ --kubelet-client-certificate=/opt/kubernetes/ssl/master/kube-apiserver.pem \\ --service-node-port-range=30000-50000" EOF cat <<EOF >/usr/lib/systemd/system/kube-apiserver.service [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable kube-apiserver systemctl restart kube-apiserver
- 脚本会创建 kube-apiserver.service 的服务,查看服务状态
$ systemctl status kube-apiserver.service ● kube-apiserver.service - Kubernetes API Server Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-02-25 04:05:07 UTC; 2s ago Docs: https://github.com/kubernetes/kubernetes Main PID: 7025 (kube-apiserver) Tasks: 11 (limit: 2317) CGroup: /system.slice/kube-apiserver.service └─7025 /opt/kubernetes/bin/kube-apiserver --logtostderr=true --v=4 --anonymous-auth=false --etcd-servers Feb 25 04:05:08 master1 kube-apiserver[7025]: I0225 04:05:08.782348 7025 endpoint.go:68] ccResolverWrapper: send Feb 25 04:05:08 master1 kube-apiserver[7025]: I0225 04:05:08.782501 7025 reflector.go:211] Listing and watching Feb 25 04:05:08 master1 kube-apiserver[7025]: I0225 04:05:08.788181 7025 store.go:1362] Monitoring apiservices.a Feb 25 04:05:08 master1 kube-apiserver[7025]: I0225 04:05:08.789982 7025 watch_cache.go:449] Replace watchCache Feb 25 04:05:08 master1 kube-apiserver[7025]: I0225 04:05:08.790831 7025 deprecated_insecure_serving.go:53] Serv Feb 25 04:05:08 master1 kube-apiserver[7025]: I0225 04:05:08.794071 7025 reflector.go:211] Listing and watching Feb 25 04:05:08 master1 kube-apiserver[7025]: I0225 04:05:08.797572 7025 watch_cache.go:449] Replace watchCache Feb 25 04:05:09 master1 kube-apiserver[7025]: I0225 04:05:09.026015 7025 client.go:361] parsed scheme: "endpoint Feb 25 04:05:09 master1 kube-apiserver[7025]: I0225 04:05:09.026087 7025 endpoint.go:68] ccResolverWrapper: send Feb 25 04:05:09 master1 kube-apiserver[7025]: I0225 04:05:09.888015 7025 aggregator.go:109] Building initial Ope lines 1-19/19 (END)
- 查看 kube-apiservce.service 的配置文件
$ cat /opt/kubernetes/cfg/kube-apiserver KUBE_APISERVER_OPTS="--logtostderr=true \ --v=4 \ --anonymous-auth=false \ --etcd-servers=https://192.168.1.67:2379,https://192.168.1.68:2379,https://192.168.1.69:2379 \ --etcd-cafile=/opt/etcd/ssl/ca.pem \ --etcd-certfile=/opt/etcd/ssl/server.pem \ --etcd-keyfile=/opt/etcd/ssl/server-key.pem \ --service-cluster-ip-range=10.254.0.0/16 \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \ --bind-address=192.168.1.67 \ --secure-port=6443 \ --client-ca-file=/opt/kubernetes/ssl/ca.pem \ --token-auth-file=/opt/kubernetes/cfg/token.csv \ --allow-privileged=true \ --tls-cert-file=/opt/kubernetes/ssl/master/kube-apiserver.pem \ --tls-private-key-file=/opt/kubernetes/ssl/master/kube-apiserver-key.pem \ --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \ --advertise-address=192.168.1.67 \ --authorization-mode=RBAC,Node \ --kubelet-https=true \ --enable-bootstrap-token-auth \ --kubelet-certificate-authority=/opt/kubernetes/ssl/ca.pem \ --kubelet-client-key=/opt/kubernetes/ssl/master/kube-apiserver-key.pem \ --kubelet-client-certificate=/opt/kubernetes/ssl/master/kube-apiserver.pem \ --service-node-port-range=30000-50000"
-
授予 kubernetes 证书访问 kubelet API 的权限
$ kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes
2.11 安装 kube-scheduler
- 在 192.168.1.67(上述生成各个组件的 kubeconfig 的机器)上复制 kube-scheduler.kubeconfig 到指定目录和其他节点的指定目录:
$ cd k8s/kubeconfig $ sudo cp kube-scheduler.kubeconfig /opt/kubernetes/cfg $ sudo scp kube-scheduler.kubeconfig root@192.168.1.68:/opt/kubernetes/cfg $ sudo scp kube-scheduler.kubeconfig root@192.168.1.69:/opt/kubernetes/cfg
- 在三台节点上执行 k8s/scripts/scheduler.sh 脚本,创建 kube-scheduler.service 服务并启动
$ sudo ./k8s/scripts/scheduler.sh
- 脚本内容如下:
#!/bin/bash systemctl stop kube-scheduler systemctl disable kube-scheduler cat <<EOF >/opt/kubernetes/cfg/kube-scheduler KUBE_SCHEDULER_OPTS="--logtostderr=true \\ --v=4 \\ --bind-address=0.0.0.0 \\ --port=10251 \\ --secure-port=10259 \\ --kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\ --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --authentication-kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\ --authorization-kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\ --client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --tls-cert-file=/opt/kubernetes/ssl/master/kube-scheduler.pem \\ --tls-private-key-file=/opt/kubernetes/ssl/master/kube-scheduler-key.pem \\ --leader-elect=true" EOF cat <<EOF >/usr/lib/systemd/system/kube-scheduler.service [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable kube-scheduler systemctl restart kube-scheduler
- 脚本会创建 kube-scheduler.service 服务,查看服务状态
$ sudo systemctl status kube-scheduler.service ● kube-scheduler.service - Kubernetes Scheduler Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-02-25 04:11:03 UTC; 1min 7s ago Docs: https://github.com/kubernetes/kubernetes Main PID: 7701 (kube-scheduler) Tasks: 10 (limit: 2317) CGroup: /system.slice/kube-scheduler.service └─7701 /opt/kubernetes/bin/kube-scheduler --logtostderr=true --v=4 --bind-address=0.0.0.0 --port=10251 - Feb 25 04:11:43 master1 kube-scheduler[7701]: I0225 04:11:43.911182 7701 reflector.go:211] Listing and watching Feb 25 04:11:43 master1 kube-scheduler[7701]: E0225 04:11:43.913844 7701 reflector.go:178] k8s.io/kubernetes/cmd Feb 25 04:11:46 master1 kube-scheduler[7701]: I0225 04:11:46.438728 7701 reflector.go:211] Listing and watching Feb 25 04:11:46 master1 kube-scheduler[7701]: E0225 04:11:46.441883 7701 reflector.go:178] k8s.io/client-go/info Feb 25 04:11:47 master1 kube-scheduler[7701]: I0225 04:11:47.086981 7701 reflector.go:211] Listing and watching Feb 25 04:11:47 master1 kube-scheduler[7701]: E0225 04:11:47.088902 7701 reflector.go:178] k8s.io/client-go/info Feb 25 04:12:09 master1 kube-scheduler[7701]: I0225 04:12:09.120429 7701 reflector.go:211] Listing and watching Feb 25 04:12:09 master1 kube-scheduler[7701]: E0225 04:12:09.123594 7701 reflector.go:178] k8s.io/client-go/info Feb 25 04:12:09 master1 kube-scheduler[7701]: I0225 04:12:09.788768 7701 reflector.go:211] Listing and watching Feb 25 04:12:09 master1 kube-scheduler[7701]: E0225 04:12:09.790724 7701 reflector.go:178] k8s.io/client-go/info lines 1-19/19 (END)
2.12 安装 kube-controller-manager
- 在 192.168.1.67(上述生成各个组件的 kubeconfig 的机器)上复制 kube-controller-manager.kubeconfig 到指定目录和其他节点的指定目录:
$ cd k8s/kubeconfig $ sudo cp kube-controller-manager.kubeconfig /opt/kubernetes/cfg $ sudo scp kube-controller-manager.kubeconfig root@192.168.1.68:/opt/kubernetes/cfg $ sudo scp kube-controller-manager.kubeconfig root@192.168.1.69:/opt/kubernetes/cfg
- 在三台节点上执行 k8s/scripts/controller-manager.sh 脚本,创建 kube-controller-manager.service 服务并启动
$ sudo ./k8s/scripts/controller-manager.sh
- 脚本内容如下:
#!/bin/bash systemctl stop kube-controller-manager.service systemctl disable kube-controller-manager.service cat <<EOF >/opt/kubernetes/cfg/kube-controller-manager KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \\ --v=4 \\ --bind-address=0.0.0.0 \\ --cluster-name=kubernetes \\ --kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\ --requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\ --authentication-kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\ --authorization-kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\ --leader-elect=true \\ --service-cluster-ip-range=10.254.0.0/16 \\ --controllers=*,bootstrapsigner,tokencleaner \\ --tls-cert-file=/opt/kubernetes/ssl/master/kube-controller-manager.pem \\ --tls-private-key-file=/opt/kubernetes/ssl/master/kube-controller-manager-key.pem \\ --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\ --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --root-ca-file=/opt/kubernetes/ssl/ca.pem \\ --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\ --secure-port=10257 \\ --use-service-account-credentials=true \\ --experimental-cluster-signing-duration=87600h0m0s" EOF #--allocate-node-cidrs=true \\ #--cluster-cidr=172.17.0.0/16 \\ cat <<EOF >/usr/lib/systemd/system/kube-controller-manager.service [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes [Service] EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable kube-controller-manager systemctl restart kube-controller-manager
- 脚本会创建 kube-controller-manager.service 服务,查看服务状态
$ systemctl status kube-controller-manager.service ● kube-controller-manager.service - Kubernetes Controller Manager Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2020-02-23 13:06:29 UTC; 15min ago Docs: https://github.com/kubernetes/kubernetes Main PID: 22526 (kube-controller) Tasks: 9 (limit: 2317) CGroup: /system.slice/kube-controller-manager.service └─22526 /opt/kubernetes/bin/kube-controller-manager --logtostderr=false --v=3 --bind-address=192.168.1.6 Feb 23 13:06:29 clean systemd[1]: Started Kubernetes Controller Manager.
- 将二进制文件目录加入环境变量:export PATH=$PATH:/opt/kubernetes/bin/
$ vim ~/.zshrc ...... export PATH=$PATH:/opt/kubernetes/bin/ $ source ~/.zshrc
- 在任意一个机器上执行如下命令,查看集群状态
$ kubectl cluster-info Kubernetes master is running at https://192.168.1.99:8443 To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. $ kubectl get cs NAME STATUS MESSAGE ERROR scheduler Healthy ok controller-manager Healthy ok etcd-1 Healthy {"health":"true"} etcd-2 Healthy {"health":"true"} etcd-0 Healthy {"health":"true"}
2.13 安装 kubelet
- 从此节开始,安装的组件均为 Node 节点使用
- 在 192.168.1.67(上述生成各个组件的 kubeconfig 的机器)上复制 bootstrap.kubeconfig 到指定目录和其他节点的指定目录:
$ cd k8s/kubeconfig $ sudo cp kube-controller-manager.kubeconfig /opt/kubernetes/cfg $ sudo scp kube-controller-manager.kubeconfig root@192.168.1.68:/opt/kubernetes/cfg $ sudo scp kube-controller-manager.kubeconfig root@192.168.1.69:/opt/kubernetes/cfg
- 在三台节点上执行 k8s/scripts/kubelet.sh 脚本,创建 kubelet.service 服务并启动,第一个参数为 Node 节点地址,第二个参数为 Node 节点在 kubernetes 中显示的名称
sudo ./k8s/scripts/kubelet.sh 192.168.1.67 node1 sudo ./k8s/scripts/kubelet.sh 192.168.1.68 node2 sudo ./k8s/scripts/kubelet.sh 192.168.1.69 node3
- 脚本内容如下:
#!/bin/bash NODE_ADDRESS=$1 NODE_NAME=$2 DNS_SERVER_IP=${3:-"10.254.0.2"} systemctl stop kubelet systemctl disable kubelet cat <<EOF >/opt/kubernetes/cfg/kubelet KUBELET_OPTS="--logtostderr=true \\ --v=4 \\ --config=/opt/kubernetes/cfg/kubelet.config \\ --node-ip=${NODE_ADDRESS} \\ --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\ --cert-dir=/opt/kubernetes/ssl/node \\ --hostname-override=${NODE_NAME} \\ --node-labels=node.kubernetes.io/k8s-master=true \\ --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\ --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0" EOF #--cni-bin-dir=/opt/cni/bin \\ #--cni-conf-dir=/opt/cni/net.d \\ #--network-plugin=cni \\ cat <<EOF >/opt/kubernetes/cfg/kubelet.config kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 address: ${NODE_ADDRESS} port: 10250 readOnlyPort: 10255 cgroupDriver: cgroupfs clusterDNS: - ${DNS_SERVER_IP} clusterDomain: cluster.local. failSwapOn: false authentication: anonymous: enabled: false webhook: enabled: true x509: clientCAFile: "/opt/kubernetes/ssl/ca.pem" authorization: mode: Webhook EOF cat <<EOF >/usr/lib/systemd/system/kubelet.service [Unit] Description=Kubernetes Kubelet After=docker.service Requires=docker.service [Service] EnvironmentFile=/opt/kubernetes/cfg/kubelet ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS Restart=on-failure KillMode=process [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable kubelet systemctl restart kubelet
- 创建 bootstrap 角色赋予权限用于连接 kube-apiserver 请求签名
$ kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
- 检查请求
$ kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-HHJMkN9RvwkgTkWGJThtsIPPlexh1Ci5vyOcjEhwk5c 29s kubelet-bootstrap Pending node-csr-Ih-JtbfHPzP8u0_YI0By7RWMPCEfaEpapi47kil1YbU 4s kubelet-bootstrap Pending node-csr-eyb0y_uxEWgPHnUQ2DyEhCK09AkirUp11O3b40zFyAQ 1s kubelet-bootstrap Pending
- 同意请求并颁发证书
$ kubectl certificate approve node-csr-HHJMkN9RvwkgTkWGJThtsIPPlexh1Ci5vyOcjEhwk5c node-csr-Ih-JtbfHPzP8u0_YI0By7RWMPCEfaEpapi47kil1YbU node-csr-eyb0y_uxEWgPHnUQ2DyEhCK09AkirUp11O3b40zFyAQ certificatesigningrequest.certificates.k8s.io/node-csr-HHJMkN9RvwkgTkWGJThtsIPPlexh1Ci5vyOcjEhwk5c approved certificatesigningrequest.certificates.k8s.io/node-csr-Ih-JtbfHPzP8u0_YI0By7RWMPCEfaEpapi47kil1YbU approved certificatesigningrequest.certificates.k8s.io/node-csr-eyb0y_uxEWgPHnUQ2DyEhCK09AkirUp11O3b40zFyAQ approved $ kubectl get csr NAME AGE REQUESTOR CONDITION node-csr-HHJMkN9RvwkgTkWGJThtsIPPlexh1Ci5vyOcjEhwk5c 3m37s kubelet-bootstrap Approved,Issued node-csr-Ih-JtbfHPzP8u0_YI0By7RWMPCEfaEpapi47kil1YbU 3m12s kubelet-bootstrap Approved,Issued node-csr-eyb0y_uxEWgPHnUQ2DyEhCK09AkirUp11O3b40zFyAQ 3m9s kubelet-bootstrap Approved,Issued
- 查看集群节点
$ kubectl get node NAME STATUS ROLES AGE VERSION node1 Ready <none> 2m36s v1.18.0-alpha.5.158+1c60045db0bd6e node2 Ready <none> 2m36s v1.18.0-alpha.5.158+1c60045db0bd6e node3 Ready <none> 2m36s v1.18.0-alpha.5.158+1c60045db0bd6e
- 已经是 Ready 状态,说明加入成功
- 由于该 Node 同时也是 Master 角色,因此需要标记一下
$ kubectl label node node1 node2 node3 node-role.kubernetes.io/master=true node/node1 labeled node/node2 labeled node/node3 labeled $ kubectl get node NAME STATUS ROLES AGE VERSION node1 Ready master 3m36s v1.18.0-alpha.5.158+1c60045db0bd6e node2 Ready master 3m36s v1.18.0-alpha.5.158+1c60045db0bd6e node3 Ready master 3m36s v1.18.0-alpha.5.158+1c60045db0bd6e
-
允许 Master 节点上部署 Pod:
$ kubectl taint nodes --all node-role.kubernetes.io/master=true:NoSchedule $ kubectl taint nodes node1 node2 node3 node-role.kubernetes.io/master-
- 注意:理论上只调用上述第二个命令即可,但是实际上会出现 “taint "node-role.kubernetes.io/master" not found” 错误,因此加上了第一个命令
2.14 安装 kube-proxy
- 在 192.168.1.67(上述生成各个组件的 kubeconfig 的机器)上复制 kube-proxy.kubeconfig 到指定目录和其他节点的指定目录:
$ cd k8s/kubeconfig $ sudo cp kube-proxy.kubeconfig /opt/kubernetes/cfg $ sudo scp kube-proxy.kubeconfig root@192.168.1.68:/opt/kubernetes/cfg $ sudo scp kube-proxy.kubeconfig root@192.168.1.69:/opt/kubernetes/cfg
- 在三台执行 k8s/scripts/proxy.sh 脚本,创建 kube-proxy.service 服务并启动,第一个参数为 Node 节点显示名称(需要与 kubelet 中的 Node 节点名称对应)
$ ./k8s/scripts/proxy.sh node1 $ ./k8s/scripts/proxy.sh node2 $ ./k8s/scripts/proxy.sh node3
- 脚本内容如下:
#!/bin/bash NODE_NAME=$1 systemctl stop kube-proxy systemctl disable kube-proxy cat <<EOF >/opt/kubernetes/cfg/kube-proxy KUBE_PROXY_OPTS="--logtostderr=true \\ --v=4 \\ --bind-address=0.0.0.0 \\ --hostname-override=${NODE_NAME} \\ --cleanup-ipvs=true \\ --cluster-cidr=10.254.0.0/16 \\ --proxy-mode=ipvs \\ --ipvs-min-sync-period=5s \\ --ipvs-sync-period=5s \\ --ipvs-scheduler=wrr \\ --masquerade-all=true \\ --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig" EOF cat <<EOF >/usr/lib/systemd/system/kube-proxy.service [Unit] Description=Kubernetes Proxy After=network.target [Service] EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl daemon-reload systemctl enable kube-proxy systemctl restart kube-proxy
- 脚本会创建 kube-proxy.service 服务,查看服务状态
$ sudo systemctl status kube-proxy.service ● kube-proxy.service - Kubernetes Proxy Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-02-25 07:32:58 UTC; 12s ago Main PID: 30924 (kube-proxy) Tasks: 7 (limit: 2317) CGroup: /system.slice/kube-proxy.service └─30924 /opt/kubernetes/bin/kube-proxy --logtostderr=true --v=4 --bind-address=0.0.0.0 --hostname-overri Feb 25 07:33:01 master3 kube-proxy[30924]: I0225 07:33:01.877754 30924 config.go:169] Calling handler.OnEndpoints Feb 25 07:33:02 master3 kube-proxy[30924]: I0225 07:33:02.027867 30924 config.go:169] Calling handler.OnEndpoints Feb 25 07:33:03 master3 kube-proxy[30924]: I0225 07:33:03.906364 30924 config.go:169] Calling handler.OnEndpoints Feb 25 07:33:04 master3 kube-proxy[30924]: I0225 07:33:04.058010 30924 config.go:169] Calling handler.OnEndpoints Feb 25 07:33:05 master3 kube-proxy[30924]: I0225 07:33:05.937519 30924 config.go:169] Calling handler.OnEndpoints Feb 25 07:33:06 master3 kube-proxy[30924]: I0225 07:33:06.081698 30924 config.go:169] Calling handler.OnEndpoints Feb 25 07:33:07 master3 kube-proxy[30924]: I0225 07:33:07.970036 30924 config.go:169] Calling handler.OnEndpoints Feb 25 07:33:08 master3 kube-proxy[30924]: I0225 07:33:08.118982 30924 config.go:169] Calling handler.OnEndpoints Feb 25 07:33:09 master3 kube-proxy[30924]: I0225 07:33:09.996659 30924 config.go:169] Calling handler.OnEndpoints Feb 25 07:33:10 master3 kube-proxy[30924]: I0225 07:33:10.148146 30924 config.go:169] Calling handler.OnEndpoints lines 1-18/18 (END)
2.15 检验安装
- 创建 yaml 文件
$ mkdir -p k8s/yamls $ cd k8s/yamls $ vim nginx-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx replicas: 2 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80
- 创建 deployment 对象,查看生成的 Pod,进入 Running 状态,说明已经成功创建
$ kubectl apply -f nginx-deployment.yaml $ kubectl get pod NAME READY STATUS RESTARTS AGE nginx-deployment-54f57cf6bf-6d4n5 1/1 Running 0 5s nginx-deployment-54f57cf6bf-zzdv4 1/1 Running 0 5s
- 查看 Pod 具体信息
$ kubectl describe pod nginx-deployment-54f57cf6bf-6d4n5 Name: nginx-deployment-54f57cf6bf-6d4n5 Namespace: default Priority: 0 Node: node3/192.168.1.69 Start Time: Tue, 25 Feb 2020 07:35:08 +0000 Labels: app=nginx pod-template-hash=54f57cf6bf Annotations: <none> Status: Running IP: 172.17.89.2 IPs: IP: 172.17.89.2 Controlled By: ReplicaSet/nginx-deployment-54f57cf6bf Containers: nginx: Container ID: docker://222b1dd1bb57fdd36b4eda31100477531f94a82c844a2f042c444f0a710faf20 Image: nginx:1.7.9 Image ID: docker-pullable://nginx@sha256:e3456c851a152494c3e4ff5fcc26f240206abac0c9d794affb40e0714846c451 Port: 80/TCP Host Port: 0/TCP State: Running Started: Tue, 25 Feb 2020 07:35:10 +0000 Ready: True Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-p92fn (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-p92fn: Type: Secret (a volume populated by a Secret) SecretName: default-token-p92fn Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned default/nginx-deployment-54f57cf6bf-6d4n5 to node3 Normal Pulled 117s kubelet, node3 Container image "nginx:1.7.9" already present on machine Normal Created 117s kubelet, node3 Created container nginx Normal Started 117s kubelet, node3 Started container nginx
- 若查看时报如下错误:
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy)
- 则需要给集群加一个 cluster-admin 权限:
$ kubectl create clusterrolebinding system:anonymous --clusterrole=cluster-admin --user=system:anonymous
3. 小结
- 当前部署 kubelet 没有以 cni 的网络插件启动,因此不能跨节点访问 pod,后续学习中加入
- 上述的脚本均上传至 github 仓库
- 欢迎各位提出问题和批评