k8s 二进制高可用集群部署

2023-09-30 19:09:40

本文档适用于 k8s 1.17+ 版本

本文章将演示 Centos7 二进制方式安装高可用 k8s 1.20+, 相对于其他版本，二进制安装方式并无太大区别，只需要区分每个组件版本的对应关系即可

生产环境中，建议使用小版本大于 5 的 kubernetes 版本，比如 1.19.5 以后的才可用于生产环境

集群安装

基本环境配置

服务器 IP 地址需改成静态 IP

虚拟IP 不要和公司内网 IP 重复，首先去 ping 一下，不通才可用。 VIP 需要和主机在同一个局域网内！

高可用 kubernetes 集群规划

主机名	IP地址	说明
k8s-master01 ~ 03	192.168.32.129 ~ 131	master 节点；3个
k8s-master-lb #有硬件负载的用硬件，云上的用云上自己的负载	192.168.32.233	keepalived 虚拟 IP
node01 ~ 02	192.168.32.132 ~ 133	worker 节点；2个

配置信息	备注
系统版本	Centos7.x
Docker 版本	19.03.x
Pod 网段	172.16.0.0/12
Service 网段	10.96.0.0/12

所有节点配置 hosts, 修改 /etc/hosts 文件如下：

[root@k8s-master01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.32.129 k8s-master01
192.168.32.130 k8s-master02
192.168.32.131 k8s-master03
192.168.32.132 node01
192.168.32.133 node02
192.168.32.233 k8s-master-lb

Centos 7 安装 yum 源如下；

yum install -y wget
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo  #阿里yum base源
wget -P /etc/yum.repos.d/ http://mirrors.aliyun.com/repo/epel-7.repo    #阿里 epel 源

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

必备工具安装

yum -y install wget jq psmics vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git

所有节点关闭防火墙，selinux, dnsmasq , swap

systemctl disable --now firewalld                               #--now 并立即生效
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager

 setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

关闭 swap 分区

swapoff -a && sysctl -w vm.swappiness=0    #关闭 swap分区 并临时设置使用虚拟内存的权重；不建议直接关闭 swap，但是k8s集群建议关闭 swap

 /etc/sysctl.conf增加一行 vm.swappiness = 0     #永久设置

sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab           #注释掉 swap




内核参数vm.swappiness控制换出运行时内存的相对权重，参数值大小对如何使用swap分区有很大联系。值越大，表示越积极使用swap分区，越小表示越积极使用物理内存。默认值swappiness=60，表示内存使用率超过100-60=40%时开始使用交换分区。swappiness=0的时候表示最大限度使用物理内存，然后才是 swap空间；swappiness＝100的时候表示积极使用swap分区，并把内存上的数据及时搬运到swap空间。（网上有的说，对于3.5以后的内核和RedHat 2.6.32之后的内核，设置为0会禁止使用swap，从而引发out of memory，这种情况可以设置为1。）

需要根据服务器运行的程序类型，来设置不同的参数值。例如，对于Oracle一般设置为10；对于MySQL一般设置为1，尽可能不用swap分区。

安装 ntpdate

rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum -y install ntpdate

所有节点同步时间，时间同步配置如下

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime       #描述本机时间
echo 'Asia/Shanghai' > /etc/timezone                          #描述本机所属的时区
ntpdate time2.aliyun.com

加入到 crontab
[root@k8s-master01 yum.repos.d]# crontab -l
*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com

所有节点配置 limit


ulimit -SHn 65535               #设置当前shell,所有用户的最大文件打开数位 65535个，立即生效
vim /etc/security/limits.conf     #永久生效，新开shell，立即生效，默认以这个配置文件为准，如果不去手动设置
# 末尾添加如下内容
[root@k8s-master01 yum.repos.d]# tail -6 /etc/security/limits.conf 
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
[root@k8s-master01 yum.repos.d]# 

ulimit 是一种 Linux 系统的内键功能，它具有一套参数集，用于为由它生成的 shell进程及其子进程的资源使用设置限制。
ulimit的设定值是 per-process 的，也就是说，每个进程有自己的limits值。
使用ulimit进行修改，是 立即生效 的。
ulimit只影响shell进程及其子进程，用户登出后失效。



#nofile 最大文件打开数
#soft   可以超过，会报警
#hard   硬性规定，不能超过
#memlock - 最大锁定内存地址空间
#noproc  - 进程的最大数目
#unlimited   无限制的

Master01 节点免密钥登陆其他节点，安装过程中生成配置文件和证书（kubeadm不需要手动生成证书)均在 Master01 上操作，集群管理也在 Master01 上操作，阿里云或者 AWS 上需要单独一台 kubectl 服务器。密钥配置如下；

ssh-keygen -t rsa
for i in k8s-master01 k8s-master02 k8s-master03 node01 node02;do ssh-copy-id -i /root/.ssh/id_rsa.pub $i;done

下载安装所有的源码文件（master01上操作就行了）

git clone https://gitee.com/luoluo160717/k8s-ha-install.git

所有节点升级系统并重启，此处升级没有升级内核，下面会单独升级内核

 yum update -y --exclude=kernel* && reboot  #CentOs7 需要升级

内核配置

Centos7 需要升级内核至4.18+，本次升级的版本为4.19

在 master01 节点下载内核

cd /root
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm

for i in k8s-master01 k8s-master02 k8s-master03 node01 node02;do scp kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm $i:/root/;done   #分发到其他节点

所有节点升级内核

yum localinstall -y kernel-ml*

所有节点更改内核启动顺序

grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg 
grubby --args="user_namespace.enable=1"  --update-kernel="$(grubby --default-kernel)"

检查默认内核是不是4.19

[root@k8s-master01 ~]# grubby --default-kernel
/boot/vmlinuz-4.19.12-1.el7.elrepo.x86_64

所有节点重启，然后检查内核是不是 4.19

[root@k8s-master01 ~]# uname -a
Linux k8s-master01 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux

所有节点安装 ipvsadm

yum install ipvsadm ipset sysstat conntrack libseccomp -y

所有节点配置 ipvs 模块，在内核 4.19+ 版本 nf_conntrack_ipv4 已经改为 nf_conntrack, 4.18 以下使用 nf_conntrack_ipv4 即可

vim /etc/modules-load.d/ipvs.conf   #默认不存在
[root@k8s-master01 ~]# cat /etc/modules-load.d/ipvs.conf      #加入如下配置
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip




然后执行 systemctl enable --now systemd-modules-load.service 即可

开启一些k8s集群中必须的内核参数，所有节点配置k8s内核

[root@k8s-master01 ~]# cat <<EOF > /etc/sysctl.d/k8s.conf    
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
net.ipv4.conf.all.route_localnet = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF


执行 sysctl --system   #加载所有的配置文件

所有节点配置完内核后，重启服务器，保证重启后内核加载

reboot
lsmod | grep --color=auto -e ip_vs -e nf_conntrack

基本组件安装

docker-ce, kubernetes 各组件等

所有节点安装 docker-ce 19.03

yum install docker-ce-19.03.* -y

温馨提示；

由于新版 kubelet 建议使用 systemd , 所以可以把 docker 的 CgroupDriver 改成 systemd

mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
"registry-mirrors": ["https://6h6ezoe5.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

所有节点设置开机自启动 docker

systemctl daemon-reload && systemctl enable --now docker

k8s 及 etcd 安装

Master01 下载 kubernetes 二进制 tar.gz 压缩包

GitHub - kubernetes/kubernetes: Production-Grade Container Scheduling and Management

选择对应版本的CHANGELOG-1.X.md文件，点击打开

点击 Server Binaries

右键复制链接或者鼠标点击下载

wget https://dl.k8s.io/v1.20.14/kubernetes-server-linux-amd64.tar.gz  #复制的链接地址

如果网络原因上不去 github ， 可以将其中dl.k8s.io修改为  storage.googleapis.com/kubernetes-release/release

新的下载地址：wget https://storage.googleapis.com/kubernetes-release/release/v1.20.14/kubernetes-server-linux-amd64.tar.gz

Releases · etcd-io/etcd (github.com) #

etcd 二进制包
地址：https://github.com/etcd-io/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-amd64.tar.gz


下载不下来的可以去网盘下载
链接：https://pan.baidu.com/s/16k1WGvhRsdu2iro-GbkNBg 
提取码：3941 
--来自百度网盘超级会员V1的分享

解压包

[root@k8s-master01 ~]# tar -xf  kubernetes-server-linux-amd64.tar.gz --strip-components 3 -C /usr/local/bin  kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}   
#解压指定内容到 /usr/local/bin/  下，



[root@k8s-master01 ~]# tar -xf etcd-v3.4.13-linux-amd64.tar.gz --strip-components 1 -C /usr/local/bin/ etcd-v3.4.13-linux-amd64/etcd{,ctl}

版本查看

[root@k8s-master01 ~]# kubelet --version
Kubernetes v1.20.14
[root@k8s-master01 ~]# etcdctl version
etcdctl version: 3.4.13
API version: 3.4
[root@k8s-master01 ~]#

将组件发送到其他节点

[root@k8s-master01 ~]# masternode='k8s-master02 k8s-master03'
[root@k8s-master01 ~]# worknode='node01 node02'

[root@k8s-master01 ~]# for i in $masternode;do scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $i:/usr/local/bin/; scp /usr/local/bin/etcd* $i:/usr/local/bin/;done


[root@k8s-master01 ~]# for i in $worknode;do scp /usr/local/bin/kube{let,-proxy} $i:/usr/local/bin/; done

所有节点创建 /opt/cni/bin 目录

mkdir -p /opt/cni/bin

master01 上操作，切换到1.20.x 分支（其他版本可以切换到其他分支)

[root@k8s-master01 ~]# cd /root/k8s-ha-install/ 
You have new mail in /var/spool/mail/root
[root@k8s-master01 k8s-ha-install]# git checkout manual-installation-v1.20.x
Branch manual-installation-v1.20.x set up to track remote branch manual-installation-v1.20.x from origin.
Switched to a new branch 'manual-installation-v1.20.x'
[root@k8s-master01 k8s-ha-install]#

生成证书

单向认证双向认证

何为SSL/TLS单向认证，双向认证？ 单向认证指的是只有一个对象校验对端的证书合法性。通常都是client来校验服务器的合法性。那么client需要一个ca.crt,服务器需要server.crt,server.key 双向认证指的是相互校验，服务器需要校验每个client,client也需要校验服务器。 server 需要 server.key 、server.crt 、ca.crt client 需要 client.key 、client.crt 、ca.crt

kubernetesV1.8版本后建议开启TLS双向认证及RBAC授权管理，以加强集群的安全管理。界内流行的开启TLS方法为基于一个“公钥基础设施（public key infrastructure，缩写为 PKI）”，使用了内部托管的认证中心（CA），常见PKI工具有CFSSL,OPENSSL等

CFSSL是CloudFlare开源的一款PKI/TLS工具。 CFSSL 包含一个命令行工具和一个用于签名，验证并且捆绑TLS证书的 HTTP API 服务。使用Go语言编写。

CFSSL包括：

一组用于生成自定义 TLS PKI 的工具
cfssl程序，是CFSSL的命令行工具
multirootca程序是可以使用多个签名密钥的证书颁发机构服务器
mkbundle程序用于构建证书池
cfssljson程序，从cfssl和multirootca程序获取JSON输出，并将证书，密钥，CSR和bundle写入磁盘

PKI借助数字证书和公钥加密技术提供可信任的网络身份。通常，证书就是一个包含如下身份信息的文件：

证书所有组织的信息
公钥
证书颁发组织的信息
证书颁发组织授予的权限，如证书有效期、适用的主机名、用途等
使用证书颁发组织私钥创建的数字签名

安装cfssl

这里我们只用到cfssl工具和cfssljson工具：

Master01 下载生成证书工具

[root@k8s-master01 ~]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/local/bin/cfssl

[root@k8s-master01 ~]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64  -O /usr/local/bin/cfssljson

[root@k8s-master01 ~]# chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson 



网络问题下载不下来的可以去网盘下载
链接：https://pan.baidu.com/s/1VcGd_FQE5GoKsaU3Jx79QA 
提取码：3941 
--来自百度网盘超级会员V1的分享

cfssl工具，子命令介绍：

bundle: 创建包含客户端证书的证书包
genkey: 生成一个key(私钥)和CSR(证书签名请求)
scan: 扫描主机问题
revoke: 吊销证书
certinfo: 输出给定证书的证书信息，跟cfssl-certinfo 工具作用一样
gencrl: 生成新的证书吊销列表
selfsign: 生成一个新的自签名密钥和签名证书
print-defaults: 打印默认配置，这个默认配置可以用作模板
serve: 启动一个HTTP API服务
gencert: 生成新的key(密钥)和签名证书
- -ca：指明ca的证书
- -ca-key：指明ca的私钥文件
- -config：指明请求证书的json文件
- -profile：与-config中的profile对应，是指根据config中的profile段来生成证书的相关信息
ocspdump
ocspsign
info: 获取有关远程签名者的信息
sign: 签名一个客户端证书，通过给定的CA和CA密钥，和主机名
ocsprefresh
ocspserve

cfssl常用命令：

cfssl gencert -initca ca-csr.json | cfssljson -bare ca ## 初始化ca
cfssl gencert -initca -ca-key key.pem ca-csr.json | cfssljson -bare ca ## 使用现有私钥, 重新生成
cfssl certinfo -cert ca.pem
cfssl certinfo -csr ca.csr

所有Master 节点创建 etcd 证书目录

[root@k8s-master01 ~]# mkdir /etc/etcd/ssl -p

所有节点创建 kubernetes 相关目录

[root@k8s-master01 ~]# mkdir -p /etc/kubernetes/pki

Master01 节点生成 etcd 证书

生成证书的 CSR 文件：证书签名请求文件，配置了一些域名，公司，单位

cd /root/k8s-ha-install/pki     

##cfssl print-defaults config > config.json # 默认证书生产策略配置模板
##cfssl print-defaults csr > csr.json #默认csr请求模板




初始化生成 etcd CA 根证书和 etcd CA 证书的 key 

[root@k8s-master01 pki]# cfssl gencert -initca etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca

[root@k8s-master01 pki]# ls /etc/etcd/ssl/
etcd-ca.csr  etcd-ca-key.pem  etcd-ca.pem

生成了 etcd ca的证书 etcd-ca.pem
生产了 etcd ca的私钥 etcd-ca-key.pem  
生成了 etcd ca的请求文件  etcd-ca.csr





生成客户端证书


[root@k8s-master01 pki]# cfssl gencert -ca=/etc/etcd/ssl/etcd-ca.pem -ca-key=/etc/etcd/ssl/etcd-ca-key.pem -config=ca-config.json -hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,192.168.32.129,192.168.32.130,192.168.32.131,192.168.32.144,192.168.32.145 -profile=kubernetes etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd



192.168.32.144  192.168.32.145  #预留地址  可以多写几个

将证书复制到其他 etcd 节点，也就是本实验中的,master02,master03节点

上边已经生成过变量，请注意

[root@k8s-master01 ~]# for i in $masternode;do ssh $i "mkdir -p /etc/etcd/ssl";for j in etcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem;do scp /etc/etcd/ssl/${j} $i:/etc/etcd/ssl/${j};done;done

Master01节点生成 kubernetes 证书

初始化根证书 ,用于k8s 集群
[root@k8s-master01 pki]# cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca










这个证书目前专属于 apiserver，加了一个 k8s-master* 域名以便内部私有 DNS 解析使用(可删除)；至于很多人问过 kubernetes 这几个能不能删掉，答案是不可以的；因为当集群创建好后，default namespace 下会创建一个叫 kubenretes 的 svc，有一些组件会直接连接这个 svc 来跟 api 通讯的，证书如果不包含可能会出现无法连接的情况；其他几个 kubernetes 开头的域名作用相同
    hosts包含的是授权范围，不在此范围的的节点或者服务使用此证书就会报证书不匹配错误。
    10.96.0.1是指kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP。
    #-hostname 配置区域，即一个证书的网站可以是*.youku.com也是可以是*.google.com

[root@k8s-master01 pki]# cfssl gencert   -ca=/etc/kubernetes/pki/ca.pem   -ca-key=/etc/kubernetes/pki/ca-key.pem   -config=ca-config.json   -hostname=10.96.0.1,192.168.32.233,127.0.0.1,k8s-master*,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,192.168.32.129,192.168.32.130,192.168.32.131   -profile=kubernetes   apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver








初始化生成apiserver的聚合证书

[root@k8s-master01 pki]# cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca



[root@k8s-master01 pki]# cfssl gencert   -ca=/etc/kubernetes/pki/front-proxy-ca.pem   -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem   -config=ca-config.json   -profile=kubernetes   front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client













生成controller-manage的证书

[root@k8s-master01 pki]# cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager
   
   
   
   
   
# 注意，如果不是高可用集群，192.168.32.233:8443改为master01的地址，8443改为apiserver的端口，默认是6443
# set-cluster：设置一个集群项,192.168.32.233是VIP
kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://192.168.32.233:8443 \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
     
     
     
     
     


# 设置一个环境项，一个上下文
[root@k8s-master01 pki]# kubectl config set-context system:kube-controller-manager@kubernetes \
    --cluster=kubernetes \
    --user=system:kube-controller-manager \
    --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig 







# set-credentials 设置一个用户项
[root@k8s-master01 pki]# kubectl config set-credentials system:kube-controller-manager \
     --client-certificate=/etc/kubernetes/pki/controller-manager.pem \
     --client-key=/etc/kubernetes/pki/controller-manager-key.pem \
     --embed-certs=true \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
     
     
     
     
     
 
 
# 使用某个环境当做默认环境
[root@k8s-master01 pki]# kubectl config use-context system:kube-controller-manager@kubernetes \
     --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
     
     
     
     
     
     
生成 scheduler 证书
[root@k8s-master01 pki]# cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler
   
   


# 注意，如果不是高可用集群，192.168.32.233:8443改为master01的地址，8443改为apiserver的端口，默认是6443


[root@k8s-master01 pki]# kubectl config set-cluster kubernetes \
     --certificate-authority=/etc/kubernetes/pki/ca.pem \
     --embed-certs=true \
     --server=https://192.168.32.233:8443 \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
     
     
kubectl config set-credentials system:kube-scheduler \
     --client-certificate=/etc/kubernetes/pki/scheduler.pem \
     --client-key=/etc/kubernetes/pki/scheduler-key.pem \
     --embed-certs=true \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
     
     
     
     
kubectl config set-context system:kube-scheduler@kubernetes \
     --cluster=kubernetes \
     --user=system:kube-scheduler \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
     
     
     
kubectl config use-context system:kube-scheduler@kubernetes \
     --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
     
     
cfssl gencert \
   -ca=/etc/kubernetes/pki/ca.pem \
   -ca-key=/etc/kubernetes/pki/ca-key.pem \
   -config=ca-config.json \
   -profile=kubernetes \
   admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin
   
   
   
 
# 注意，如果不是高可用集群，192.168.32.233:8443改为master01的地址，8443改为apiserver的端口，默认是6443
kubectl config set-cluster kubernetes     --certificate-authority=/etc/kubernetes/pki/ca.pem     --embed-certs=true     --server=https://192.168.32.233:8443     --kubeconfig=/etc/kubernetes/admin.kubeconfig



kubectl config set-credentials kubernetes-admin     --client-certificate=/etc/kubernetes/pki/admin.pem     --client-key=/etc/kubernetes/pki/admin-key.pem     --embed-certs=true     --kubeconfig=/etc/kubernetes/admin.kubeconfig




kubectl config set-context kubernetes-admin@kubernetes     --cluster=kubernetes     --user=kubernetes-admin     --kubeconfig=/etc/kubernetes/admin.kubeconfig



kubectl config use-context kubernetes-admin@kubernetes     --kubeconfig=/etc/kubernetes/admin.kubeconfig

创建ServiceAccount Key

[root@k8s-master01 pki]# openssl genrsa -out /etc/kubernetes/pki/sa.key 2048


[root@k8s-master01 pki]# openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

发送证书至其他节点

for NODE in k8s-master02 k8s-master03; do 
for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do 
scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE};
done; 
for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do 
scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE};
done;
done;

查看证书文件

[root@k8s-master01 pki]# ls /etc/kubernetes/pki/
admin.csr      apiserver.csr      ca.csr      controller-manager.csr      front-proxy-ca.csr      front-proxy-client.csr      sa.key         scheduler-key.pem
admin-key.pem  apiserver-key.pem  ca-key.pem  controller-manager-key.pem  front-proxy-ca-key.pem  front-proxy-client-key.pem  sa.pub         scheduler.pem
admin.pem      apiserver.pem      ca.pem      controller-manager.pem      front-proxy-ca.pem      front-proxy-client.pem      scheduler.csr

[root@k8s-master01 pki]# ls /etc/kubernetes/pki/ |wc -l
23

Kubernetes系统组件配置

Etcd配置（所有Master节点）

etcd配置大致相同，注意修改每个Master节点的etcd配置的主机名和IP地址

Master01 节点

[root@k8s-master01 ~]# vim /etc/etcd/etcd.config.yml

name: 'k8s-master01'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.32.129:2380'
listen-client-urls: 'https://192.168.32.129:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.32.129:2380'
advertise-client-urls: 'https://192.168.32.129:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.32.129:2380,k8s-master02=https://192.168.32.130:2380,k8s-master03=https://192.168.32.131:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false

Master02 节点

[root@k8s-master02 ~]#  vim /etc/etcd/etcd.config.yml
name: 'k8s-master02'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.32.130:2380'
listen-client-urls: 'https://192.168.32.130:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.32.130:2380'
advertise-client-urls: 'https://192.168.32.130:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.32.129:2380,k8s-master02=https://192.168.32.130:2380,k8s-master03=https://192.168.32.131:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false

Master03 节点

[root@k8s-master03 ~]#  vim /etc/etcd/etcd.config.yml
name: 'k8s-master03'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://192.168.32.131:2380'
listen-client-urls: 'https://192.168.32.131:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://192.168.32.131:2380'
advertise-client-urls: 'https://192.168.32.131:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'k8s-master01=https://192.168.32.129:2380,k8s-master02=https://192.168.32.130:2380,k8s-master03=https://192.168.32.131:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'
  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false

创建Service

创建etcd service并启动（所有Master节点）

vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service

创建etcd的证书目录（所有Master节点）



mkdir /etc/kubernetes/pki/etcd
ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/
systemctl daemon-reload
systemctl enable --now etcd

查看集群状态（任意master）

[root@k8s-master01 ~]# ETCDCTL_API=3
[root@k8s-master01 ~]# etcdctl --endpoints="192.168.32.129:2379,192.168.32.130:2379,192.168.32.131:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.32.129:2379 | c391821cb3f30b1a |  3.4.13 |   25 kB |     false |      false |      2315 |         10 |                 10 |        |
| 192.168.32.130:2379 | d65c253897bed4d4 |  3.4.13 |   20 kB |      true |      false |      2315 |         10 |                 10 |        |
| 192.168.32.131:2379 | f6e8226e0b0ed00e |  3.4.13 |   20 kB |     false |      false |      2315 |         10 |                 10 |        |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@k8s-master01 ~]#

高可用配置

高可用配置（注意：如果不是高可用集群，haproxy和keepalived无需安装）如果在云上安装也无需执行此章节的步骤，可以直接使用云上的服务

安装keepalived和haproxy（所有Master节点）

yum install keepalived haproxy -y

所有Master节点配置HAProxy，配置都一样

vim /etc/haproxy/haproxy.cfg 
global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend k8s-master
  bind 0.0.0.0:8443
  bind 127.0.0.1:8443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-master01    192.168.32.129:6443  check
  server k8s-master02    192.168.32.130:6443  check
  server k8s-master03    192.168.32.131:6443  check

配置KeepAlived（Master节点）

注意每个节点的IP和网卡（interface参数）

Master01

vim /etc/keepalived/keepalived.conf 

! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5 
    weight -5
    fall 2
    rise 1
}
vrrp_instance VI_1 {
    state MASTER
    interface ens32
    mcast_src_ip 192.168.32.129
    virtual_router_id 51
    priority 101
    nopreempt
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.32.233
    }
    track_script {
      chk_apiserver 
} }

Master02

vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5 
    weight -5
    fall 2
    rise 1
 
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens32
    mcast_src_ip 192.168.32.130
    virtual_router_id 51
    priority 100
    nopreempt
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.32.233
    }
    track_script {
      chk_apiserver 
} }

Master03

vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 5 
    weight -5
    fall 2
    rise 1
 
}
vrrp_instance VI_1 {
    state BACKUP
    interface ens32
    mcast_src_ip 192.168.32.131
    virtual_router_id 51
    priority 100
    nopreempt
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.32.233
    }
    track_script {
      chk_apiserver 
} }

健康检查脚本（所有master节点）

cat > /etc/keepalived/check_apiserver.sh  << EFO
#!/bin/bash

err=0
for k in $(seq 1 3)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi
EFO


# 授权
chmod +x /etc/keepalived/check_apiserver.sh

节点启动haproxy和keepalived（所有master节点）

systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived

VIP测试（master01）

重要：如果安装了keepalived和haproxy，需要测试keepalived是否是正常的

# 看到有VIP绑定到ens33网卡上了
[root@k8s-master01 pki]# ip addr show ens32
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:ce:bd:c6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.32.129/24 brd 192.168.32.255 scope global dynamic ens32
       valid_lft 1164sec preferred_lft 1164sec
    inet 192.168.32.233/32 scope global ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fece:bdc6/64 scope link 
       valid_lft forever preferred_lft forever
[root@k8s-master01 pki]# 


# 任意节点，检查haproxy
telnet 192.168.32.233 8443


如果ping不通且telnet没有出现 "]"，则认为VIP不可以，不可在继续往下执行，需要排查keepalived的问题，比如防火墙和selinux，haproxy和keepalived的状态，监听端口等
所有节点查看防火墙状态必须为disable和inactive：systemctl status firewalld
所有节点查看selinux状态，必须为disable：getenforce
master节点查看haproxy和keepalived状态：systemctl status keepalived haproxy
master节点查看监听端口：netstat -lntp

Kubernetes组件配置

Apiserver

所有Master节点创建kube-apiserver service，# 注意，如果不是高可用集群，192.168.32.233改为master01的地址

注意本文档使用的k8s service网段为10.96.0.0/12，该网段不能和宿主机的网段、Pod网段的重复，请按需修改

Master01配置

[root@k8s-master01 ~]# vim /usr/lib/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
      --v=2  \
      --logtostderr=true  \
      --allow-privileged=true  \
      --bind-address=0.0.0.0  \
      --secure-port=6443  \
      --insecure-port=0  \
      --advertise-address=192.168.32.129 \            # 主机IP    记得把注释删了
      --service-cluster-ip-range=10.96.0.0/12  \     # service网段
      --service-node-port-range=30000-32767  \
      --etcd-servers=https://192.168.32.129:2379,https://192.168.32.130:2379,https://192.168.32.131:2379 \
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \
      --enable-bootstrap-token-auth=true  \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \
      --requestheader-allowed-names=aggregator  \
      --requestheader-group-headers=X-Remote-Group  \
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \
      --requestheader-username-headers=X-Remote-User
      # --token-auth-file=/etc/kubernetes/token.csv

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Master02

[root@k8s-master02 ~]# vim /usr/lib/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
      --v=2  \
      --logtostderr=true  \
      --allow-privileged=true  \
      --bind-address=0.0.0.0  \
      --secure-port=6443  \
      --insecure-port=0  \
      --advertise-address=192.168.32.130 \          # 主机IP
      --service-cluster-ip-range=10.96.0.0/12  \   # servicer IP
      --service-node-port-range=30000-32767  \
      --etcd-servers=https://192.168.32.129:2379,https://192.168.32.130:2379,https://192.168.32.131:2379 \
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \
      --enable-bootstrap-token-auth=true  \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \
      --requestheader-allowed-names=aggregator  \
      --requestheader-group-headers=X-Remote-Group  \
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \
      --requestheader-username-headers=X-Remote-User
      # --token-auth-file=/etc/kubernetes/token.csv

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Master03

[root@k8s-master03 ~]# vim /usr/lib/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
      --v=2  \
      --logtostderr=true  \
      --allow-privileged=true  \
      --bind-address=0.0.0.0  \
      --secure-port=6443  \
      --insecure-port=0  \
      --advertise-address=192.168.32.131 \          # 主机IP     
      --service-cluster-ip-range=10.96.0.0/12  \   # servicer IP
      --service-node-port-range=30000-32767  \
      --etcd-servers=https://192.168.32.129:2379,https://192.168.32.130:2379,https://192.168.32.131:2379 \
      --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem  \
      --etcd-certfile=/etc/etcd/ssl/etcd.pem  \
      --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem  \
      --client-ca-file=/etc/kubernetes/pki/ca.pem  \
      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \
      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \
      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \
      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \
      --service-account-issuer=https://kubernetes.default.svc.cluster.local \
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \
      --authorization-mode=Node,RBAC  \
      --enable-bootstrap-token-auth=true  \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \
      --requestheader-allowed-names=aggregator  \
      --requestheader-group-headers=X-Remote-Group  \
      --requestheader-extra-headers-prefix=X-Remote-Extra-  \
      --requestheader-username-headers=X-Remote-User
      # --token-auth-file=/etc/kubernetes/token.csv

Restart=on-failure
RestartSec=10s
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

启动apiserver（所有Master节点）

systemctl daemon-reload && systemctl enable --now kube-apiserver


# 检测kube-server状态
systemctl status kube-apiserver

配置kube-controller-manager service （所有Master节点）

注意本文档使用的k8s Pod网段为172.16.0.0/12，该网段不能和宿主机的网段、k8s Service网段的重复，请按需修改

三个Master 节点配置文件内容相同

vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
      --v=2 \
      --logtostderr=true \
      --address=127.0.0.1 \
      --root-ca-file=/etc/kubernetes/pki/ca.pem \
      --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \
      --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \
      --service-account-private-key-file=/etc/kubernetes/pki/sa.key \
      --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \
      --leader-elect=true \
      --use-service-account-credentials=true \
      --node-monitor-grace-period=40s \
      --node-monitor-period=5s \
      --pod-eviction-timeout=2m0s \
      --controllers=*,bootstrapsigner,tokencleaner \
      --allocate-node-cidrs=true \
      --cluster-cidr=172.16.0.0/12 \
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \
      --node-cidr-mask-size=24
      
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

启动（所有matser节点）

systemctl daemon-reload && systemctl enable --now kube-controller-manager

查看状态（所有matser节点）

systemctl  status kube-controller-manager

配置kube-scheduler service（所有Master节点）

所有master节点配置文件相同

vim /usr/lib/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-scheduler \
      --v=2 \
      --logtostderr=true \
      --address=127.0.0.1 \
      --leader-elect=true \
      --kubeconfig=/etc/kubernetes/scheduler.kubeconfig

Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

启动

 systemctl daemon-reload && systemctl enable --now kube-scheduler

查看状态

systemctl status kube-scheduler

TLS Bootstrapping配置

Kubernetes 在 1.4 版本(我记着是)推出了 TLS bootstrapping 功能；这个功能主要解决了以下问题:

当集群开启了 TLS 认证后，每个节点的 kubelet 组件都要使用由 apiserver 使用的 CA 签发的有效证书才能与 apiserver 通讯；此时如果节点多起来，为每个节点单独签署证书将是一件非常繁琐的事情；

TLS bootstrapping 功能就是让 kubelet 先使用一个预定的低权限用户连接到 apiserver，然后向 apiserver 申请证书，kubelet 的证书由 apiserver 动态签署；在配合 RBAC 授权模型下的工作

创建bootstrap（Master01节点）

注意，如果不是高可用集群，192.168.32.233:8443改为master01的地址，8443改为apiserver的端口，默认是6443

[root@k8s-master01 ~]# cd /root/k8s-ha-install/bootstrap

kubectl config set-cluster kubernetes     --certificate-authority=/etc/kubernetes/pki/ca.pem     --embed-certs=true     --server=https://192.168.32.233:8443     --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
kubectl config set-credentials tls-bootstrap-token-user     --token=c8ad9c.2e4d610cf3e7426e --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
kubectl config set-context tls-bootstrap-token-user@kubernetes     --cluster=kubernetes     --user=tls-bootstrap-token-user     --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
kubectl config use-context tls-bootstrap-token-user@kubernetes     --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig

注意：如果要修改bootstrap.secret.yaml的token-id和token-secret，需保证各字段的对应关系，还有上面命令中的 --token 字段的对应

[root@k8s-master01 bootstrap]# mkdir -p /root/.kube ; cp /etc/kubernetes/admin.kubeconfig /root/.kube/config
[root@k8s-master01 bootstrap]# kubectl create -f bootstrap.secret.yaml

Node节点配置

复制证书至Node（这里各master 也充当了node）节点，1.19版本以后，建议各master节点，把 kubelet,kube-proxy,都装上，占用不了多少资源，可以通过污点等方式，不运行业务pod就好了

cd /etc/kubernetes/

for NODE in k8s-master02 k8s-master03 node01 node02; do
     ssh $NODE mkdir -p /etc/kubernetes/pki /etc/etcd/ssl /etc/etcd/ssl
     for FILE in etcd-ca.pem etcd.pem etcd-key.pem; do
       scp /etc/etcd/ssl/$FILE $NODE:/etc/etcd/ssl/
     done
     for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do
       scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}
 done
 done

Kubelet配置

创建相关目录（所有节点）

mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/

配置kubelet service（所有节点）

vim  /usr/lib/systemd/system/kubelet.service

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

[Service]
ExecStart=/usr/local/bin/kubelet

Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

配置kubelet service的配置文件（所有节点）

vim /etc/systemd/system/kubelet.service.d/10-kubelet.conf

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

kubelet的配置文件（所有节点）启动所有节点kubelet

注意：如果更改了k8s的service网段，需要更改kubelet-conf.yml 的clusterDNS:配置，改成k8s Service网段的第十个地址，比如10.96.0.10（k8s的service网段开始设置的是10.96.0.0/12）

vim /etc/kubernetes/kubelet-conf.yml

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s

启动kubelet（所有节点）

systemctl daemon-reload
systemctl enable --now kubelet

查看集群状态（matser01上）

[root@k8s-master01 ~]# kubectl get nodes
NAME           STATUS     ROLES    AGE     VERSION
k8s-master01   NotReady   <none>   6m19s   v1.20.14
k8s-master02   NotReady   <none>   5m57s   v1.20.14
k8s-master03   NotReady   <none>   5m57s   v1.20.14
node01         NotReady   <none>   6m48s   v1.20.14
node02         NotReady   <none>   6m48s   v1.20.14

kube-proxy配置

注意，如果不是高可用集群，192.168.32.233:8443改为master01的地址，8443改为apiserver的端口，默认是6443

Master01执行

[root@k8s-master01 ~]# cd /root/k8s-ha-install

kubectl -n kube-system create serviceaccount kube-proxy
kubectl create clusterrolebinding system:kube-proxy         --clusterrole system:node-proxier         --serviceaccount kube-system:kube-proxy
SECRET=$(kubectl -n kube-system get sa/kube-proxy \
    --output=jsonpath='{.secrets[0].name}')
JWT_TOKEN=$(kubectl -n kube-system get secret/$SECRET \
--output=jsonpath='{.data.token}' | base64 -d)
PKI_DIR=/etc/kubernetes/pki
K8S_DIR=/etc/kubernetes
kubectl config set-cluster kubernetes     --certificate-authority=/etc/kubernetes/pki/ca.pem     --embed-certs=true     --server=https://192.168.32.233:8443     --kubeconfig=${K8S_DIR}/kube-proxy.kubeconfig
kubectl config set-credentials kubernetes     --token=${JWT_TOKEN}     --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
kubectl config set-context kubernetes     --cluster=kubernetes     --user=kubernetes     --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
kubectl config use-context kubernetes     --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig

发送kube-proxy的systemd Service文件发送到其他节点（master01上）

如果更改了集群Pod的网段，需要更改kube-proxy/kube-proxy.conf的clusterCIDR: 172.16.0.0/12参数为pod的网段

[root@k8s-master01 ~]# vim /root/k8s-ha-install/kube-proxy/kube-proxy.conf
clusterCIDR: 172.16.0.0/12   #把这个字段更改为，自己的pod 网段

分发配置文件（master01上）

[root@k8s-master01 ~]# cd /root/k8s-ha-install
for NODE in k8s-master01 k8s-master02 k8s-master03; do
     scp ${K8S_DIR}/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig
     scp kube-proxy/kube-proxy.conf $NODE:/etc/kubernetes/kube-proxy.conf
     scp kube-proxy/kube-proxy.service $NODE:/usr/lib/systemd/system/kube-proxy.service
 done

for NODE in node01 node02; do
     scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig
     scp kube-proxy/kube-proxy.conf $NODE:/etc/kubernetes/kube-proxy.conf
     scp kube-proxy/kube-proxy.service $NODE:/usr/lib/systemd/system/kube-proxy.service
 done

启动kube-proxy（所有节点）

 systemctl daemon-reload && systemctl enable --now kube-proxy

安装Calico

[root@k8s-master01 ~]# cd /root/k8s-ha-install/calico/

# 修改calico-etcd.yaml的以下位置

sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.32.129:2379,https://192.168.32.130:2379,https://192.168.32.131:2379"#g' calico-etcd.yaml



ETCD_CA=`cat /etc/kubernetes/pki/etcd/etcd-ca.pem | base64 | tr -d '\n'`
ETCD_CERT=`cat /etc/kubernetes/pki/etcd/etcd.pem | base64 | tr -d '\n'`
ETCD_KEY=`cat /etc/kubernetes/pki/etcd/etcd-key.pem | base64 | tr -d '\n'`


sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico-etcd.yaml


sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico-etcd.yaml


# 更改此处为自己的pod网段
POD_SUBNET="172.16.0.0/12"


sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@#   value: "192.168.0.0/16"@  value: '"${POD_SUBNET}"'@g' calico-etcd.yaml

apply

[root@k8s-master01 calico]# kubectl apply -f calico-etcd.yaml

查看容器状态

[root@k8s-master01 calico]# kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-5f6d4b864b-6jp52   1/1     Running   0          3m27s
calico-node-2glmr                          1/1     Running   0          3m26s
calico-node-bvfkj                          1/1     Running   0          3m26s
calico-node-ffbwq                          1/1     Running   0          3m26s
calico-node-tpk2b                          1/1     Running   0          3m26s
calico-node-vv9r7                          1/1     Running   0          3m26s
[root@k8s-master01 calico]# kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
k8s-master01   Ready    <none>   43m   v1.20.14
k8s-master02   Ready    <none>   43m   v1.20.14
k8s-master03   Ready    <none>   43m   v1.20.14
node01         Ready    <none>   43m   v1.20.14
node02         Ready    <none>   43m   v1.20.14

安装CoreDNS

如果更改了k8s service的网段需要将coredns的serviceIP改成k8s service网段的第十个IP

cd /root/k8s-ha-install/

sed -i "s#10.96.0.10#x.x.x.10#g" CoreDNS/coredns.yaml

安装coredns

[root@k8s-master01 k8s-ha-install]# kubectl  create -f CoreDNS/coredns.yaml

安装Metrics Server

在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率

[root@k8s-master01 ~]# cd /root/k8s-ha-install/metrics-server-0.4.x/
[root@k8s-master01 metrics-server-0.4.x]# kubectl  create -f .

安装dashboard

Dashboard用于展示集群中的各类资源，同时也可以通过Dashboard实时查看Pod的日志和在容器中执行一些命令等。

安装指定版本dashboard

[root@k8s-master01 ~]# cd /root/k8s-ha-install/dashboard/

[root@k8s-master01 dashboard]# kubectl  create -f .

登录dashboard

# 更改dashboard的svc为NodePort
[root@k8s-master01 ~]# kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard
将ClusterIP更改为NodePort

# 查看端口号
[root@k8s-master01 dashboard]#  kubectl get svc kubernetes-dashboard -n kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.107.11.77   <none>        443:31214/TCP   2m49s


# 根据自己的实例端口号，通过任意安装了kube-proxy的宿主机或者VIP的IP+端口即可访问到dashboard

页面访问：https://192.168.32.233:31214

查看token值

[root@k8s-master01 ~]# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

集群验证

安装busybox（master01上）

cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
EOF

验证步骤（matser01上）

1.  Pod必须能解析Service
2.  Pod必须能解析跨namespace的Service
3.  每个节点都必须要能访问Kubernetes的kubernetes svc 443和kube-dns的service 53
4.  Pod和Pod之前要能通
    a)  同namespace能通信
    b)  跨namespace能通信
    c)  跨机器能通信

步骤演示（matser01上）

# 首先查看po是否安装成功
[root@k8s-master01 ~]# kubectl get po
NAME      READY   STATUS    RESTARTS   AGE
busybox   1/1     Running   0          3m11s

# 查看svc是否正常
[root@k8s-master01 ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   163m

# 查看Pod是否能能解析Service
[root@k8s-master01 ~]# kubectl exec  busybox -n default -- nslookup kubernetes 
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

# 查看Pod是否能解析跨namespace的Service
[root@k8s-master01 ~]# kubectl exec  busybox -n default -- nslookup kube-dns.kube-system
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kube-dns.kube-system
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

# 跟我以上结果一致就成功了

使用telnet命令验证

# 所有节点安装telnet命令,有的话忽略
yum install -y telnet

# 所有机器 10.96.0.1  443  kubernetes svc 443
# 所有机器 10.96.0.10 53   kube-dns的service 53
# 不会自动断开就是成功了
telnet 10.96.0.1 443
telnet 10.96.0.10 53

Trying 10.96.0.1...
Connected to 10.96.0.1.
Escape character is '^]'.

使用curl命令验证（所有机器）

[root@k8s-master01 ~]# curl 10.96.0.10:53
curl: (52) Empty reply from server

容器验证（master01上）

[root@k8s-master01 ~]# kubectl get po -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-5f6d4b864b-pq2qw   1/1     Running   0          62m
calico-node-75blv                          1/1     Running   0          62m
calico-node-hw27b                          1/1     Running   0          62m
calico-node-k2wdf                          1/1     Running   0          62m
calico-node-l58lz                          1/1     Running   0          62m
calico-node-v2qlq                          1/1     Running   0          62m
coredns-867d46bfc6-8vzrk                   1/1     Running   0          72m
metrics-server-595f65d8d5-kgn8c            1/1     Running   0          60m

[root@k8s-master01 ~]# kubectl get po -n kube-system -owide
NAME                                       READY   STATUS    RESTARTS   AGE   IP              NODE           NOMINATED NODE   READINESS GATES
calico-kube-controllers-5f6d4b864b-pq2qw   1/1     Running   0          63m   192.168.1.100   k8s-master01   <none>           <none>
calico-node-75blv                          1/1     Running   0          63m   192.168.1.103   k8s-node01     <none>           <none>
calico-node-hw27b                          1/1     Running   0          63m   192.168.1.101   k8s-master02   <none>           <none>
calico-node-k2wdf                          1/1     Running   0          63m   192.168.1.100   k8s-master01   <none>           <none>
calico-node-l58lz                          1/1     Running   0          63m   192.168.1.102   k8s-master03   <none>           <none>
calico-node-v2qlq                          1/1     Running   0          63m   192.168.1.104   k8s-node02     <none>           <none>
coredns-867d46bfc6-8vzrk                   1/1     Running   0          73m   172.161.125.2   k8s-node01     <none>           <none>
metrics-server-595f65d8d5-kgn8c            1/1     Running   0          62m   172.161.125.1   k8s-node01     <none>           <none>

# 能进去就ok
[root@k8s-master01 ~]# kubectl exec -it calico-node-v2qlq -n  kube-system  -- sh
sh-4.4#

# 进入node01，然后能ping通node02就行
[root@k8s-master01 ~]# kubectl exec -it calico-node-v2qlq -n  kube-system  -- bash
[root@k8s-node02 /]# ping 192.168.1.104
PING 192.168.1.104 (192.168.1.104) 56(84) bytes of data.
64 bytes from 192.168.1.104: icmp_seq=1 ttl=64 time=0.123 ms
64 bytes from 192.168.1.104: icmp_seq=2 ttl=64 time=0.090 ms
^C
--- 192.168.1.104 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 46ms
rtt min/avg/max/mdev = 0.090/0.106/0.123/0.019 ms

常用配置

# 所有节点都改
vim /etc/docker/daemon.json

{
"registry-mirrors": ["https://6h6ezoe5.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"],
"max-concurrent-downloads": 10,
"max-concurrent-uploads": 5,
"log-opts": {
  "max-size": "300m",
  "max-file": "2"
},
"live-restore": true
}

# 所有节点改完重启docker
systemctl daemon-reload && systemctl restart docker


max-concurrent-downloads # 下载并发数,最大开启线程下载镜像
max-concurrent-uploads   # 上传并发数，同上
max-size                 # 日志文件最大到多少切割 
max-file                 # 日志文件保留个数 
live-restore             # 开启这个参数，重启docker，不会导致 容器重启

vim /usr/lib/systemd/system/kube-controller-manager.service
 # 找个位置加上,在三个master节点
 --experimental-cluster-signing-duration=876000h0m0s \
 
 # 改完重启
 systemctl daemon-reload && systemctl restart kube-controller-manager
 
 
 ##TLS bootstrapping 时的证书实际是由 kube-controller-manager 组件来签署的，也就是说证书有效期是 kube-controller-manager 组件控制的
默认为 8760h0m0s，将其改为 87600h0m0s 即 10 年后再进行 TLS bootstrapping 签署证书即可
bootstrapping 会在证书快要过期的时候自动续签

# 所有节点，更换成以下的配置文件
[root@k8s-node02 ~]# cat /etc/systemd/system/kubelet.service.d/10-kubelet.conf 

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig"
Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384    --image-pull-progress-deadline=30m "
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS



#增加了  --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        安全加固
        
 #增加了   --image-pull-progress-deadline=30m 
#镜像拉取进度最大时间，如果在这段时间拉取镜像没有任何进展，将取消拉取

码农公寓