K8S介绍：
官方文档：https://kubernetes.io/zh/docs/concepts/overview/what-is-kubernetes/

（1）添加阿里docker 源
shell> wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

（2）安装docker
shell>yum -y install docker-ce
shell>docker -v
shell> systemctl enable docker
shell>systemctl start docker

（3）安装kubernetes,增加源
shell> cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
##所有节点安装
shell> yum install -y kubelet kubeadm kubectl
shell> systemctl enable kubelet && systemctl start kubelet

（4）初始化k8s master
shell> kubeadm init --apiserver-advertise-address 10.10.202.140 --pod-network-cidr=10.244.0.0/16

kubeadm init \
--apiserver-advertise-address=10.10.202.140 \
--image-repository registry.aliyuncs.com/google_containers \
--pod-network-cidr=10.244.0.0/16
--apiserver-advertise-address 指明用 Master 的哪个 interface 与 Cluster 的其他节点通信。
如果 Master 有多个 interface，建议明确指定，如果不指定，kubeadm 会自动选择有默认网关的 interface。
--pod-network-cidr 指定 Pod 网络的范围。Kubernetes 支持多种网络方案，而且不同网络方案对 --pod-network-cidr 有自己的要求，这里设置为 10.244.0.0/16 是因为我们将使用 flannel 网络方案，必须设置成这个 CIDR

[root@node140 /]# kubeadm init \

--apiserver-advertise-address=10.10.202.140 \
--image-repository registry.aliyuncs.com/google_containers \
--pod-network-cidr=10.244.0.0/16
W1211 22:26:52.608250 70792 version.go:101] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1211 22:26:52.608464 70792 version.go:102] falling back to the local client version: v1.17.0
W1211 22:26:52.608775 70792 validation.go:28] Cannot validate kube-proxy config - no validator is available
W1211 22:26:52.608797 70792 validation.go:28] Cannot validate kubelet config - no validator is available
[init] Using Kubernetes version: v1.17.0
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [node140 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.10.202.140]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [node140 localhost] and IPs [10.10.202.140 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [node140 localhost] and IPs [10.10.202.140 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W1211 22:27:45.746769 70792 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W1211 22:27:45.748837 70792 manifests.go:214] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 34.003938 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node node140 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node node140 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: y6wdsf.dkce7wf8lij4rbgf
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.10.202.140:6443 --token y6wdsf.dkce7wf8lij4rbgf \
--discovery-token-ca-cert-hash sha256:2c307c40531df0dec0908647a9913c09174a0962531694c383fbc14315c1ae07

kubeadm 执行初始化前的检查。

② 生成 token 和证书。

③ 生成 KubeConfig 文件，kubelet 需要这个文件与 Master 通信。

④ 安装 Master 组件，会从 goolge 的 Registry 下载组件的 Docker 镜像，这一步可能会花一些时间，主要取决于网络质量。

⑤ 安装附加组件 kube-proxy 和 kube-dns。

⑥ Kubernetes Master 初始化成功。

⑦ 提示如何配置 kubectl，后面会实践。

⑧ 提示如何安装 Pod 网络，后面会实践。

⑨ 提示如何注册其他节点到 Cluster，后面会实践。

（5）添加tab
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

（6）安装pod 网络
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

（7） 主机添加进入集群

shell> kubeadm join 10.10.202.140:6443 --token y6wdsf.dkce7wf8lij4rbgf \
--discovery-token-ca-cert-hash sha256:2c307c40531df0dec0908647a9913c09174a0962531694c383fbc14315c1ae07

报错：3个报错
[preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher

第一个错误：dockers不是systemd启动 ,处理过程修改docker.service
#ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecStart=/usr/bin/dockerd --exec-opt native.cgroupdriver=systemd

第二个没有设置： systemctl enable kubelet.service
第三个内核没设置： echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
具体根据报错进行解决

（7） 配置kubectl
shell> mkdir -p $HOME/.kube
shell>sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
shell> sudo chown $(id -u):$(id -g) $HOME/.kube/config

（8）查看集群状态
shell> kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}

（9）将节点加入集群
第一步：环境准备
1.node节点关闭防火墙和selinux
2.禁用swap

3. 解析主机名
   4.启动内核功能
   启动kubeket

第二步：加入node141 node142 node143
shell> kubeadm join 10.10.202.141:6443 --token y6wdsf.dkce7wf8lij4rbgf \
--discovery-token-ca-cert-hash sha256:2c307c40531df0dec0908647a9913c09174a0962531694c383fbc14315c1ae07

(10)查看集群
shell> kubectl get nodes
NAME STATUS ROLES AGE VERSION
node140 Ready master 90d v1.17.0
node141 Ready <none> 90d v1.17.0
node142 Ready <none> 90d v1.17.0
node143 Ready <none> 90d v1.17.0
等一会才会变成read状态

（11）移除一个node节点的方法：
（1）进入维护模式
shell> kubectl drain host1 --delete-local-data --force --ignore-daemonsets
（2）删除节点
shell> kubectl delete node node141