一、安装docker
1. docs.docker.com
参照步骤：
https://docs.docker.com/engine/install/centos/
为了加快docker安装过程，将官方的镜像做以下修改：
#官方给的镜像
#yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

#建议使用阿里云的镜像进行加速，要不太慢了
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

systemctl start docker
systemctl enable docker
2.镜像加速器
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://1veb6h8t.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

二、
1. 关闭防火墙
systemctl disable firewalld
systemctl stop firewalld
2. swapoff -a
vi /etc/fstab
注释掉最后一行:/dev/mapper....
重启虚拟机:reboot now
3. 禁用SELinux
修改文件/etc/sysconfig/selinux，将SELINUX=enforcing修改为SELINUX=disabled
三、安装kubeadm
1. 配置/etc/yum.repos.d/kubernetes.repo，内容如下：
cat >> /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
EOF
2. yum install 安装 kubeadm/kubelet/kubectl
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
3.systemctl start kubelet
systemctl enable kubelet
4. kubeadm config images list
k8s.gcr.io/kube-apiserver:v1.22.2
k8s.gcr.io/kube-controller-manager:v1.22.2
k8s.gcr.io/kube-scheduler:v1.22.2
k8s.gcr.io/kube-proxy:v1.22.2
k8s.gcr.io/pause:3.5
k8s.gcr.io/etcd:3.5.0-0
k8s.gcr.io/coredns/coredns:v1.8.4

5.下载Kubernetes的相关镜像
images=( # 下面的镜像应该去除"k8s.gcr.io/"的前缀，版本换成上面获取到的版本
kube-apiserver:v1.22.2
kube-controller-manager:v1.22.2
kube-scheduler:v1.22.2
kube-proxy:v1.22.2
pause:3.5
etcd:3.5.0-0
coredns:v1.8.4
)
for imageName in ${images[@]} ; do
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
done

docker tag k8s.gcr.io/coredns:v1.8.4 k8s.gcr.io/coredns/coredns:v1.8.4
docker rmi k8s.gcr.io/coredns:v1.8.4

问题：DOCKER PULL 镜像失败一直卡在EXTRACTING和DOWNLOAD，已经配置了阿里云加速；将kubelet先停掉，因为还没有安装完,kubelet会报错，
可看日志文件tail -f /var/log/messages；同时可以看一下 docker根目录空间是否不足：df -hl /var/lib/docker

6. 在master节点预运行
kubeadm init phase preflight --ignore-preflight-errors=NumCPU

[preflight] Running pre-flight checks
[WARNING Hostname]: hostname "k8s-master" could not be reached
[WARNING Hostname]: hostname "k8s-master": lookup k8s-master on 10.33.120.13:53: no such host
将 127.0.0.1 映射本机的hostname
/etc/hosts 增加
127.0.0.1 k8s-master
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
--ignore-preflight-errors=NumCPU
[ERROR Swap]: running with swap on is not supported. Please disable swap
swapoff -a
vi /etc/fstab
注释掉最后一行:/dev/mapper....
重启虚拟机:reboot now
[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
echo "1">/proc/sys/net/bridge/bridge-nf-call-iptables
echo "1">/proc/sys/net/bridge/bridge-nf-call-ip6tables
modprobe br_netfilter

[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

7. 运行kubeadm init命令安装Master节点

kubeadm init --apiserver-advertise-address=192.168.56.108 --service-cidr=10.1.0.0/16 --pod-network-cidr=10.222.0.0/16 --apiserver-cert-extra-sans=192.168.56.108 --ignore-preflight-errors=NumCPU

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.56.108:6443 --token 8imilc.kn6uyyylfnukr4d8 \
--discovery-token-ca-cert-hash sha256:94c145d4dcfe0394efa5989907e20a061f84511dcc0b4a1cda3fa74926362a07

8. 因为是root用户，执行：
export KUBECONFIG=/etc/kubernetes/admin.conf

9.CNI网络插件的安装，在master上执行；
如选择Calico CNI插件，可运行如下命令：
kubectl apply -f "https://docs.projectcalico.org/manifests/calico.yaml"

10.检查是否安装正常
kubectl get pods --all-namespaces

四.node节点安装
1. yum install 安装 kubeadm/kubelet （node 节点不需要安装kubectl）
yum install -y kubelet kubeadm --disableexcludes=kubernetes
2. 加入集群
kubeadm join 192.168.56.108:6443 --token 8imilc.kn6uyyylfnukr4d8 \
--discovery-token-ca-cert-hash sha256:94c145d4dcfe0394efa5989907e20a061f84511dcc0b4a1cda3fa74926362a07
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

五. 后续运维
在node上执行kubeadm reset 可以断开node，然后重新join。
在master上执行kubeadm reset后可以重新init
重新启动后，要执行如下命令：
swapoff -a
systemctl daemon-reload
systemctl restart kubelet

六、docker registry

docker pull registry

docker run -p 5000:5000 -v /home/registry_images:/var/lib/registry -d registry

七、jenkins插件

ssh/publish over ssh/git parameter/Maven Integration plugin

Post Steps
执行shell
#!/bin/sh

jarName=spring-boot-demo-0.0.1-SNAPSHOT.jar
jarFolder=ph
projectName=maven
docker_path=${WORKSPACE}
cp ${WORKSPACE}/target/${jarName} ${docker_path}
sh /root/docker_dir/deploy_docker.sh ${projectName} ${docker_path} ${jarName}

====== deploy_docker.sh
#!/bin/sh
# maven01 $workspace $jarname
# ${projectName} ${docker_path} ${jarName}

set -e
projectName=$1
docker_path=$2
appName=$3

#user_name=
#password=

tag=$(date +%s)
server_path=192.168.56.109:5000
target_image=${projectName}:${tag}
#${BUILD_NUMBER}
echo ${target_image}
cd ${docker_path}
docker build --build-arg app=${appName} -t ${target_image} .
docker tag ${target_image} ${server_path}/${projectName}
echo The name of image is "${server_path}/${target_image}"
docker push ${server_path}/${projectName}:latest

docker rmi -f $(docker images | grep ${projectName} | grep ${tag} | awk '{print $3}' | head -n 1)

==== 设置jenkins服务器到k8s master ssh 免密登录
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub root@192.168.56.108

== 设置构建后操作，将yaml文件拷贝到k8s master上并运行应用
set -e
echo ok
echo ${WORKSPACE}
docker_path=${WORKSPACE}
scp ${WORKSPACE}/*.yaml 192.168.56.108:/root/
ssh 192.168.56.108 '/opt/bin/kubectl apply -f /root/kube.yaml'
ssh 192.168.56.108 '/opt/bin/kubectl get svc|grep maven'

==============
swapoff -a
setenforce 0
systemctl daemon-reload
systemctl restart kubelet