一、初始化
参考链接：https://kubernetes.io/docs/reference/setup-tools/kubeadm/
1.1、配置要求
- 节点配置为2c4g+
- 集群中所有机器之间互通
- hostname 和mac地址集群内唯一
- CentOS版本为7，最好是7.9，低版本的建议更新下
- 集群中所有节点可以访问外网拉取镜像
- swap分区关闭
1.2、初始化
1.所有主机上都要操作
systemctop disable firewalld;systemctl stop firewalld
swapoff -a ;sed -i '/swap/s/^/#/g' /etc/fstab
setenforce 0;sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
2.主机名初始化
hostnamectl set-hostname $主机名 #不同的主机山设置不同的主机名
[root@master1 $]# vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.74.128 master1 registry.mt.com
192.168.74.129 master2
192.168.74.130 master3
3、内核初始化-所有主机
[root@master1 yaml]# cat /etc/sysctl.d/kubernetes.conf
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0 # 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它
vm.overcommit_memory=1 # 不检查物理内存是否够用
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=655360000
fs.nr_open=655360000
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
[root@master1 yaml]# sysctl --system
4、时间同步
ntpdate time.windows.com
二、安装
2.1、安装docker
# 在所有节点上执行：
wget -O /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
yum install docker-ce-18.06.1-3.el7
# 修改/etc/docker/docker-daemon.json配置文件
{
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"],
"insecure-registries": ["www.mt.com:9500"]
}
说明：
registry-mirrors 为aliyun镜像加速设置的地址，用于加速下载国外的镜像。不然下载会很慢
#修改完成后，docker info验证
systemctl disbale docker;systemctl enable docker
[root@master1 $]# docker info |tail -10
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
www.mt.com:9500
127.0.0.0/8
Registry Mirrors:
https://b9pmyelo.mirror.aliyuncs.com/
Live Restore Enabled: false
2.2、安装kubectl/kubeadm/kubelet
#创建repo文件 /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
#baseurl=https://mirrors.tuna.tsinghua.edu.cn/kubernetes/yum/repos/kubernetes-el7-$basearch
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
#安装
yum install kubeadm-1.18.0 kubectl-1.18.0 kubelet-1.18.0
#启动服务
systemctl start kubelet;systemctl enable kubelet
2.3、初始化
#只在master上运行
[root@master1 ~]# kubeadm init --apiserver-advertise-address=192.168.74.128 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.18.0 --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16
W0503 16:48:24.338256 38262 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.74.128]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master1 localhost] and IPs [192.168.74.128 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master1 localhost] and IPs [192.168.74.128 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0503 16:49:05.984728 38262 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0503 16:49:05.985466 38262 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 17.002641 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: i91fyq.rp8jv1t086utyxif
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
# 创建kubectl访问apiserver 配置文件。在master上执行
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 创建network
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
#加入集群使用
kubeadm join 192.168.74.128:6443 --token i91fyq.rp8jv1t086utyxif \
--discovery-token-ca-cert-hash sha256:a31d6b38fec404eda854586a4140069bc6e3154241b59f40a612e24e9b89bf37
[root@master1 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/google_containers/kube-proxy v1.18.0 43940c34f24f 13 months ago 117MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.18.0 74060cea7f70 13 months ago 173MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.18.0 d3e55153f52f 13 months ago 162MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.18.0 a31f78c7c8ce 13 months ago 95.3MB
registry.aliyuncs.com/google_containers/pause 3.2 80d28bedfe5d 14 months ago 683kB
registry.aliyuncs.com/google_containers/coredns 1.6.7 67da37a9a360 15 months ago 43.8MB
registry.aliyuncs.com/google_containers/etcd 3.4.3-0 303ce5db0e90 18 months ago 288MB
k8s.gcr.io/etcd 3.2.24 3cab8e1b9802 2 years ago 220MB
k8s.gcr.io/coredns 1.2.2 367cdc8433a4 2 years ago 39.2MB
k8s.gcr.io/kubernetes-dashboard-amd64 v1.10.0 0dab2435c100 2 years ago 122MB
k8s.gcr.io/flannel v0.10.0-amd64 f0fad859c909 3 years ago 44.6MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 3 years ago 742kB
[root@master1 ~]#
# 创建kubectl配置文件：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
2.4、加入集群
[root@master1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 NotReady master 36m v1.18.0
# node1和node2上都执行
kubeadm join 192.168.74.128:6443 --token i91fyq.rp8jv1t086utyxif --discovery-token-ca-cert-hash sha256:a31d6b38fec404eda854586a4140069bc6e3154241b59f40a612e24e9b89bf37
[root@master1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 NotReady master 40m v1.18.0
master2 NotReady <none> 78s v1.18.0
master3 NotReady <none> 31s v1.18.0
2.5、安装网络插件
[root@master1 yaml]# kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
[root@master1 yaml]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-l9h6x 1/1 Running 0 60m
coredns-7ff77c879f-lp9p2 1/1 Running 0 60m
etcd-master1 1/1 Running 0 60m
kube-apiserver-master1 1/1 Running 0 60m
kube-controller-manager-master1 1/1 Running 0 60m
kube-flannel-ds-524n6 1/1 Running 0 45s
kube-flannel-ds-8gqwp 1/1 Running 0 45s
kube-flannel-ds-fgvpf 1/1 Running 0 45s
kube-proxy-2fz26 1/1 Running 0 21m
kube-proxy-46psf 1/1 Running 0 20m
kube-proxy-p7jnd 1/1 Running 0 60m
kube-scheduler-master1 1/1 Running 0 60m
[root@master1 yaml]# kubectl get nodes #在安装网络插件之前STATUS都是NotReady状态
NAME STATUS ROLES AGE VERSION
master1 Ready master 60m v1.18.0
master2 Ready <none> 21m v1.18.0
master3 Ready <none> 20m v1.18.0
三、验证
[root@master1 yaml]# kubectl create deployment nginx --image=nginx
[root@master1 yaml]# kubectl expose deployment nginx --port=80 --type=NodePort
[root@master1 yaml]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-f89759699-fvlfg 1/1 Running 0 99s 10.244.2.3 master3 <none> <none>
[root@master1 yaml]# kubectl get svc/nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx NodePort 10.105.249.76 <none> 80:30326/TCP 85s
[root@master1 yaml]# curl -s -I master2:30326 #访问集群中任意一个node节点的 30326端口都可以通
HTTP/1.1 200 OK
Server: nginx/1.19.10
Date: Mon, 03 May 2021 09:58:47 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 13 Apr 2021 15:13:59 GMT
Connection: keep-alive
ETag: "6075b537-264"
Accept-Ranges: bytes
[root@master1 yaml]# curl -s -I master3:30326
HTTP/1.1 200 OK
Server: nginx/1.19.10
Date: Mon, 03 May 2021 09:58:50 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 13 Apr 2021 15:13:59 GMT
Connection: keep-alive
ETag: "6075b537-264"
Accept-Ranges: bytes
[root@master1 yaml]# curl -s -I master1:30326 #master1为master节点非worker节点，
^C
镜像：
[root@master1 yaml]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.14.0-rc1 0a1a2818ce59 2 weeks ago 67.9MB
registry.aliyuncs.com/google_containers/kube-proxy v1.18.0 43940c34f24f 13 months ago 117MB
registry.aliyuncs.com/google_containers/kube-controller-manager v1.18.0 d3e55153f52f 13 months ago 162MB
registry.aliyuncs.com/google_containers/kube-apiserver v1.18.0 74060cea7f70 13 months ago 173MB
registry.aliyuncs.com/google_containers/kube-scheduler v1.18.0 a31f78c7c8ce 13 months ago 95.3MB
registry.aliyuncs.com/google_containers/pause 3.2 80d28bedfe5d 14 months ago 683kB
registry.aliyuncs.com/google_containers/coredns 1.6.7 67da37a9a360 15 months ago 43.8MB
registry.aliyuncs.com/google_containers/etcd 3.4.3-0 303ce5db0e90 18 months ago 288MB
[root@master2 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.14.0-rc1 0a1a2818ce59 2 weeks ago 67.9MB
registry.aliyuncs.com/google_containers/kube-proxy v1.18.0 43940c34f24f 13 months ago 117MB
registry.aliyuncs.com/google_containers/pause 3.2 80d28bedfe5d 14 months ago 683kB
registry.aliyuncs.com/google_containers/coredns 1.6.7 67da37a9a360 15 months ago 43.8MB
[root@master3 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.14.0-rc1 0a1a2818ce59 2 weeks ago 67.9MB
nginx latest 62d49f9bab67 2 weeks ago 133MB
registry.aliyuncs.com/google_containers/kube-proxy v1.18.0 43940c34f24f 13 months ago 117MB
registry.aliyuncs.com/google_containers/pause 3.2 80d28bedfe5d 14 months ago 683kB
registry.aliyuncs.com/google_containers/coredns 1.6.7 67da37a9a360 15 months ago 43.8MB