【Kubernetes Series】Kuberadm 方式部署Kubernets集群

# Kubernetes Deploy

本文记载Kubernetes通过kubeadm方式安装的详细过程及遇到的问题与解决方法。

更新于2020-08-02

 

## Kubernetes源

考虑到国内的网络环境,采用 *阿里云* mirrors.aliyun.com 的源.

> https://developer.aliyun.com/mirror/kubernetes

> 下载地址:https://mirrors.aliyun.com/kubernetes/

> 官方主页:https://kubernetes.io/

 

 

aliyun-kubernetes.repo

aliyun-docker-ce.repo

 

yum install docker-ce

yum install -y kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2

 

(1)借助NTP服务设置节点时间精确同步;

(2)通过DNS完成各节点的主机名解析,测试环境主机数量较少时也可以使用hosts文件进行;

(3)关闭各节点的iptables或firewalld服务,并确保它们被禁止随系统引导过程启动;

(4)各节点禁用Selinux(否则在运行容器时可以会遇到各种奇葩报错);

(5)各节点禁用所有的swap设备(生产环境中强烈建议禁用,虽说K8S也支持配置参数来启用swap但这样会降低集群性能,使用"swapoff -a"只是临时关闭交换分区使用,永久关闭需要编辑"/etc/fstab"文件将挂载swap哪一行前面加一个"#"进行注释);

(6)若要使用ipvs模型的proxy,各节点还需要载入ipvs相关的各模块

 

 

## 1. Kubernetes Repo

 

> /etc/yum.repos.d/kubernetes-aliyun.repo

 

```shell

[kubernetes]

name=Kubernetes

baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/

enabled=1

gpgcheck=1

repo_gpgcheck=1

gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

```

 

## 2. Yum install Kubeadm

 

```shell

setenforce 0

yum install -y kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2

yum install -y --nogpgcheck kubelet kubeadm kubectl

systemctl enable kubelet && systemctl start kubelet

 

[root@master01 ~]# kubeadm config images list

W0727 15:53:36.203783 21851 version.go:102] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

W0727 15:53:36.203925 21851 version.go:103] falling back to the local client version: v1.18.2

W0727 15:53:36.204122 21851 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]

k8s.gcr.io/kube-apiserver:v1.18.2

k8s.gcr.io/kube-controller-manager:v1.18.2

k8s.gcr.io/kube-scheduler:v1.18.2

k8s.gcr.io/kube-proxy:v1.18.2

k8s.gcr.io/pause:3.2

k8s.gcr.io/etcd:3.4.3-0

k8s.gcr.io/coredns:1.6.7

```

 

## 3. 调整docker

 

```

docker info

 

cat >/etc/docker/daemon.json <<EOF

{

"exec-opts": ["native.cgroupdriver=systemd"],

"registry-mirrors": ["https://2325obkh.mirror.aliyuncs.com"]

}

EOF

```

 

docker自1.13版起会自动设置iptable的FORWARD默认策略为DROP,这可能会影响kubernetes集群依赖的报文转发功能,因此,需要在docker服务启动后,重新将FORWARD链的默认策略设置为ACCEPT。

 

vim /usr/lib/systemd/system/docker.service"

在"ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock"之后

新增一行"ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT"(意思是docker服务器启动成功后会执行执行该命令)

 

 

```shell

vim /etc/sysconfig/kubelet

KUBELET_EXTRA_ARGS="--fail-swap-on=false"

```

systemctl daemon-reload

 

## 4. 调整系统参数

 

```shell

yum install chrony

chronyc sourcestats -v

swapoff -a

setenforce=0

systemctl stop firewalld.service

```

 

> 加载ipvs模块

 

```shell

vim /etc/sysconfig/modules/ipvs.modules

 

#!/usr/bin/env bash

 

ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs"

 

for mod in $(ls $ipvs_mods_dir | grep -o "^[^.]*");do

/usr/sbin/modinfo -F filename $mod &> /dev/null

if [ $? -eq 0 ];then

/sbin/modprobe $mod

fi

done

 

# master与node上均执行

chmod +x /etc/sysconfig/modules/ipvs.modules

 

lsmod | grep ip_vs

```

 

kubeadm config print init-defaults

 

kubeadm config images list

kubeadm config images pull

 

 

 

 

 

 

docker images

docker image ls

docker image tag old_image:version new_image:version

docker image pull

docker image rm

 

registry.cn-hangzhou.aliyuncs.com__google_containers__kube-proxy__v1.18.2.tar = registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.2 =

registry.cn-hangzhou.aliyuncs.com__google_containers__pause__v3.2.tar = registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 = k8s.gcr.io/pause:3.2

registry.aliyuncs.com__google_containers__etcd__v3.4.3-0.tar = registry.aliyuncs.com/google_containers/etcd:3.4.3-0 = k8s.gcr.io/etcd:3.4.3-0

registry.cn-hangzhou.aliyuncs.com__google_containers__coredns__v1.6.7.tar = registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.7 = k8s.gcr.io/coredns:1.6.7

 

docker save registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy -o registry.cn-hangzhou.aliyuncs.com__google_containers__kube-proxy__v1.18.2.tar

docker save registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver -o registry.cn-hangzhou.aliyuncs.com__google_containers__kube-apiserver__v1.18.2.tar

docker save registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager -o registry.cn-hangzhou.aliyuncs.com__google_containers__kube-controller-manager__v1.18.2.tar

docker save registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler -o registry.cn-hangzhou.aliyuncs.com__google_containers__kube-scheduler__v1.18.2.tar

docker save registry.cn-hangzhou.aliyuncs.com/google_containers/pause -o registry.cn-hangzhou.aliyuncs.com__google_containers__pause__v3.2.tar

docker save registry.aliyuncs.com/google_containers/etcd -o registry.aliyuncs.com__google_containers__etcd__v3.4.3-0.tar

docker save registry.cn-hangzhou.aliyuncs.com/google_containers/coredns -o registry.cn-hangzhou.aliyuncs.com__google_containers__coredns__v1.6.7.tar

 

docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.2 k8s.gcr.io/kube-proxy:v1.18.2

docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.18.2 k8s.gcr.io/kube-apiserver:v1.18.2

docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.18.2 k8s.gcr.io/kube-controller-manager:v1.18.2

docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.18.2 k8s.gcr.io/kube-scheduler:v1.18.2

docker image tag registry.aliyuncs.com/google_containers/pause:3.2 k8s.gcr.io/pause:3.2

docker image tag registry.aliyuncs.com/google_containers/etcd:3.4.3-0 k8s.gcr.io/etcd:3.4.3-0

docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.7 k8s.gcr.io/coredns:1.6.7

 

[root@master01 ~]# docker image ls

REPOSITORY TAG IMAGE ID CREATED SIZE

k8s.gcr.io/kube-proxy v1.18.2 0d40868643c6 3 months ago 117MB

k8s.gcr.io/kube-controller-manager v1.18.2 ace0a8c17ba9 3 months ago 162MB

k8s.gcr.io/kube-scheduler v1.18.2 a3099161e137 3 months ago 95.3MB

k8s.gcr.io/kube-apiserver v1.18.2 6ed75ad404bd 3 months ago 173MB

k8s.gcr.io/pause 3.2 80d28bedfe5d 5 months ago 683kB

k8s.gcr.io/coredns 1.6.7 67da37a9a360 6 months ago 43.8MB

k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90 9 months ago 288MB

 

> 初始化CLuster

 

>> 初始测试dry-run

 

kubeadm init \

--apiserver-advertise-address=192.168.221.134 \

--image-repository registry.aliyuncs.com/google_containers \

--kubernetes-version v1.18.2 \

--pod-network-cidr=10.244.0.0/16 \

--dry-run

 

>> init cluster

 

kubeadm init \

--apiserver-advertise-address=192.168.221.134 \

--image-repository registry.aliyuncs.com/google_containers \

--kubernetes-version v1.18.2 \

--pod-network-cidr=10.244.0.0/16

 

```shell

[root@master01 tmp]# kubeadm init --apiserver-advertise-address=192.168.221.134 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.18.2 --pod-network-cidr=10.244.0.0/16

W0727 16:40:13.862004 32012 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]

[init] Using Kubernetes version: v1.18.2

[preflight] Running pre-flight checks

[preflight] Pulling images required for setting up a Kubernetes cluster

[preflight] This might take a minute or two, depending on the speed of your internet connection

[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'

[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"

[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"

[kubelet-start] Starting the kubelet

[certs] Using certificateDir folder "/etc/kubernetes/pki"

[certs] Generating "ca" certificate and key

[certs] Generating "apiserver" certificate and key

[certs] apiserver serving cert is signed for DNS names [master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.221.134]

[certs] Generating "apiserver-kubelet-client" certificate and key

[certs] Generating "front-proxy-ca" certificate and key

[certs] Generating "front-proxy-client" certificate and key

[certs] Generating "etcd/ca" certificate and key

[certs] Generating "etcd/server" certificate and key

[certs] etcd/server serving cert is signed for DNS names [master01 localhost] and IPs [192.168.221.134 127.0.0.1 ::1]

[certs] Generating "etcd/peer" certificate and key

[certs] etcd/peer serving cert is signed for DNS names [master01 localhost] and IPs [192.168.221.134 127.0.0.1 ::1]

[certs] Generating "etcd/healthcheck-client" certificate and key

[certs] Generating "apiserver-etcd-client" certificate and key

[certs] Generating "sa" key and public key

[kubeconfig] Using kubeconfig folder "/etc/kubernetes"

[kubeconfig] Writing "admin.conf" kubeconfig file

[kubeconfig] Writing "kubelet.conf" kubeconfig file

[kubeconfig] Writing "controller-manager.conf" kubeconfig file

[kubeconfig] Writing "scheduler.conf" kubeconfig file

[control-plane] Using manifest folder "/etc/kubernetes/manifests"

[control-plane] Creating static Pod manifest for "kube-apiserver"

[control-plane] Creating static Pod manifest for "kube-controller-manager"

W0727 16:40:22.595129 32012 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"

[control-plane] Creating static Pod manifest for "kube-scheduler"

W0727 16:40:22.596079 32012 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"

[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s

[apiclient] All control plane components are healthy after 24.507077 seconds

[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace

[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster

[upload-certs] Skipping phase. Please see --upload-certs

[mark-control-plane] Marking the node master01 as control-plane by adding the label "node-role.kubernetes.io/master=''"

[mark-control-plane] Marking the node master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

[bootstrap-token] Using token: a4imyy.ttddd23g8ddh7dv7

[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles

[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes

[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials

[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token

[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster

[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace

[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key

[addons] Applied essential addon: CoreDNS

[addons] Applied essential addon: kube-proxy

 

Your Kubernetes control-plane has initialized successfully!

 

To start using your cluster, you need to run the following as a regular user:

 

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

 

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

https://kubernetes.io/docs/concepts/cluster-administration/addons/

 

Then you can join any number of worker nodes by running the following on each as root:

 

kubeadm join 192.168.221.134:6443 --token a4imyy.ttddd23g8ddh7dv7 \

--discovery-token-ca-cert-hash sha256:4f784226e480a6de709b1f21e6e67ec02050942e634968002c1b86e670db9b0c

[root@master01 tmp]#

```

 

 

> 检查kubelet.service 状态

 

```shell

[root@master01 ~]# systemctl status kubelet -l

● kubelet.service - kubelet: The Kubernetes Node Agent

Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)

Drop-In: /usr/lib/systemd/system/kubelet.service.d

└─10-kubeadm.conf

Active: active (running) since Mon 2020-07-27 16:40:53 CST; 1min 25s ago

Docs: https://kubernetes.io/docs/

Main PID: 33413 (kubelet)

Tasks: 16

Memory: 74.7M

CGroup: /system.slice/kubelet.service

└─33413 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2 --fail-swap-on=false

 

Jul 27 16:41:54 master01 kubelet[33413]: E0727 16:41:54.116034 33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Jul 27 16:41:55 master01 kubelet[33413]: W0727 16:41:55.025914 33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d

Jul 27 16:41:59 master01 kubelet[33413]: E0727 16:41:59.233911 33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Jul 27 16:42:00 master01 kubelet[33413]: W0727 16:42:00.028419 33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d

Jul 27 16:42:04 master01 kubelet[33413]: E0727 16:42:04.354686 33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Jul 27 16:42:05 master01 kubelet[33413]: W0727 16:42:05.029246 33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d

Jul 27 16:42:09 master01 kubelet[33413]: E0727 16:42:09.444976 33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Jul 27 16:42:10 master01 kubelet[33413]: W0727 16:42:10.048128 33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d

Jul 27 16:42:14 master01 kubelet[33413]: E0727 16:42:14.551861 33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Jul 27 16:42:15 master01 kubelet[33413]: W0727 16:42:15.049274 33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d

[root@master01 ~]#

 

 

## 部署Network Plugin

 

mkdir .kube

cp /etc/kubernetes/admin.conf .kube/config

 

```shell

[root@master01 ~]# kubectl get nodes

NAME STATUS ROLES AGE VERSION

master01 NotReady master 11m v1.18.2

# 添加一个node后

[root@master01 ~]# kubectl get nodes

NAME STATUS ROLES AGE VERSION

centos77minimal NotReady <none> 41s v1.18.2

master01 NotReady master 22m v1.18.2

```

 

node 添加ipvs.modules

node 导入k8s-image.tar

cp master:/etc/sysconfig/kubelet node

 

 

 

kubeadm reset

kubeadm token create --print-join-command

kubectl get pods -n kube-system

 

# 查看pod产生的日志

kubectl logs -f calico-node-5rgfg -n kube-system

 

kubectl describe pod calico-node-5rgfg -n kube-system 查看pod的具体描述

 

kubectl -n kube-system get ev 查看集群环境输出

 

 

 

## flannel

sysctl net.bridge.bridge-nf-call-iptables=1

上一篇:laravel 控制器调用命令传参


下一篇:阿里云apt-get安装包时Err:2 http://mirrors.cloud.aliyuncs.com/ubuntu xenial-security/main amd64 git amd64 1: