CentOS7.6配置do.cker和K.B.S

2023-08-16 15:48:34

 方法一：

节点及功能

主机名

IP

Master、etcd、registry

K8s-01

10.8.8.31

Node1

K8s-02

10.8.8.32

Node2

K8s-03

10.8.8.33

一：环境搭建：（各节点均需配置）

　　1.1：新建虚拟机（最好重新安装，不要用镜像文件）

　　　　编辑网卡文件：vi /etc/sysconfig/network-scripts/ifcfg-ens33

TYPE=Ethernet

PROXY_METHOD=none

BROWSER_ONLY=no

#BOOTPROTO=dhcp

BOOTPROTO=static

DEFROUTE=yes

IPV4_FAILURE_FATAL=no

IPV6INIT=yes

IPV6_AUTOCONF=yes

IPV6_DEFROUTE=yes

IPV6_FAILURE_FATAL=no

IPV6_ADDR_GEN_MODE=stable-privacy

NAME=ens33

#UUID=1bc6ef33-bdb7-4f3d-8021-b138426828ed

DEVICE=ens33

#ONBOOT=no

ONBOOT=yes

IPADDR=10.8.8.31

NETMASK=255.255.255.0

GATEWAY=10.8.8.2

DNS1=8.8.8.8

DNS2=1.1.1.1

　　

　　1.2：修改机器名

　　　　hostnamectl set-hostname k8s-01

　　1.3：安装ansible

　　　　yum install -y ansible

　　　　vi /etc/ansible/hosts

# This is the default ansible 'hosts' file.

#

# It should live in /etc/ansible/hosts

#

#   - Comments begin with the '#' character

#   - Blank lines are ignored

#   - Groups of hosts are delimited by [header] elements

#   - You can enter hostnames or ip addresses

#   - A hostname/ip can be a member of multiple groups

[k8s]

10.8.8.31

10.8.8.32

10.8.8.33

[master]

10.8.8.31

[node]

10.8.8.32

10.8.8.33

　　1.4：配置ssh互信，所有结点都要运行

　　　　ssh-keygen -t rsa

[root@localhost ~]# ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:NFGl8BmAOW6ch93oiRuBLzNS1jY5dcIU6bGpwLUyUeQ root@k8s-01

The key's randomart image is:

+---[RSA 2048]----+

|     oo=*=o..    |

|    ..= *+.+     |

|   . BEXoO+      |

|    O #.B..      |

|   o B OS.       |

|  . + = o        |

|   . + o         |

|      .          |

|                 |

+----[SHA256]-----+

　

　　　　ssh-copy-id -i /root/.ssh/id_rsa.pub 10.8.8.31

　　　　ssh-copy-id -i /root/.ssh/id_rsa.pub 10.8.8.32

　　　　ssh-copy-id -i /root/.ssh/id_rsa.pub 10.8.8.33

[root@localhost .ssh]# ssh-copy-id -i /root/.ssh/id_rsa.pub 10.8.8.33

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"

The authenticity of host '10.8.8.33 (10.8.8.33)' can't be established.

ECDSA key fingerprint is SHA256:ozAbIXZWFBIwjiypTD23hQ9ioBr81+MZd1TGCQcc0o8.

ECDSA key fingerprint is MD5:9d:0c:48:4f:c4:50:7c:08:71:33:9e:86:13:46:b3:12.

Are you sure you want to continue connecting (yes/no)? yes

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

root@10.8.8.33's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '10.8.8.33'"

and check to make sure that only the key(s) you wanted were added.

　　

　　1.5：用ansible重启集群

　　　　ansible all -a 'reboot'

　　

　　1.6：ansible all -a 'yum update -y

　　　　  ansible all -a 'yum install -y net-tools.x86_64'

　　　　  ansible all -a 'yum install -y vim-enhanced.x86_64'

　　　　 ansible all -a 'yum install -y wget'

　　　　  ansible all -a 'yum install -y tree'

　　　　  ansible all -a 'yum install -y ntp ntpdate'

　　　　  echo '*/10 *  *  *  * root    ntpdate cn.pool.ntp.org' >> /etc/crontab

　　　　  ansible all -a 'ntpdate cn.pool.ntp.org'

　　　　  编辑hosts文件，并分发到各节点

　　　　　　vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

10.8.8.31       k8s-01

10.8.8.32       k8s-02

10.8.8.33       k8s-03

　　　　　　scp /etc/hosts root@10.8.8.32:/etc/

　　1.7：关闭防火墙

　　　　　　[root@k8s-01 ~]# ansible all -a 'systemctl stop firewalld'

　　　　　　[root@k8s-01 ~]# ansible all -a 'systemctl disable firewalld'

　　　　　　[root@k8s-01 ~]# ansible all -a 'systemctl mask firewalld'

　　1.8：关闭selinux

https://www.cnblogs.com/liwei0526vip/p/5644163.html  （sed用法）

　　　　　　getenforce

　　　　　　vim /etc/selinux/config

　　　　　　SELINUX=disabled

　　　　　　ansible all -a "sed -i '7s/.*/#&/' /etc/selinux/config"

　　　　　　ansible all -a "sed -i '7a SELINUX=disabled' /etc/selinux/config"

　　1.9：关闭交换分区swap

　　　　swapoff -a

　　　　rm /dev/mapper/centos-swap

　　　　sed -i 's/.*swap.*/#&/' /etc/fstab

　　　　ansible all -a 'swapoff -a'

　　　　ansible all -a 'rm /dev/mapper/centos-swap'

　　　　ansible all -a "sed -i 's/.*swap.*/#&/' /etc/fstab"

　　　　ansible all -a 'reboot'

二：部署etcd（yum方法）

https://blog.csdn.net/xiaozhangdetuzi/article/details/81302405

https://www.jianshu.com/p/e892997b387b

　　2.1：所有节点安装etcd

　　　　ansible all -a 'yum install -y etcd'

　　2.2：配置etcd.conf

　　　　vim /etc/etcd/etcd.conf

[root@k8s-01 ~]# vim /etc/etcd/etcd.conf (原文件)

#[Member]

#ETCD_CORS=""

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

#ETCD_WAL_DIR=""

#ETCD_LISTEN_PEER_URLS="http://localhost:2380"

ETCD_LISTEN_CLIENT_URLS="http://localhost:2379"

#ETCD_MAX_SNAPSHOTS="5"

#ETCD_MAX_WALS="5"

ETCD_NAME="default"

#ETCD_SNAPSHOT_COUNT="100000"

#ETCD_HEARTBEAT_INTERVAL="100"

#ETCD_ELECTION_TIMEOUT="1000"

#ETCD_QUOTA_BACKEND_BYTES="0"

#ETCD_MAX_REQUEST_BYTES="1572864"

#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"

#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"

#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"

#

#[Clustering]

#ETCD_INITIAL_ADVERTISE_PEER_URLS="http://localhost:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379"

#ETCD_DISCOVERY=""

#ETCD_DISCOVERY_FALLBACK="proxy"

#ETCD_DISCOVERY_PROXY=""

#ETCD_DISCOVERY_SRV=""

#ETCD_INITIAL_CLUSTER="default=http://localhost:2380"

#ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

#ETCD_INITIAL_CLUSTER_STATE="new"

#ETCD_STRICT_RECONFIG_CHECK="true"

#ETCD_ENABLE_V2="true"

#

#[Proxy]

#ETCD_PROXY="off"

#ETCD_PROXY_FAILURE_WAIT="5000"

#ETCD_PROXY_REFRESH_INTERVAL="30000"

#ETCD_PROXY_DIAL_TIMEOUT="1000"

#ETCD_PROXY_WRITE_TIMEOUT="5000"

#ETCD_PROXY_READ_TIMEOUT="0"

#

#[Security]

#ETCD_CERT_FILE=""

#ETCD_KEY_FILE=""

#ETCD_CLIENT_CERT_AUTH="false"

#ETCD_TRUSTED_CA_FILE=""

#ETCD_AUTO_TLS="false"

#ETCD_PEER_CERT_FILE=""

#ETCD_PEER_KEY_FILE=""

#ETCD_PEER_CLIENT_CERT_AUTH="false"

#ETCD_PEER_TRUSTED_CA_FILE=""

#ETCD_PEER_AUTO_TLS="false"

#

#[Logging]

#ETCD_DEBUG="false"

#ETCD_LOG_PACKAGE_LEVELS=""

#ETCD_LOG_OUTPUT="default"

#

#[Unsafe]

#ETCD_FORCE_NEW_CLUSTER="false"

#

#[Version]

#ETCD_VERSION="false"

#ETCD_AUTO_COMPACTION_RETENTION="0"

#

#[Profiling]

#ETCD_ENABLE_PPROF="false"

#ETCD_METRICS="basic"

#

#[Auth]

#ETCD_AUTH_TOKEN="simple"

　　　　　　k8s-01

[root@k8s-01 ~]# vim /etc/etcd/etcd.conf

#[Member]
#节点名称
ETCD_NAME="k8s-01"
#指定节点的数据存储目录
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#监听URL，用于与其他节点通讯
ETCD_LISTEN_PEER_URLS="http://10.8.8.31:2380"
#对外提供服务的地址,客户端会连接到这里和 etcd 交互
ETCD_LISTEN_CLIENT_URLS="http://10.8.8.31:2379,http://127.0.0.1:2379"

#[Clustering]
#该节点同伴监听地址，这个值会告诉集群中其他节点
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.8.8.31:2380"
#对外公告的该节点客户端监听地址，这个值会告诉集群中其他节点
ETCD_ADVERTISE_CLIENT_URLS="http://10.8.8.31:2379"
#集群中所有节点的信息,
#格式为 node1=http://ip1:2380,node2=http://ip2:2380,… 
#注意：这里的 node1 是节点的 --name 指定的名字；后面的 ip1:2380 是 --initial-advertise-peer-urls 指定的值
ETCD_INITIAL_CLUSTER="k8s-01=http://10.8.8.31:2380,k8s-02=http://10.8.8.32:2380,k8s-03=http://10.8.8.33:2380"
#创建集群的 token，这个值每个集群保持唯一。
#这样的话，如果你要重新创建集群，即使配置和之前一样，也会再次生成新的集群和节点 uuid；否则会导致多个集群之间的冲突，造成未知的错误
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
#新建集群的时候，这个值为 new ；假如已经存在的集群，这个值为 existing
ETCD_INITIAL_CLUSTER_STATE="new" 

　　　　　　k8s-02

#[Member]

ETCD_NAME="k8s-02"

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_PEER_URLS="http://10.8.8.32:2380"

ETCD_LISTEN_CLIENT_URLS="http://10.8.8.32:2379,http://127.0.0.1:2379"

#[Clustering]

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.8.8.32:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://10.8.8.32:2379"

ETCD_INITIAL_CLUSTER="k8s-01=http://10.8.8.31:2380,k8s-02=http://10.8.8.32:2380,k8s-03=http://10.8.8.33:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_INITIAL_CLUSTER_STATE="new"

　　2.3：etcd集群，

　　　　所有节点分别运行

　　　　　　systemctl start etcd

　　　　或ansible运行

　　　　　　ansible all -a 'systemctl start etcd'

　　　　检测启动情况：

　　　　　　 etcdctl member list

[root@k8s-01 etcd]# etcdctl member list

21a69e29ab8d1218: name=k8s-02 peerURLs=http://10.8.8.32:2380 clientURLs=http://10.8.8.32:2379 isLeader=true

3df47f4e2d43b21a: name=k8s-03 peerURLs=http://10.8.8.33:2380 clientURLs=http://10.8.8.33:2379 isLeader=false

5b118d787e1ab5d3: name=k8s-01 peerURLs=http://10.8.8.31:2380 clientURLs=http://10.8.8.31:2379 isLeader=false

　　　　　　k8s-02 isLeader=true 为主节点

　　　　　　etcdctl -C http://10.8.8.31:2379 cluster-health

[root@k8s-01 etcd]# etcdctl -C http://10.8.8.31:2379 cluster-health

member 21a69e29ab8d1218 is healthy: got healthy result from http://10.8.8.32:2379

member 3df47f4e2d43b21a is healthy: got healthy result from http://10.8.8.33:2379

member 5b118d787e1ab5d3 is healthy: got healthy result from http://10.8.8.31:2379

cluster is healthy

　　　　设置开机启动：

　　　　　　ansible all -a 'systemctl enable etcd'

 三：安装docker

　　3.1：安装docker yum （各节点都要安装）

　　　　ansible all -a 'yum install -y docker'

　　　　ansible all -a 'docker version'　　（报错如下）

[root@k8s-01 etcd]# ansible all -a 'docker version'

10.8.8.31 | FAILED | rc=1 >>

Client:

 Version:         1.13.1

 API version:     1.26

 Package version: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?non-zero return code

10.8.8.32 | FAILED | rc=1 >>

Client:

 Version:         1.13.1

 API version:     1.26

 Package version: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?non-zero return code

10.8.8.33 | FAILED | rc=1 >>

Client:

 Version:         1.13.1

 API version:     1.26

 Package version: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?non-zero return code

　　　　　　ansible all -a 'systemctl daemon-reload'

　　　　　　ansible all -a 'systemctl restart docker'

　　　　　    ansible all -a 'docker version'

[root@k8s-01 etcd]# ansible all -a 'docker version'

10.8.8.31 | SUCCESS | rc=0 >>

Client:

 Version:         1.13.1

 API version:     1.26

 Package version: docker-1.13.1-91.git07f3374.el7.centos.x86_64

 Go version:      go1.10.3

 Git commit:      07f3374/1.13.1

 Built:           Wed Feb 13 17:10:12 2019

 OS/Arch:         linux/amd64

Server:

 Version:         1.13.1

 API version:     1.26 (minimum version 1.12)

 Package version: docker-1.13.1-91.git07f3374.el7.centos.x86_64

 Go version:      go1.10.3

 Git commit:      07f3374/1.13.1

 Built:           Wed Feb 13 17:10:12 2019

 OS/Arch:         linux/amd64

 Experimental:    false

10.8.8.33 | SUCCESS | rc=0 >>

Client:

 Version:         1.13.1

 API version:     1.26

 Package version: docker-1.13.1-91.git07f3374.el7.centos.x86_64

 Go version:      go1.10.3

 Git commit:      07f3374/1.13.1

 Built:           Wed Feb 13 17:10:12 2019

 OS/Arch:         linux/amd64

Server:

 Version:         1.13.1

 API version:     1.26 (minimum version 1.12)

 Package version: docker-1.13.1-91.git07f3374.el7.centos.x86_64

 Go version:      go1.10.3

 Git commit:      07f3374/1.13.1

 Built:           Wed Feb 13 17:10:12 2019

 OS/Arch:         linux/amd64

 Experimental:    false

10.8.8.32 | SUCCESS | rc=0 >>

Client:

 Version:         1.13.1

 API version:     1.26

 Package version: docker-1.13.1-91.git07f3374.el7.centos.x86_64

 Go version:      go1.10.3

 Git commit:      07f3374/1.13.1

 Built:           Wed Feb 13 17:10:12 2019

 OS/Arch:         linux/amd64

Server:

 Version:         1.13.1

 API version:     1.26 (minimum version 1.12)

 Package version: docker-1.13.1-91.git07f3374.el7.centos.x86_64

 Go version:      go1.10.3

 Git commit:      07f3374/1.13.1

 Built:           Wed Feb 13 17:10:12 2019

 OS/Arch:         linux/amd64

 Experimental:    false

　　　　　　　ifconfig

[root@k8s-01 etcd]# ifconfig

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500

        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0

        ether 02:42:7f:71:21:01  txqueuelen 0  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 0  bytes 0 (0.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 10.8.8.31  netmask 255.255.255.0  broadcast 10.8.8.255

        inet6 fe80::4e95:1400:1371:99a4  prefixlen 64  scopeid 0x20<link>

        ether 00:0c:29:0b:69:ff  txqueuelen 1000  (Ethernet)

        RX packets 83459  bytes 43293262 (41.2 MiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 60528  bytes 7960462 (7.5 MiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

        inet 127.0.0.1  netmask 255.0.0.0

        inet6 ::1  prefixlen 128  scopeid 0x10<host>

        loop  txqueuelen 1000  (Local Loopback)

        RX packets 1358  bytes 731784 (714.6 KiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 1358  bytes 731784 (714.6 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

　　　　3.2：设置开机启动：

　　　　　　ansible all -a 'systemctl enable docker'

四：安装kubernetes

　　4.1：安装kubernetes（各节点都要安装）

　　　　ansible all -a 'yum install -y kubernetes'

　　4.2：kubernetes master上要运行以下组件　　　

　　　　Kubernets API Server

　　　　Kubernets Controller Manager

　　　　Kubernets Scheduler

　　

　　4.3：配置并启动master：（在master机器编辑）

　　　　4.3.1：apiserver

https://segmentfault.com/a/1190000002920092

　　　　　　vim /etc/kubernetes/apiserver

　　　　　　apiserver原文件：

[root@k8s-01 ~]# vim /etc/kubernetes/apiserver

（原文件）

###

# kubernetes system config

#

# The following values are used to configure the kube-apiserver

#

# The address on the local server to listen to.

KUBE_API_ADDRESS="--insecure-bind-address=127.0.0.1"

# The port on the local server to listen on.

# KUBE_API_PORT="--port=8080"

# Port minions listen on

# KUBELET_PORT="--kubelet-port=10250"

# Comma separated list of nodes in the etcd cluster

KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"

# Address range to use for services

KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

# default admission control policies

KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"

# Add your own!

KUBE_API_ARGS=""

　　　　　　apiserver修改后文件：

[root@k8s-01 ~]# vim /etc/kubernetes/apiserver 

###

# kubernetes system config

#

# The following values are used to configure the kube-apiserver

#

# The address on the local server to listen to.
# –insecure-bind-address： apiserver绑定主机的非安全端口，设置0.0.0.0表示绑定所有IP地址

KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"

# The port on the local server to listen on.
# –insecure-port: apiserver绑定主机的非安全端口号，默认为8080

KUBE_API_PORT="--port=8080"

# Port minions listen on

# KUBELET_PORT="--kubelet-port=10250"

# Comma separated list of nodes in the etcd cluster

KUBE_ETCD_SERVERS="--etcd-servers=http://10.8.8.31:2379,http://10.8.8.32:2379,http://10.8.8.33:2379"

# Address range to use for services
# –service-cluster-ip-range: Kubernetes集群中service的虚拟IP地址范围，以CIDR表示，该IP范围不能与物理机的真实IP段有重合
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

# default admission control policies
# –admission_control: kubernetes集群的准入控制设置，各控制模块以插件的形式依次生效
# NamespaceExists它会观察所有的请求，如果请求尝试创建一个不存在的namespace，则这个请求被拒绝
# LimitRanger他会观察所有的请求，确保没有违反已经定义好的约束条件，这些条件定义在namespace中LimitRange对象中
#   如果在kubernetes中使用LimitRange对象，则必须使用这个插件
# SecurityContextDeny这个插件将会将使用了 SecurityContext的pod中定义的选项全部失效
# serviceAccount为运行在pod内的进程添加了相应的认证信息
# ResourceQuota它会观察所有的请求，确保在namespace中ResourceQuota对象处列举的container没有任何异常，
#   如果在kubernetes中使用了ResourceQuota对象，就必须使用这个插件来约束container，
#   推荐在admission control参数列表中，这个插件排最后一个！

#KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"

# serviceAccount为运行在pod内的进程添加了相应的认证信息（因此例未做认证，所以此项取消）
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"

# Add your own!

KUBE_API_ARGS=""

　　　　4.3.2：config

　　　　　　vim /etc/kubernetes/config

　　　　　　config原文件：

[root@k8s-01 ~]# vim /etc/kubernetes/config 

###

# kubernetes system config

#

# The following values are used to configure various aspects of all

# kubernetes services, including

#

#   kube-apiserver.service

#   kube-controller-manager.service

#   kube-scheduler.service

#   kubelet.service

#   kube-proxy.service

# logging to stderr means we get it in the systemd journal

KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug

KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers

KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver

KUBE_MASTER="--master=http://127.0.0.1:8080"

　　　　config修改后文件：

[root@k8s-01 ~]# vim /etc/kubernetes/config 

###

# kubernetes system config

#

# The following values are used to configure various aspects of all

# kubernetes services, including

#

#   kube-apiserver.service

#   kube-controller-manager.service

#   kube-scheduler.service

#   kubelet.service

#   kube-proxy.service

# logging to stderr means we get it in the systemd journal

KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug

KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers

KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver

KUBE_MASTER="--master=http://10.8.8.31:8080"

　　　　　　4.3.3：master节点启动服务并设置开机启动

　　　　　　　　　　[root@k8s-01 ~]# systemctl start kube-apiserver

　　　　　　　　　　[root@k8s-01 ~]# systemctl enable kube-apiserver

　　　　　　　　　　[root@k8s-01 ~]# systemctl start kube-controller-manager

　　　　　　　　　　[root@k8s-01 ~]# systemctl enable kube-controller-manager

　　　　　　　　　　[root@k8s-01 ~]# systemctl start kube-scheduler

　　　　　　　　　　[root@k8s-01 ~]# systemctl enable kube-scheduler

　　　　4.4：配置并启动node：（node节点机器操作）

　　　　　　4.4.1：在kubernetes node上需要运行以下组件：

　　　　　　　　　　Kubelet

　　　　　　　　　　Kubernets Proxy

　　　　　　4.4.2：config

　　　　　　　　　　vim /etc/kubernetes/config

　　　　　　　　　　config原文件

[root@k8s-02 ~]# vim /etc/kubernetes/config 

###

# kubernetes system config

#

# The following values are used to configure various aspects of all

# kubernetes services, including

#

#   kube-apiserver.service

#   kube-controller-manager.service

#   kube-scheduler.service

#   kubelet.service

#   kube-proxy.service

# logging to stderr means we get it in the systemd journal

KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug

KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers

KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver

KUBE_MASTER="--master=http://127.0.0.1:8080"

　　　　　　config修改后文件：

[root@k8s-02 etcd]# vim /etc/kubernetes/config 

###

# kubernetes system config

#

# The following values are used to configure various aspects of all

# kubernetes services, including

#

#   kube-apiserver.service

#   kube-controller-manager.service

#   kube-scheduler.service

#   kubelet.service

#   kube-proxy.service

# logging to stderr means we get it in the systemd journal

KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug

KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers

KUBE_ALLOW_PRIV="--allow-privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver

KUBE_MASTER="--master=http://10.8.8.31:8080"

　　　　　　4.4.3：kubelet

　　　　　　　　　　vim /etc/kubernetes/kubelet

　　　　　　　　　　kubelet原文件：

[root@k8s-02 ~]# vim /etc/kubernetes/kubelet 

###

# kubernetes kubelet (minion) config

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)

KUBELET_ADDRESS="--address=127.0.0.1"

# The port for the info server to serve on

# KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname

KUBELET_HOSTNAME="--hostname-override=127.0.0.1"

# location of the api-server

KUBELET_API_SERVER="--api-servers=http://127.0.0.1:8080"

# pod infrastructure container

KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

# Add your own!

KUBELET_ARGS=""

　　　　　　　　kubelet修改后文件：

[root@k8s-02 etcd]# vim /etc/kubernetes/kubelet

###

# kubernetes kubelet (minion) config

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)

KUBELET_ADDRESS="--address=0.0.0.0"

# The port for the info server to serve on

# KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname

KUBELET_HOSTNAME="--hostname-override=10.8.8.32"

# location of the api-server

KUBELET_API_SERVER="--api-servers=http://10.8.8.31:8080"

# pod infrastructure container

KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

# Add your own!

KUBELET_ARGS=""

　　　　　　4.4.4：启动服务并添加开机启动

　　　　　　　　　　[root@k8s-02 ~]# systemctl start kubelet

　　　　　　　　　　[root@k8s-02 ~]# systemctl enable kubelet

　　　　　　　　　　[root@k8s-02 ~]# systemctl start kube-proxy

　　　　　　　　　　[root@k8s-02 ~]# systemctl enable kube-proxy

坑！

vim /etc/kubernetes/kubelet中

KUBELET_HOSTNAME="--hostname-override=10.8.8.32"这里如果没有配置hostname，status会报错

3月 20 11:19:56 k8s-02 kube-proxy[29412]: E0320 11:19:56.256315   29412 server.go:421] Can't get Node "k8s-02", assuming iptables proxy, err: nodes "k8s-02" not found

修改配置为： KUBELET_HOSTNAME="--hostname-override=k8s-02"

重启服务：systemctl restart kube-proxy

　　　　　　　　

　　4.5：进入master机器，查看node状态

　　　　kubectl -s http://10.8.8.31:8080 get node

[root@k8s-01 ~]# kubectl -s http://10.8.8.31:8080 get node

NAME        STATUS     AGE

10.8.8.32   NotReady   1h

k8s-02      Ready      2m

k8s-03      Ready      7m

五：配置网络flannel 

　　5.1：安装flannel（各节点机器都要安装）

　　　　[root@k8s-01 ~]# ansible all -a 'yum install -y flannel'

　　5.2：配置flannel

　　　　　　vim /etc/sysconfig/flanneld

[root@k8s-01 ~]# vim /etc/sysconfig/flanneld 

# Flanneld configuration options  

# etcd url location.  Point this to the server where etcd runs

FLANNEL_ETCD_ENDPOINTS="http://10.8.8.31:2379,http://10.8.8.32:2379,http://10.8.8.33:2379"

# etcd config key.  This is the configuration key that flannel queries

# For address range assignment

FLANNEL_ETCD_PREFIX="/atomic.io/network"

# Any additional options that you want to pass

#FLANNEL_OPTIONS=""

　　5.3：配置flannel网段

　　　　　　etcdctl mk /atomic.io/network/config '{ "Network":"10.10.0.0/16" }'

　　　　　　Flannel使用Etcd进行配置，来保证多个Flannel实例之间的配置一致性，所以需要在etcd上进行如下配置：（‘/atomic.io/network/config’这个key与上文/etc/sysconfig/flannel中的配置项FLANNEL_ETCD_PREFIX是相对应的，错误的话启动就会出错）

[root@k8s-01 ~]# etcdctl mk /atomic.io/network/config '{ "Network":"10.10.0.0/16" }'

{ "Network":"10.10.0.0/16" }

　　5.4：启动flannel服务并重启kubernetes服务

　　　　5.4.1：master上启动

　　　　　　[root@k8s-01 ~]# ansible master -a 'systemctl start flanneld'

　　　　　　[root@k8s-01 ~]# ansible master -a 'systemctl enable flanneld'

　　　　　　ifconfig 可以看到flannel信息

[root@k8s-01 ~]# ifconfig

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500

        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0

        ether 02:42:7f:71:21:01  txqueuelen 0  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 0  bytes 0 (0.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        inet 10.8.8.31  netmask 255.255.255.0  broadcast 10.8.8.255

        inet6 fe80::4e95:1400:1371:99a4  prefixlen 64  scopeid 0x20<link>

        ether 00:0c:29:0b:69:ff  txqueuelen 1000  (Ethernet)

        RX packets 900960  bytes 259734166 (247.7 MiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 843207  bytes 139504742 (133.0 MiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1472

        inet 10.10.43.0  netmask 255.255.0.0  destination 10.10.43.0

        inet6 fe80::da51:4e1c:3fdb:4c90  prefixlen 64  scopeid 0x20<link>

        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 3  bytes 144 (144.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

        inet 127.0.0.1  netmask 255.0.0.0

        inet6 ::1  prefixlen 128  scopeid 0x10<host>

        loop  txqueuelen 1000  (Local Loopback)

        RX packets 245215  bytes 80894269 (77.1 MiB)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 245215  bytes 80894269 (77.1 MiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

　　　　　　[root@k8s-01 ~]# ansible master -a 'systemctl restart docker'

　　　　　　[root@k8s-01 ~]# ansible master -a 'systemctl restart kube-apiserver'

　　　　　　[root@k8s-01 ~]# ansible master -a 'systemctl restart kube-controller-manager'

　　　　　　[root@k8s-01 ~]# ansible master -a 'systemctl restart kube-scheduler'

　　　　5.4.2：node上启动

　　　　　　[root@k8s-01 ~]# ansible node -a 'systemctl start flanneld'

　　　　　　[root@k8s-01 ~]# ansible node -a 'systemctl enable flanneld'

　　　　　　[root@k8s-01 ~]# ansible node -a 'systemctl restart docker'

　　　　　　[root@k8s-01 ~]# ansible node -a 'systemctl restart kubelet'

　　　　　　[root@k8s-01 ~]# ansible node -a 'systemctl restart kube-proxy'

 六：kubernetes-dashboard安装

https://www.cnblogs.com/zhenyuyaodidiao/p/6500897.html

https://blog.csdn.net/qq1083062043/article/details/84949924

https://www.cnblogs.com/fengzhihai/p/9851470.html

https://www.cnblogs.com/yy-cxd/p/6650573.html

　　6.1：准备registry.access.redhat.com/rhel7/pod-infrastructure:latest  （各节点都下载）

　　　　　　wget http://mirror.centos.org/centos/7/os/x86_64/Packages/python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm

　　　　　　rpm2cpio python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm | cpio -iv --to-stdout ./etc/rhsm/ca/redhat-uep.pem | tee /etc/rhsm/ca/redhat-uep.pem

　　　　　　vim /etc/rhsm/ca/redhat-uep.pem 已经有数据

　　　　　　docker pull registry.access.redhat.com/rhel7/pod-infrastructure:latest

　　6.2：拉取kubernetes-dashboard-amd64:v1.5.1　　（西游记）

　　　　　　docker pull gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1

　　6.3：docker文件转tar包　　（master上执行）

　　　　　　docker save gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1 > dashboard.tar

　　　　　　docker save registry.access.redhat.com/rhel7/pod-infrastructure:latest > podinfrastructure.tar

　　6.4：tar包转docker

　　　　　　docker load < dashboard.tar

　　　　　　

　　6.5：准备yaml文件

　　　　mkdir -p /etc/kubernetes/yamlfile

　　　　cd /etc/kubernetes/yamlfile　　

　　　　wget https://rawgit.com/kubernetes/kubernetes/master/cluster/addons/dashboard/dashboard-controller.yaml

　　　　wget https://rawgit.com/kubernetes/kubernetes/master/cluster/addons/dashboard/dashboard-service.yaml

　　　　vim dashboard.yaml

  1 apiVersion: extensions/v1beta1

  2 kind: Deployment

  3 metadata:

  4 # Keep the name in sync with image version and

  5 # gce/coreos/kube-manifests/addons/dashboard counterparts

  6   name: kubernetes-dashboard-latest

  7   namespace: kube-system

  8 spec:

  9   replicas: 1

 10   template:

 11     metadata:

 12       labels:

 13         k8s-app: kubernetes-dashboard

 14         version: latest

 15         kubernetes.io/cluster-service: "true"

 16     spec:

 17       containers:

 18       - name: kubernetes-dashboard

 19         image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1

 20         resources:

 21           # keep request = limit to keep this container in guaranteed class

 22           limits:

 23             cpu: 100m

 24             memory: 50Mi

 25           requests:

 26             cpu: 100m

 27             memory: 50Mi

 28         ports:

 29         - containerPort: 9090

 30         args:

 31          -  --apiserver-host=http://10.8.8.31:8080

 32         livenessProbe:

 33           httpGet:

 34             path: /

 35             port: 9090

 36           initialDelaySeconds: 30

 37           timeoutSeconds: 30

　　　　vim dashboardsvc.yaml

  1 apiVersion: v1

  2 kind: Service

  3 metadata:

  4   name: kubernetes-dashboard

  5   namespace: kube-system

  6   labels:

  7     k8s-app: kubernetes-dashboard

  8     kubernetes.io/cluster-service: "true"

  9 spec:

 10   selector:

 11     k8s-app: kubernetes-dashboard

 12   ports:

 13   - port: 80

 14     targetPort: 9090

　　6.6：用yaml启动

　　　　　　kubectl create -f dashboard.yaml

　　　　　　kubectl create -f dashboardsvc.yaml

[root@k8s-01 yamlfail]# kubectl create -f dashboard.yaml

deployment "kubernetes-dashboard-latest" created

[root@k8s-01 yamlfail]# kubectl create -f dashboardsvc.yaml

service "kubernetes-dashboard" created

　　　　　　删除方法：

　　　　　　　　kubectl delete -f xxx.yaml

　　　　　　　　kubectl delete deployment kubernetes-dashboard-latest --namespace=kube-system

　　　　　　　　kubectl delete svc  kubernetes-dashboard --namespace=kube-system

　　　　　　注意:

　　　　　　　　kubectl get deployment --all-namespaces

　　　　　　　　不要直接删除pod，使用kubectl请删除拥有该pod的Deployment。如果直接删除pod，则Deployment将会重新创建该pod。

　　　　　　　　

　　6.7：查看pod状态

　　　　　　kubectl get pod --all-namespaces

[root@k8s-01 yamlfail]# kubectl get pod --all-namespaces
NAMESPACE     NAME                                          READY     STATUS    RESTARTS   AGE
kube-system   kubernetes-dashboard-latest-190610294-c027r   1/1       Running   0          1h

　　　　　　

　　　　　　kubectl get svc  --all-namespaces

[root@k8s-01 yamlfail]# kubectl get svc  --all-namespaces
NAMESPACE     NAME                   CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
default       kubernetes             10.254.0.1      <none>        443/TCP   2d
kube-system   kubernetes-dashboard   10.254.112.86   <none>        80/TCP    1h

　　　　　　kubectl get pod  -o wide  --all-namespaces

[root@k8s-01 yamlfail]# kubectl get pod  -o wide  --all-namespaces
NAMESPACE     NAME                                          READY     STATUS    RESTARTS   AGE       IP           NODE
kube-system   kubernetes-dashboard-latest-190610294-c027r   1/1       Running   0          1h        10.10.49.2   k8s-02

　　6.8：web访问

　　　　　　http://10.8.8.31:8080/ui

Error: 'dial tcp 10.10.49.2:9090: getsockopt: connection timed out'

Trying to reach: 'http://10.10.49.2:9090/'

　　　　　　

　　6.9：curl 10.10.49.2:9090

[root@k8s-02 ~]# curl 10.10.49.2:9090

 <!doctype html> <html ng-app="kubernetesDashboard"> <head> <meta charset="utf-8"> <title>Kubernetes Dashboard</title> <link rel="icon" type="image/png" href="assets/images/kubernetes-logo.png"> <meta name="viewport" content="width=device-width"> <link rel="stylesheet" href="static/vendor.a0fa0655.css"> <link rel="stylesheet" href="static/app.968d5cf5.css"> </head> <body> <!--[if lt IE 10]>

      <p class="browsehappy">You are using an <strong>outdated</strong> browser.

      Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your

      experience.</p>

    <![endif]--> <kd-chrome layout="column" layout-fill> </kd-chrome> <script src="static/vendor.89dbb771.js"></script> <script src="api/appConfig.json"></script> <script src="static/app.50ef120b.js"></script> </body> </html>

　　　　　　通过查看网卡信息，k8s-03为10.10.80.0网段

　　　　　　[root@k8s-03 zm]# ifconfig

flannel0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1472

        inet 10.10.80.0  netmask 255.255.0.0  destination 10.10.80.0

        inet6 fe80::3624:5df7:a344:fc0e  prefixlen 64  scopeid 0x20<link>

        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 3  bytes 144 (144.0 B)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

　　

　　　　　　在k8s-03机器上可以ping通dashboard，其他机器不通

[root@k8s-03 zm]# ping 10.10.80.2

PING 10.10.80.2 (10.10.80.2) 56(84) bytes of data.

64 bytes from 10.10.80.2: icmp_seq=1 ttl=64 time=0.058 ms

64 bytes from 10.10.80.2: icmp_seq=2 ttl=64 time=0.043 ms

　　　　　　查看docker ip：

　　　　　　　　docker inspect -f '{{.Name}} - {{.NetworkSettings.IPAddress }}' $(docker ps -aq)

[root@k8s-03 zz]# docker inspect -f '{{.Name}} - {{.NetworkSettings.IPAddress }}' $(docker ps -aq)

/k8s_kubernetes-dashboard.88d5a45d_kubernetes-dashboard-latest-190610294-zxgtw_kube-system_9ba7a9b3-4bbc-11e9-958a-000c290b69ff_e5226d0a -

/k8s_POD.28c50bab_kubernetes-dashboard-latest-190610294-zxgtw_kube-system_9ba7a9b3-4bbc-11e9-958a-000c290b69ff_c5434807 - 10.10.80.2

/k8s_kubernetes-dashboard.88d5a45d_kubernetes-dashboard-latest-190610294-zxgtw_kube-system_9ba7a9b3-4bbc-11e9-958a-000c290b69ff_443e86fe -

/k8s_POD.28c50bab_kubernetes-dashboard-latest-190610294-zxgtw_kube-system_9ba7a9b3-4bbc-11e9-958a-000c290b69ff_618335e7 -

　　　　　　　　kubectl cluster-info

[root@k8s-01 yamlfail]# kubectl cluster-info

Kubernetes master is running at http://localhost:8080

kubernetes-dashboard is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

　　　　 解决办法：

　　　　　　各节点开启ip转发：

　　　　　　　　echo "net.ipv4.ip_forward = 1" >>/usr/lib/sysctl.d/50-default.conf

　　　　　　各节点修改flannel配置文件：

　　　　　　　　vim /etc/sysconfig/flanneld

填坑：

　　　　[root@k8s-01 yamlfile]# kubectl create -f dashboard-controller.yaml

[root@k8s-01 yamlfile]# kubectl create -f dashboard-controller.yaml

Error from server (AlreadyExists): error when creating "dashboard-controller.yaml": serviceaccounts "kubernetes-dashboard" already exists

yaml: line 50: did not find expected key

　　　　用如下方法删除

　　　　　　kubectl delete -f kubernetes-dashboard.yaml

[root@k8s-01 yamlfile]# kubectl delete -f kubernetes-dashboard.yaml

secret "kubernetes-dashboard-certs" deleted

serviceaccount "kubernetes-dashboard" deleted

　　　　再次创建

　　　　　　kubectl create -f dashboard-controller.yaml

[root@k8s-01 yamlfile]# kubectl create -f dashboard-controller.yaml

serviceaccount "kubernetes-dashboard" created

error: yaml: line 50: did not find expected key

　　　　

　　　　　　kubectl create -f dashboard-service.yaml

[root@k8s-01 yamlfile]# kubectl create -f dashboard-service.yaml

service "kubernetes-dashboard" created

　　　　查看：

　　　　　　kubectl get svc --all-namespaces

[root@k8s-01 yamlfile]# kubectl get svc --all-namespaces

NAMESPACE     NAME                   CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE

default       kubernetes             10.254.0.1      <none>        443/TCP   23h

kube-system   kubernetes-dashboard   10.254.227.33   <none>        443/TCP   4m

　　　　网页访问：

　　　　　　http://10.8.8.31:8080/ui （不成功）

坑!

https://www.cnblogs.com/guyeshanrenshiwoshifu/p/9147238.html

　　　　查看pod：

　　　　　kubectl get pods --all-namespaces

[root@k8s-01 yamlfile]# kubectl get pods --all-namespaces

NAMESPACE     NAME                                    READY     STATUS              RESTARTS   AGE

kube-system   kubernetes-dashboard-1468570674-zxgtw   0/1       ContainerCreating   0          7m

　　　　查看信息：

　　　　　　kubectl describe pod kubernetes-dashboard-2498798083-tgwsn --namespace=kube-system

[root@k8s-01 yamlfile]# kubectl describe pod kubernetes-dashboard-2498798083-tgwsn --namespace=kube-system

Name:		kubernetes-dashboard-2498798083-tgwsn

Namespace:	kube-system

Node:		k8s-03/10.8.8.33

Start Time:	Thu, 21 Mar 2019 12:04:12 +0800

Labels:		app=kubernetes-dashboard

		pod-template-hash=2498798083

Status:		Pending

IP:

Controllers:	ReplicaSet/kubernetes-dashboard-2498798083

Containers:

  kubernetes-dashboard:

    Container ID:

    Image:		gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1

    Image ID:

    Port:		9090/TCP

    Args:

      --apiserver-host=http://10.8.8.31:8080

    State:			Waiting

      Reason:			ContainerCreating

    Ready:			False

    Restart Count:		0

    Liveness:			http-get http://:9090/ delay=30s timeout=30s period=10s #success=1 #failure=3

    Volume Mounts:		<none>

    Environment Variables:	<none>

Conditions:

  Type		Status

  Initialized 	True

  Ready 	False

  PodScheduled 	True

No volumes.

QoS Class:	BestEffort

Tolerations:	dedicated=master:Equal:NoSchedule

Events:

  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason		Message

  ---------	--------	-----	----			-------------	--------	------		-------

  1h		1m		18	{kubelet k8s-03}			Warning		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "POD" with ErrImagePull: "image pull failed for registry.access.redhat.com/rhel7/pod-infrastructure:latest, this may be because there are no credentials on this request.  details: (open /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt: no such file or directory)"

  1h	2s	296	{kubelet k8s-03}		Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "POD" with ImagePullBackOff: "Back-off pulling image \"registry.access.redhat.com/rhel7/pod-infrastructure:latest\""

　　　　　　cd /etc/docker/certs.d/registry.access.redhat.com/

[root@k8s-01 ~]# cd /etc/docker/certs.d/registry.access.redhat.com/

[root@k8s-01 registry.access.redhat.com]# ll

总用量 0

lrwxrwxrwx 1 root root 27 3月  20 09:06 redhat-ca.crt -> /etc/rhsm/ca/redhat-uep.pem
[root@k8s-01 registry.access.redhat.com]# cd /etc/rhsm/ca/
[root@k8s-01 ca]# ll
总用量 0

　　　　　　生成：redhat-uep.pem

　　　　　　　　wget http://mirror.centos.org/centos/7/os/x86_64/Packages/python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm

　　　　　　　　rpm2cpio python-rhsm-certificates-1.19.10-1.el7_4.x86_64.rpm | cpio -iv --to-stdout ./etc/rhsm/ca/redhat-uep.pem | tee /etc/rhsm/ca/redhat-uep.pem

　　　　　　　　vim /etc/rhsm/ca/redhat-uep.pem 已经有数据

　　　　　　　　docker pull registry.access.redhat.com/rhel7/pod-infrastructure:latest

　　　　　　删除并重新生成

　　　　　　　　cd /etc/kubernetes/yamlfile/

　　　　　　　　kubectl delete -f dashboard-controller.yaml

　　　　　　　　kubectl delete -f dashboard-service.yaml

　　　　　　　　kubectl create -f dashboard-controller.yaml

　　　　　　　　kubectl create -f dashboard-service.yaml

七：继续测试kube-ui

　　7.1：web访问：

　　　　http://10.8.8.31:8080/ui

　　　　自动跳转到：

　　　　http://10.8.8.31:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/

　　　　报错：

{

  "kind": "Status",

  "apiVersion": "v1",

  "metadata": {},

  "status": "Failure",

  "message": "endpoints \"kubernetes-dashboard\" not found",

  "reason": "NotFound",

  "details": {

    "name": "kubernetes-dashboard",

    "kind": "endpoints"

  },

  "code": 404

}

　　

　　7.2：重启各结点及创建docker

　　　　测试网络联通性，master上要能ping通个node上的docker

　　　　cd /etc/kubernetes/yamlfail

　　　　kubectl create -f dashboard.yaml

　　　　kubectl create -f dashboardsvc.yaml

[root@k8s-01 yamlfail]# kubectl create -f dashboard.yaml

deployment "kubernetes-dashboard-latest" created

您在 /var/spool/mail/root 中有新邮件

[root@k8s-01 yamlfail]# kubectl create -f dashboardsvc.yaml

service "kubernetes-dashboard" created

　　

　　7.3：查看状态

　　　　kubectl get deployment --all-namespaces

　　　　kubectl get svc --all-namespaces

　　　　kubectl get pod -o wide --all-namespaces

[root@k8s-01 yamlfail]# kubectl get deployment --all-namespaces

NAMESPACE     NAME                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE

kube-system   kubernetes-dashboard-latest   1         1         1            1           22s

[root@k8s-01 yamlfail]# kubectl get svc  --all-namespaces

NAMESPACE     NAME                   CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE

default       kubernetes             10.254.0.1       <none>        443/TCP   14d

kube-system   kubernetes-dashboard   10.254.157.175   <none>        80/TCP    38s

[root@k8s-01 yamlfail]# kubectl get pod  -o wide  --all-namespaces

NAMESPACE     NAME                                          READY     STATUS    RESTARTS   AGE       IP          NODE

kube-system   kubernetes-dashboard-latest-190610294-nf0jc   1/1       Running   0          59s       10.10.7.2   k8s-03

　　

　　7.4：再次web访问：

　　　　http://10.8.8.31:8080/ui

　　　　http://10.8.8.31:8080/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard/#/workload?namespace=default

CentOS7.6配置do.cker和K.B.S

感谢：

条例清晰！（方法一）

https://www.cnblogs.com/zhenyuyaodidiao/p/6500830.html

https://www.cnblogs.com/zhenyuyaodidiao/p/6500897.html

有点繁杂：（方法二）

https://www.cnblogs.com/netsa/p/8279045.html

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~      方法二      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

方法二：

 未完结！！！

一：环境准备

　　1.1：删除原有go/golang

whereis go

whereis golang

whereis gocode #如果需要的话

//找到后删除

rm -rf  xxx

　　1.2：下载

　　　　https://studygolang.com/dl

　　　　wget https://studygolang.com/dl/golang/go1.12.linux-amd64.tar.gz

　　

　　1.3：解压到指定目录

　　　　tar -C /usr/local/ -zxvf go1.12.linux-amd64.tar.gz

　　　　cd /usr/local/go

　　

　　1.4：建立gopath目录

　　　　mkdir -p /home/gocode

　　

　　1.5：添加环境

　　　　vim /etc/profile

export GOROOT=/usr/local/go

export GOPATH=/home/gocode

export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

　　　　source /etc/profile

　　　　验证是否成功

　　　　go version

　　

　　1.6：安装git

　　　　yum install -y git

　　1.7：下载

 go get -v github.com/gin-gonic/gin

 go get -v github.com/go-sql-driver/mysql

 go get -v github.com/robfig/cron

　　1.8：测试

vim helloworld.go 

package main

import "fmt"

func main() {

fmt.Printf("Hello, world.\n")

} 

运行 go run helloworld.go

编译 go build helloworld.go

go install

编译后的文件运行为 ./helloworld

后台运行：

Linux 在运行程序的尾部加入&，或者nohup ./example &

　　1.9：SSH免密

https://blog.csdn.net/wangganggang3168/article/details/80568049

https://blog.csdn.net/wang704987562/article/details/78904350

　　　　ssh-keygen -t rsa （各节点均需执行）

[root@docker-01 ~]# ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:UhHoFCQ/SyuQdw61fWVPkQn/jhY59HwTvG/SpfC4CXk root@docker-01

The key's randomart image is:

+---[RSA 2048]----+

|    ..+oo.  +.o+ |

|   . +oo . o ++  |

|  o oo* o .   +o |

|   o =.= .   . =o|

|    . = S   . +o*|

|     . .   . + B=|

|          o E * =|

|           o + o |

|            o    |

+----[SHA256]-----+

　　　　把各节点的id_rsa.pub内容集中拷贝到authorized_keys

　　　　vim /root/.ssh/id_rsa.pub

　　　　vim /root/.ssh/authorized_keys

　　　　scp authorized_keys root@docker-01:/root/.ssh/

　　　　分别登入其他节点，分别ssh到各节点，第一次ssh会有提示，输入yes后解除

[root@docker-02 .ssh]# ssh docker-04

The authenticity of host 'docker-04 (10.8.8.24)' can't be established.

ECDSA key fingerprint is SHA256:8UK41mz0DDPjzQ7UPH9ADOFYBN34cMFJVXaOJ5gADx0.

ECDSA key fingerprint is MD5:15:63:19:03:ad:fb:a6:e8:3d:74:01:0b:ab:88:88:0b.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'docker-04,10.8.8.24' (ECDSA) to the list of known hosts.

Last login: Wed Mar 13 21:02:52 2019 from docker-01

　　

二：生成证书：

　　2.1：

　　　　参考：https://kubernetes.io/zh/docs/concepts/cluster-administration/certificates/#创建证书

　　　   CFSSL方法

　　　　https://kubernetes.io/zh/docs/concepts/cluster-administration/certificates/#cfssl

　　　　下载安装：cd /zz

curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /bin/cfssl

curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /bin/cfssl-certinfo

curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /bin/cfssljson

chmod +x cfssl*

　　

　　2.2：创建ca-config.jaon：

　　　　mkdir -p /opt/ssl && cd /opt/ssl

　　　　可用cfssl print-defaults config > ca-config.json自动生成后按需改配置

　　　　vim ca-config.jaon

{

  "signing": {

    "default": {

      "expiry": "87600h"

    },

    "profiles": {

      "kubernetes": {

        "usages": [

          "signing",

          "key encipherment",

          "server auth",

          "client auth"

        ],

        "expiry": "87600h"

      }

    }

  }

}

　　　　过期时间配置为10年

　　　　ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数，后续在签名证书时使用某个profile

　　　　signing：表示该证书可用于签名其它证书，生成的ca.pem证书中CA=TRUE；

　　　　server auth：表示client可以用该CA对server提供的证书进行验证；

　　　　client auth：表示server可以用该CA对client提供的证书进行验证

　　2.3：创建ca-csr.jaon

　　　　可用cfssl print-defaults csr > ca-csr.json自动生成后按需修改

　　　　vim ca-csr.jaon

{

  "CN": "kubernetes",

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names":[{

    "C": "<country>",

    "ST": "<state>",

    "L": "<city>",

    "O": "<organization>",

    "OU": "<organization unit>"

  }]

}

　　

{

  "CN": "kubernetes",

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "System"

    }

  ]

}

　　　　CN : Common Name，kube-apiserver从证书中提取该字段作为请求的用户名；

　　　　O : Organization，kube-apiserver从证书中提取该字段作为请求用户所属的组；

　　2.4：生成证书密钥：　　　

　　　　cfssl gencert -initca ca-csr.json | cfssljson -bare ca
[root@docker-01 ssl]# vim ca-csr.json

[root@docker-01 ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca

2019/03/13 11:01:36 [INFO] generating a new CA key and certificate from CSR

2019/03/13 11:01:36 [INFO] generate received request

2019/03/13 11:01:36 [INFO] received CSR

2019/03/13 11:01:36 [INFO] generating key: rsa-2048

2019/03/13 11:01:36 [INFO] encoded CSR

2019/03/13 11:01:36 [INFO] signed certificate with serial number 377680744285591674329230033735744500343528771314

[root@docker-01 ssl]# ll

总用量 20

-rw-r--r--. 1 root root  284 3月  12 21:33 ca-config.json

-rw-r--r--. 1 root root 1001 3月  13 11:01 ca.csr

-rw-r--r--. 1 root root  208 3月  13 11:01 ca-csr.json

-rw-------. 1 root root 1679 3月  13 11:01 ca-key.pem

-rw-r--r--. 1 root root 1359 3月  13 11:01 ca.pem

　　

　　2.5：创建kubernetes证书

　　　　vim kubernetes-csr.json

{

    "CN": "kubernetes",

    "hosts": [

      "127.0.0.1",

      "10.8.8.21",

      "10.8.8.22",

      "10.8.8.23",

      "10.8.8.24",

      "kubernetes",

      "kubernetes.default",

      "kubernetes.default.svc",

      "kubernetes.default.svc.cluster",

      "kubernetes.default.svc.cluster.local"

    ],

    "key": {

        "algo": "rsa",

        "size": 2048

    },

    "names": [

        {

            "C": "CN",

            "ST": "BeiJing",

            "L": "BeiJing",

            "O": "k8s",

            "OU": "System"

        }

    ]

}

　　2.6：生成kubernetes密钥

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

[root@docker-01 ssl]# vim kubernetes-csr.json

[root@docker-01 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

2019/03/13 11:21:38 [INFO] generate received request

2019/03/13 11:21:38 [INFO] received CSR

2019/03/13 11:21:38 [INFO] generating key: rsa-2048

2019/03/13 11:21:38 [INFO] encoded CSR

2019/03/13 11:21:38 [INFO] signed certificate with serial number 466577397722502141135271666270895637824536137432

2019/03/13 11:21:38 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for

websites. For more information see the Baseline Requirements for the Issuance and Management

of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);

specifically, section 10.2.3 ("Information Requirements").

　　上面WARNING先忽略

　

　　2.7：创建admin证书

　　　　vim admin-csr.json

{

  "CN": "kubernetes-admin",

  "hosts": [

        "10.8.8.21",

        "10.8.8.22",

        "10.8.8.23",

        "10.8.8.24"

  ],

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "system:masters",

      "OU": "System"

    }

  ]

}

　　　　kube-apiserver将提取CN作为客户端的用户名，这里是kubernetes-admin，

　　　　将提取O作为用户所属的组，这里是system:master。

　　　　kube-apiserver预定义了一些 RBAC使用的ClusterRoleBindings，

　　　　例如 cluster-admin将组system:masters与 ClusterRole cluster-admin绑定，

　　　　而cluster-admin拥有访问kube-apiserver的所有权限，

　　　　因此kubernetes-admin这个用户将作为集群的超级管理员。

　　　2.8：生成admin密钥

　　　　cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

[root@docker-01 ssl]# vim admin-csr.json

[root@docker-01 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

2019/03/13 13:19:32 [INFO] generate received request

2019/03/13 13:19:32 [INFO] received CSR

2019/03/13 13:19:32 [INFO] generating key: rsa-2048

2019/03/13 13:19:33 [INFO] encoded CSR

2019/03/13 13:19:33 [INFO] signed certificate with serial number 542875374330312060082808070092917596528046572224

2019/03/13 13:19:33 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for

websites. For more information see the Baseline Requirements for the Issuance and Management

of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);

specifically, section 10.2.3 ("Information Requirements").

　　　2.9：创建kube-proxy-csr.json证书

　　　　　　vim kube-proxy-csr.json

{

  "CN": "system:kube-proxy",

  "hosts": [],

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "System"

    }

  ]

　　　　指定证书User为 system:kube-proxy

　　　　kube-apiserver 预定义的RoleBinding cluster-admin

　　　　将User system:kube-proxy与Role system:node-proxier绑定，

　　　　将Role授予调用kube-apiserver Proxy相关API的权限；

　　生成kube-proxy证书和密钥

　　　　cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy

[root@docker-01 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy

2019/03/13 13:30:08 [INFO] generate received request

2019/03/13 13:30:08 [INFO] received CSR

2019/03/13 13:30:08 [INFO] generating key: rsa-2048

2019/03/13 13:30:08 [INFO] encoded CSR

2019/03/13 13:30:08 [INFO] signed certificate with serial number 567732124973226627997281945626780290685046730115

2019/03/13 13:30:08 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for

websites. For more information see the Baseline Requirements for the Issuance and Management

of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);

specifically, section 10.2.3 ("Information Requirements").

　　校验证书：看输出内容和json是否一致

　　　　cfssl-certinfo -cert kubernetes.pem

组件 证书 说明
etcd ca.pem、kubernetes-key.pem、kubernetes.pem 和kube-apiserver通用
kube-apiserver ca.pem、kubernetes-key.pem、kubernetes.pem kube-controller、kube-scheduler和apiserver都是部署在master可以使用非安全通行，不再单独安装证书。
kube-proxy ca.pem、kube-proxy-key.pem、kube-proxy.pem  
kubectl ca.pem、admin-key.pem、admin.pem  

三：搭建Etcd

https://www.jianshu.com/p/98b8fa3e3596

　　各节点均需执行！！！

　　3.1：关闭selinux

　　　　getenforce

　　　　vim /etc/selinux/config

　　　　SELINUX=disabled

　　3.2：关闭交换分区swap

　　　　swapoff -a

　　　　rm /dev/mapper/centos-swap

　　　　sed -i 's/.*swap.*/#&/' /etc/fstab

　　3.3：设置内核

　　　　vim /etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

　　　　sysctl -p /etc/sysctl.conf

　　3.4：环境配置

　　　　vim /root/.bash_profile

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

        . ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

export NODE_NAME=docker-01

export NODE_IP=10.8.8.21

export NODE_IPS="10.8.8.21 10.8.8.22 10.8.8.23 10.8.8.24"

export ETCD_NODES=docker-01=https://10.8.8.21:2380,docker-02=https://10.8.8.22:2380,docker-03=https://10.8.8.23:2380,docker-04=https://10.8.8.24:2380

~

　　3.5：etcd证书配置

　　　　cd /etc/kubernetes/ssl

　　　　创建etcd签名请求

　　　　vim etcd-csr.json

{

  "CN": "etcd",

  "hosts": [

    "127.0.0.1",

    "10.8.8.21",

    "10.8.8.22",

    "10.8.8.23",

    "10.8.8.24"

  ],

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "System"

    }

  ]

}

　　　　生成etcd证书和秘钥

　　　　　　cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
[root@docker-01 ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

2019/03/13 16:09:32 [INFO] generate received request

2019/03/13 16:09:32 [INFO] received CSR

2019/03/13 16:09:32 [INFO] generating key: rsa-2048

2019/03/13 16:09:33 [INFO] encoded CSR

2019/03/13 16:09:33 [INFO] signed certificate with serial number 398364810642443697380742999828998753293408212966

2019/03/13 16:09:33 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for

websites. For more information see the Baseline Requirements for the Issuance and Management

of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);

specifically, section 10.2.3 ("Information Requirements").

[root@docker-01 ssl]#

　　

　　3.6：安装ETCD

　　　　https://coreos.com/etcd/docs/latest/dl_build.html

　　　　tar -zxvf etcd-v3.3.12-linux-amd64.tar.gz

　　　　cd etcd-v3.3.12-linux-amd64

　　　　cp etcd* /usr/local/bin/

　　　  export ETCDCTL_API=3

　　　　env

　　　　拷贝到其他节点：

　　　　　　scp /usr/local/bin/etcd* root@docker-02:/usr/local/bin/

　　　　　　scp /usr/local/bin/etcd* root@docker-03:/usr/local/bin/

　　　　　　scp /usr/local/bin/etcd* root@docker-04:/usr/local/bin/

　　　　创建etcd工作目录

　　　　　　mkdir -p /var/lib/etcd　

　　　　　　如果没有配置这个目录，会现现Failed at step CHDIR spawning /usr/local/bin/etcd: No such file or directory的错误信息。

　　　　创建配置文件目录

　　　　　　mkdir -p /etc/etcd

　　3.7：创建ETCD的配置文件

　　　　/etc/etcd/etcd-key.conf：存放我们证书的配置信息

　　　　/etc/etcd/etcd.conf：存放ETCD集群的配置信息

　　　　vim /etc/etcd/etcd-key.conf

ETCD_KEY='--cert-file=/etc/kubernetes/ssl/etcd.pem --key-file=/etc/kubernetes/ssl/etcd-key.pem --peer-cert-file=/etc/kubernetes/ssl/etcd.pem --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem --trusted-ca-file=/etc/kubernetes/ssl/ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem'

　　　　vim /etc/etcd/etcd.conf

　　　　master配置：

ETCD_NAME='--name=k8s-master'

ETCD_DATA_DIR='--data-dir=/data/etcd'

ETCD_INITIAL_CLUSTER_STATE='--initial-cluster-state=new'

ETCD_INITIAL_CLUSTER_TOKEN='--initial-cluster-token=etcd-cluster-0'

ETCD_INITIAL_ADVERTISE_PEER_URLS='--initial-advertise-peer-urls=http://10.8.8.21:2380'

ETCD_LISTEN_PEER_URLS='--listen-peer-urls=http://10.8.8.21:2380'

ETCD_LISTEN_CLIENT_URLS='--listen-client-urls=http://10.8.8.21:2379,http://127.0.0.1:2379'

ETCD_ADVERTISE_CLIENT_URLS='--advertise-client-urls=http://10.8.8.21:2379'

ETCD_INITIAL_CLUSTER='--initial-cluster=k8s-master=http://10.8.8.21:2380,k8s-node02=http://10.8.8.22:2380,k8s-node03=http://10.8.8.23:2380,k8s-node04=http://10.8.8.24:2380'

#ETCD_KEY='/etc/kubernetes/ssl/'

　　

　　　　node配置：

ETCD_NAME='--name=k8s-node02'

ETCD_DATA_DIR='--data-dir=/data/etcd'

ETCD_INITIAL_CLUSTER_STATE='--initial-cluster-state=new'

ETCD_INITIAL_CLUSTER_TOKEN='--initial-cluster-token=etcd-cluster-0'

ETCD_INITIAL_ADVERTISE_PEER_URLS='--initial-advertise-peer-urls=http://10.8.8.22:2380'

ETCD_LISTEN_PEER_URLS='--listen-peer-urls=http://10.8.8.22:2380'

ETCD_LISTEN_CLIENT_URLS='--listen-client-urls=http://10.8.8.22:2379,http://127.0.0.1:2379'

ETCD_ADVERTISE_CLIENT_URLS='--advertise-client-urls=http://10.8.8.22:2379'

ETCD_INITIAL_CLUSTER='--initial-cluster=k8s-master=http://10.8.8.21:2380,k8s-node02=http://10.8.8.22:2380,k8s-node03=http://10.8.8.23:2380,k8s-node04=http://10.8.8.24:2380'

#ETCD_KEY='/etc/kubernetes/ssl/'

　　　　/etc/etcd/etcd.conf文件中等号左边键与/usr/lib/systemd/system/etcd.service中$后的命名一致

　　　　/etc/etcd/etcd.conf文件中等号右边单引号中等号左边的值与etcd --help中命名一致，如不一致启动集群时会报错

　　　　vim /var/log/messages

Mar 14 13:53:22 docker-01 systemd: Starting Etcd Server...

Mar 14 13:53:22 docker-01 etcd: error verifying flags, 'k8s_master' is not a valid flag. See 'etcd --help'.

Mar 14 13:53:22 docker-01 systemd: etcd.service: main process exited, code=exited, status=1/FAILURE

Mar 14 13:53:22 docker-01 systemd: Failed to start Etcd Server.

Mar 14 13:53:22 docker-01 systemd: Unit etcd.service entered failed state.

Mar 14 13:53:22 docker-01 systemd: etcd.service failed.

Mar 14 13:53:23 docker-01 systemd: Stopped Etcd Server.

　　

　　3.8：添加服务

　　　　vim /usr/lib/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

#Documentation=https://github.com/coreos

[Service]

Type=notify

WorkingDirectory=/data/etcd/

EnvironmentFile=-/etc/etcd/etcd.conf

EnvironmentFile=-/etc/etcd/etcd-key.conf

ExecStart=/usr/local/bin/etcd \

    $ETCD_NAME \

    $ETCD_DATA_DIR \

    $ETCD_INITIAL_CLUSTER_STATE \

    $ETCD_INITIAL_CLUSTER_TOKEN \

    $ETCD_INITIAL_ADVERTISE_PEER_URLS \

    $ETCD_LISTEN_PEER_URLS \

    $ETCD_LISTEN_CLIENT_URLS \

    $ETCD_ADVERTISE_CLIENT_URLS \

    $ETCD_INITIAL_CLUSTER \

    $ETCD_KEY

Restart=on-failure

RestartSec=5

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target　
　　　　name
　　　　　　节点名称

　　　　data-dir
　　　　　　指定节点的数据存储目录

　　　　listen-peer-urls
　　　　　　 监听URL，用于与其他节点通讯

　　　　listen-client-urls
　　　　　　对外提供服务的地址：比如 http://ip:2379,http://127.0.0.1:2379 ，客户端会连接到这里和 etcd 交互

　　　　initial-advertise-peer-urls
　　　　　　该节点同伴监听地址，这个值会告诉集群中其他节点

　　　　initial-cluster
　　　　　　集群中所有节点的信息，格式为 node1=http://ip1:2380,node2=http://ip2:2380,… 。注意：这里的 node1 是节点的 --name 指定的名字；后面的 ip1:2380 是 --initial-advertise-peer-urls 指定的值

　　　　initial-cluster-state
　　　　　　新建集群的时候，这个值为 new ；假如已经存在的集群，这个值为 existing

　　　　initial-cluster-token
　　　　　　创建集群的 token，这个值每个集群保持唯一。这样的话，如果你要重新创建集群，即使配置和之前一样，也会再次生成新的集群和节点 uuid；否则会导致多个集群之间的冲突，造成未知的错误

　　　　advertise-client-urls
　　　　　　对外公告的该节点客户端监听地址，这个值会告诉集群中其他节点

　　　　在不同的设备上要替换name、initial-advertise-peer-urls、listen-peer-urls、listen-client-urls、advertise-client-urls中的名称和IP

　　复制到其他服务器：

　　　　scp etcd.service root@docker-02:/usr/lib/systemd/system/

　　修改配置文件etcd.service

　　

　　3.9：启动etcd集群（各节点均需启动，启动主节点时会停滞很久，因为其他节点还未开启）

　　　　修改vim /usr/lib/systemd/system/etcd.service文件后要重新加载

　　　　　　systemctl daemon-reload

　　　　　　systemctl start etcd.service

　　　　　   systemctl stop etcd.service

　　　　报错处理：

　　　　　　3.9.1：connection refused

Mar 14 14:32:46 docker-01 etcd: health check for peer 7d8eee4f1e1ab8e9 could not connect: dial tcp 10.8.8.22:2380: connect: connection refused (prober "ROUND_TRIPPER_SNAPSHOT")

　　　　　　　　ssh连接不通

[root@docker-01 system]# ssh 10.8.8.24 -p 2380

ssh: connect to host 10.8.8.24 port 2380: Connection refused

　　　　　　　　解决方法：先启动node节点机器，后启动master

　　　　　　3.9.2：etcd.service服务配置文件中设置的工作目录WorkingDirectory=xxx目录必须存在，并且建好，否则/var/log/message报错

Mar 14 15:25:21 docker-03 systemd: Starting Etcd Server...

Mar 14 15:25:21 docker-03 systemd: Failed at step CHDIR spawning /usr/local/bin/etcd: No such file or directory

Mar 14 15:25:21 docker-03 systemd: etcd.service: main process exited, code=exited, status=200/CHDIR

Mar 14 15:25:21 docker-03 systemd: Failed to start Etcd Server.

Mar 14 15:25:21 docker-03 systemd: Unit etcd.service entered failed state.

Mar 14 15:25:21 docker-03 systemd: etcd.service failed.

Mar 14 15:25:23 docker-03 systemd: Stopped Etcd Server.

　　

　　　　　　3.9.3：request cluster ID mismatch

　　　　　　　　https://blog.51cto.com/1666898/2156165

Mar 15 08:38:22 docker-01 etcd: request cluster ID mismatch (got ce8738a43379cfa0 want 25c4c375d3f1f1e)

Mar 15 08:38:22 docker-01 etcd: rejected connection from "10.8.8.22:57202" (error "tls: first record does not look like a TLS handshake", ServerName "")

　　　　　　　　删除配置文件中--data-dir项！

　　　　　　　　解决办法：删除了etcd集群所有节点中的--data_dir的内容
　　　　　　　　分析： 因为集群搭建过程，单独启动过单一etcd,做为测试验证，集群内第一次启动其他etcd服务时候，是通过发现服务引导的，所以需要删除旧的成员信息
　　　　　　　　参考：One of the member was bootstrapped via discovery service. You must remove the previous data-dir to clean up the member information. Or the member will ignore the new configuration and start with the old configuration. That is why you see the mismatch.

　　　　　　3.9.4：以下是最终启动成功的etcd.service，note节点机器只用修改红字部分为本节点信息

　　　　　　　　vim /usr/lib/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

#Documentation=https://github.com/coreos

[Service]

User=root

After=network.target

After=network-online.target

Wants=network-online.target

#Documentation=https://github.com/coreos

[Service]

User=root

Type=notify

WorkingDirectory=/data/etcd/

ExecStart=/usr/local/bin/etcd \

--name=k8s-master \

--cert-file=/etc/kubernetes/ssl/etcd.pem \

--key-file=/etc/kubernetes/ssl/etcd-key.pem \

--trusted-ca-file=/etc/kubernetes/ssl/ca.pem \

--peer-cert-file=/etc/kubernetes/ssl/etcd.pem \

--peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \

--peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \

--peer-client-cert-auth \

--client-cert-auth \

--listen-peer-urls=https://10.8.8.21:2380 \

--initial-advertise-peer-urls=https://10.8.8.21:2380 \

--listen-client-urls=https://10.8.8.21:2379,https://127.0.0.1:2379 \

--advertise-client-urls=https://10.8.8.21:2379 \

--initial-cluster-token=etcd-cluster-0 \

--initial-cluster=k8s-master=https://10.8.8.21:2380,k8s-node02=https://10.8.8.22:2380,k8s-node03=https://10.8.8.23:2380,k8s-node04=https://10.8.8.24:2380 \

--initial-cluster-state=new

Restart=on-failure

RestartSec=5

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

　　

　　　　3.9.5：检查各节点情况

　　　　　　　　etcdctl 2.2.1版本用如下方法

etcdctl 　--ca-file=/etc/kubernetes/ssl/ca.pem \
    　　　 --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
　　　　    --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
　　　　    --endpoints=https://10.8.8.21:2379,https://10.8.8.22:2379, \
　　　　　　　　https://10.8.8.23:2379,https://10.8.8.24:2379 cluster-health

　　　　　　　　etcdctl 3.3.12版本用如下方法

etcdctl --cacert=/etc/kubernetes/ssl/ca.pem \
   　　 --cert=/etc/kubernetes/ssl/kubernetes.pem \
　　    --key=/etc/kubernetes/ssl/kubernetes-key.pem \
　　    --endpoints=https://10.8.8.21:2379,https://10.8.8.22:2379, \
　　　　　　https://10.8.8.23:2379,https://10.8.8.24:2379 endpoint health

　　　　　　　　回显信息

[root@docker-02 network]# etcdctl --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem --endpoints=https://10.8.8.21:2379,https://10.8.8.22:2379,https://10.8.8.23:2379,https://10.8.8.24:2379 cluster-health
member 9464641f79dde42 is healthy: got healthy result from https://10.8.8.23:2379
member 250662a51b30eed5 is healthy: got healthy result from https://10.8.8.24:2379
member 3255ddeea7f12617 is healthy: got healthy result from https://10.8.8.21:2379
member b488eb3b12837d51 is healthy: got healthy result from https://10.8.8.22:2379
cluster is healthy

　　　　

　　　　3.9.6：export ETCDCTL_API=3　　这个变量要记得设置！！！否则会报错！

[root@docker-02 ~]# etcdctl mkdir /test-etcd

Error:  x509: certificate signed by unknown authority

　　

[root@docker-02 ~]# export ETCDCTL_API=3

[root@docker-02 ~]# systemctl restart etcd

[root@docker-02 ~]# etcdctl member list

9464641f79dde42, started, k8s-node03, https://10.8.8.23:2380, https://10.8.8.23:2379

250662a51b30eed5, started, k8s-node04, https://10.8.8.24:2380, https://10.8.8.24:2379

3255ddeea7f12617, started, k8s-master, https://10.8.8.21:2380, https://10.8.8.21:2379

b488eb3b12837d51, started, k8s-node02, https://10.8.8.22:2380, https://10.8.8.22:2379

　　

四：安装Flannel

　　4.1：下载并安装flannel

　　　　　　wget https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz

　　　　　　tar -zxvf flannel-v0.11.0-linux-amd64.tar.gz -C /zm/flannel

[root@docker-01 zm]# tar -zxvf flannel-v0.11.0-linux-amd64.tar.gz

flanneld

mk-docker-opts.sh

README.md

　　　　　　mv flanneld /usr/bin/

　　　　　　mv mk-docker-opts.sh /usr/bin/

　　　　创建服务文件：

https://blog.csdn.net/bbwangj/article/details/81205244

　　　　　　vim /usr/lib/systemd/system/flanneld.service

[Unit]

Description=flannel

Before=docker.service

[Service]

ExecStart=/usr/bin/flanneld

[Install]

WantedBy=multi-user.target

RequiredBy=docker.service

[Unit]

Description=Flanneld overlay address etcd agent

After=network.target

After=network-online.target

Wants=network-online.target

After=etcd.service

Before=docker.service

[Service]

Type=notify

EnvironmentFile=/etc/sysconfig/flanneld

EnvironmentFile=-/etc/sysconfig/docker-network

ExecStart=/usr/bin/flanneld -etcd-endpoints=${FLANNEL_ETCD} -etcd-prefix=${FLANNEL_ETCD_KEY} $FLANNEL_OPTIONS

ExecStartPost=/usr/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker

Restart=on-failure

[Install]

WantedBy=multi-user.target

RequiredBy=docker.service

　　

　　　　　mkdir -p /etc/systemd/system/flanneld.service.d/ && vim /etc/systemd/system/flanneld.service.d/flannel.conf

[Service]

Environment="FLANNELD_ETCD_ENDPOINTS=http://10.8.8.21:2379"

Environment="FLANNELD_ETCD_PREFIX=/usr/local/flannel/network"

　　　　　　检查是否生效

[root@docker-01 system]# systemctl daemon-reload

[root@docker-01 system]# systemctl show flanneld --property Environment

Environment=FLANNELD_ETCD_ENDPOINTS=http://10.8.8.21:2379 FLANNELD_ETCD_PREFIX=/usr/local/flannel/network

　　　　　　启动flannel

　　　　　　　　systemctl start flanneld

　　　　　　设置IP：

　报错（1）：

[root@docker-02 ~]# etcdctl mk /usr/local/flannel/network/config '{"Network":"10.9.0.0/16","SubnetMin":"10.9.1.0","SubnetMax":"10.9.254.0"}'

Error:  dial tcp 127.0.0.1:4001: connect: connection refused

　　　　　　　修改配置文件：

　　　　　　　　vim /usr/lib/systemd/system/etcd.service　

　　　　　　　　　　--listen-client-urls=https://10.8.8.22:2379,https://127.0.0.1:2379 \

　　　　　　　　　　改为：

　　　　　　　　　　--listen-client-urls=https://10.8.8.22:2379,http://127.0.0.1:2379 \

报错（2）：

[root@docker-02 ~]# systemctl daemon-reload

[root@docker-02 ~]# systemctl stop etcd

[root@docker-02 ~]# systemctl start etcd

[root@docker-02 ~]# etcdctl mk /usr/local/flannel/network/config '{"Network":"10.9.0.0/16","SubnetMin":"10.9.1.0","SubnetMax":"10.9.254.0"}'

Error:  x509: certificate signed by unknown authority

　　执行以下两条命令

etcdctl --endpoints=https://10.8.8.21:2379,https://10.8.8.22:2379,https://10.8.8.23:2379,https://10.8.8.24:2379 \

  --ca-file=/etc/kubernetes/ssl/ca.pem \

  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \

  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \

  mkdir /usr/local/flannel/network


etcdctl --endpoints=https://10.8.8.21:2379,https://10.8.8.22:2379,https://10.8.8.23:2379,https://10.8.8.24:2379 \

  --ca-file=/etc/kubernetes/ssl/ca.pem \

  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \

  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \

  mk /usr/local/flannel/network/config '{"Network":"10.9.0.0/16","SubnetLen":24,"Backend":{"Type":"host-gw"}}’
[root@docker-02 network]# etcdctl --endpoints=https://10.8.8.21:2379,https://10.8.8.22:2379,https://10.8.8.23:2379,https://10.8.8.24:2379   --ca-file=/etc/kubernetes/ssl/ca.pem   --cert-file=/etc/kubernetes/ssl/kubernetes.pem   --key-file=/etc/kubernetes/ssl/kubernetes-key.pem   set /usr/local/flannel/network/config '{"Network":"10.9.0.0/16","Backend":{"Type":"vxlan"}}'

　　

　核对信息

　　　　声明变量：

ETCD_ENDPOINTS=‘https://10.8.8.21:2379,https://10.8.8.22:2379,https://10.8.8.23:2379,https://10.8.8.24:2379’
etcdctl --endpoints=${ETCD_ENDPOINTS} \

--ca-file=/etc/kubernetes/ssl/ca.pem \

--cert-file=/etc/kubernetes/ssl/kubernetes.pem \

--key-file=/etc/kubernetes/ssl/kubernetes-key.pem \

ls /kube-centos/network/subnets

[root@docker-02 /]# etcdctl --endpoints=https://10.8.8.21:2379,https://10.8.8.22:2379,https://10.8.8.23:2379,https://10.8.8.24:2379 --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem get /usr/local/flannel/network/config

{"Network":"10.9.0.0/16","SubnetLen":24,"Backend":{"Type":"host-gw"}}

　　

　

　

　　

感谢：

https://www.cnblogs.com/zhenyuyaodidiao/p/6500830.html

GO:

https://blog.csdn.net/xianchanghuang/article/details/82722064

k8s:

https://www.cnblogs.com/netsa/p/8126155.html

https://blog.csdn.net/qq_36207775/article/details/82343807

https://www.cnblogs.com/xuchenCN/p/9479737.html

etcd：

https://www.jianshu.com/p/98b8fa3e3596

flannel：

https://www.cnblogs.com/ZisZ/p/9212820.html

docker：

https://www.cnblogs.com/ZisZ/p/8962194.html

上一篇：安装Pygame（Python3.6，windows）


下一篇：wxPython缩放图片控件的一个小例子