基于Kubernetes构建企业容器云

2022-06-20 22:12:03

前言

团队成员有DBA、运维、Python开发，由于需要跨部门向公司私有云团队申请虚拟机，此时我在思考能否在现有已申请的虚拟机之上，再进行更加细粒度的资源隔离和划分，让本团队的成员使用，也就是在私有云上构建docker容器云，于是研究下Kubernetes，看一下能否找到一些突破点？Kubernetes （库伯耐踢死），省略了Kubernetes中间8个字母，简称K8S；

什么是Kubernets?

kubernets是一种容器编排的解决方案；（The solution to do container orchestrate）

什么是容器技术?（what's the container technology?）

很久以前app上线流程是：

选择1台物理机（First you should choice a phycisal server）然后在这个物理机上部署您的app（second intalling your application in the phycisal server）；

缺点：

部署慢

效率低

很难迁移

虚拟化技术出现之后（virtualization）

优势 The advantage of Virtualization

Resourse pool like CPU Memory（CPU、内存资源可以动态分配）

Easy to extension: Add more VMs or phycial mechine （便于扩展）

缺点 The disadvantage of virtualization

每个虚拟机都需要安装自己的操作系统 Each pyhsical server requires one OS

Container Technology

如果我开发了一个非常成熟的运维平台，我想把源码买给个世界各地的运维团队；

我的要求：1次打包后能运行在世界各地；

打包（build/packet）

运输（ship）

让它运行在世界各地（run）

容器就像装满了各种杂物的集装箱，方便各种运输工具运输到世界各地；

Lowser:熟练使用各种工具并可以解决领导提出的问题

Senior:深谙各种工具的设计理念

First rate: 各种理念模型在大脑中形成，在某时某地创造出更好的工具；

2.容器技术和虚拟化技术的区别？

虚拟化技术隔离了操作系统层，而容器技术更加彻底它隔离了应用层；

3.容器技术和虚拟化技术的联系？

通常情况下我们线上docker环境是这样的

在虚拟机上搭建docker进行

4.什么是容器编排技术？container orchestration

容器编排就是在docker容器之上再封装1层（Container Orchestration layer 横跨了 docker层和全部主机节点）；

这层的功能是把我们的货物（app）智能调度到1个最佳的集装箱上（docker），并集中管理这些集装箱（docker）（动态创建和销毁）

5.编排技术实现工具

Docker Swarm

Mesos

Kubernetes（最火）

Kubernetes诞生于谷歌公司的1个Brog项目

在2015年7月 released 1.0版本之后把它贡献给了 Cloud Native Computing Foundtion(CFCF)基金会

简称 K8S

Kuberbetes的架构

Kubernetes是一款容器编排工具/容器管理（不仅仅支持docker）管理平台；

Kubernetes的设计采用了传统的分布式架构：由于Master节点和Node节点共同组成了Kubernetes集群；

我们可以通过调用其API or CLI(Comand Iine Interface)命令行kubect的方式和Kubernetes的Master节点进行交互；

1.Kubernetes Master节点

Master节点相当于K8S的大脑

Kubectl：控制Kubernetes的命令行工具

API Server：基于REST api的对外管理接口

Scheduler：调度任务（创建1组容器）到指定Node上执行

Controler Manager：控制管理器Controler-manager里面有1个复制控制器(如果客户端要创建3个容器，我会检查node中创建的是不是3个？)

etcd：etcd集群存储Kubernetes集群中所有的数据

2.Node节点组件

node节点就是干活，一般来说我们需要部署1个APP，用户发起创建容器操作之后master节点会智能调度到node节点上

Kubelet：相当于master装在各个Node节点上的1个agent（管理Pod以及容器、镜像、Volum等，实现对节点进行管理）

Kube-porxy：为容器中的web服务，提供网络代理和负载均衡功能，支持Iptables和LVS

Docker Engin：负责节点容器的真正创建、管理

3.Kubernetes工作流程

假设我现在要使用Kubectl命令行创建1组容器出来，大致都会经历那些流程呢？

A.Kubectl把命令提交到API Server

B.Secheduler获取任务，算出最佳的node

C.API Server通知最佳node上的代理（Kublet）

D.Kublet调用docker-engin进行实际的容器创建工作

实验环境准备

本文主要介绍使用Centos7 安装Kubernetes 1.10.1版本

1.设置VMware Workstation的为NAT模式

2.修改网卡名称并设置静态IP地址

[root@remote network-scripts]# cd /etc/sysconfig/network-scripts/

[root@remote network-scripts]# mv ifcfg-ems33 ifcfg-eth0

重命名该ifcfg配置文件

GRUB_TIMEOUT=

GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"

GRUB_DEFAULT=saved

GRUB_DISABLE_SUBMENU=true

GRUB_TERMINAL_OUTPUT="console"

GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet net.ifnames=0 biosdevname=0"

GRUB_DISABLE_RECOVERY="true"

/etc/default/grub

[root@localhost network-scripts]# grub2-mkconfig -o /boot/grub2/grub.cfg

Generating grub configuration file ...

Found linux image: /boot/vmlinuz-3.10.-.el7.x86_64

Found initrd image: /boot/initramfs-3.10.-.el7.x86_64.img

Found linux image: /boot/vmlinuz--rescue-eb3d805e301049b0a680718a9cc3bec0

Found initrd image: /boot/initramfs--rescue-eb3d805e301049b0a680718a9cc3bec0.img

grub2-mkconfig -o /boot/grub2/grub.cfg

TYPE="Ethernet"

BOOTPROTO="static"

DEFROUTE="yes"

PEERDNS="no"

PEERROUTES="yes"

NAME="eth0"

DEVICE="eth0"

ONBOOT="yes"

IPADDR="192.168.56.11"

NETMASK="255.255.255.0"

GATEWAY="192.168.56.2"

DNS="8.8.8.8"

vim /etc/sysconfig/network-scripts/ifcfg-eth0

重启

3.关闭 firewalld&selinux服务

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

#     enforcing - SELinux security policy is enforced.

#     permissive - SELinux prints warnings instead of enforcing.

#     disabled - No SELinux policy is loaded.

SELINUX=disabled

# SELINUXTYPE= can take one of three two values:

#     targeted - Targeted processes are protected,

#     minimum - Modification of targeted policy. Only selected processes are protected.

#     mls - Multi Level Security protection.

SELINUXTYPE=targeted

vim /etc/sysconfig/selinux 永久关闭selinux

[root@localhost zhanggen]# setenforce

[root@localhost zhanggen]# getenforce

Permissive

[root@localhost zhanggen]# systemctl stop firewalld

4.系统环境准备

[root@linux-node1 ~]# cd /etc/yum.repos.d/

[root@linux-node1 yum.repos.d]# wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

[root@linux-node1 ~]# yum install -y docker-ce

[root@linux-node1 ~]# systemctl start docker

安装docker

5.安装包

Kubernetes的安装包分为

kubernetes.tar.gz 　　　　　　　　　　　　源码包

kubernetes-server-linux-amd64.tar.gz 　　服务端包

kubernetes-node-linux-amd64.tar.gz node节点包

kubernetes-client-linux-amd64.tar.gz 客户端工具包

[root@linux-node1 桌面]# mv k8s-v1.10.1-manual /usr/local/src/

[root@linux-node1 桌面]# cd /usr/src/

[root@linux-node1 src]# ls

debug  kernels

[root@linux-node1 src]# cd /usr/local/src/k8s-v1.10.1-manual/

[root@linux-node1 k8s-v1.10.1-manual]# ls

k8s-v1.10.1

[root@linux-node1 k8s-v1.10.1-manual]# cd k8s-v1.10.1/

[root@linux-node1 k8s-v1.10.1]# ls

cfssl-certinfo_linux-amd64       flannel-v0.10.0-linux-amd64.tar.gz

cfssljson_linux-amd64            kubernetes-client-linux-amd64.tar.gz

cfssl_linux-amd64                kubernetes-node-linux-amd64.tar.gz

cni-plugins-amd64-v0.7.1.tgz     kubernetes-server-linux-amd64.tar.gz

etcd-v3.2.18-linux-amd64.tar.gz  kubernetes.tar.gz

[root@linux-node1 k8s-v1.10.1]#

把Kubernetes的安装包移动到 /usr/local/src/

6.Kubernetes部署目录

配置文件、二进制文件、ssl证书、日志

mkdir -p /opt/kubernetes/{cfg,bin,ssl,log}

7.配置Kubernetes的环境变量

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

        . ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin:/opt/kubernetes/bin

export PATH

vim ~/.bash_profile

source ~/.bash_profile

Centos7安装Kubernetes 1.10.1版本

kubernetes项目是由Go写的，所以都是一堆已经编译好的二进制文件，想要安装它们无需编译；

比较麻烦的地方就是Kubernetes的组件间通信是基于SSL协议，所以要为安装每个组件都要生成、分发证书；

下载安装包----->复制到安装目录---------->写配置文件------->生成证书----------> 分发证书--------->启动

集群CA证书的创建和分发

从Kubernetes 1.8.x开始部署Kubernetes都需要使用TLS证书进行通信加密

本环节主要是安装cfssl，生成证书、并把证书存放在 /usr/local/src/ssl目录下，然后分发到其他节点去；

1.安装 cfssl

[root@linux-node1 ~]# cd /usr/local/src

[root@linux-node1 src]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

[root@linux-node1 src]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

[root@linux-node1 src]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

[root@linux-node1 src]# chmod +x cfssl*

[root@linux-node1 src]# mv cfssl-certinfo_linux-amd64 /opt/kubernetes/bin/cfssl-certinfo

[root@linux-node1 src]# mv cfssljson_linux-amd64  /opt/kubernetes/bin/cfssljson

[root@linux-node1 src]# mv cfssl_linux-amd64  /opt/kubernetes/bin/cfssl

---------------同步到其他node节点---------------------------------------------------------------

[root@linux-node1 bin]# scp /opt/kubernetes/bin/cfssl* 192.168.56.12:/opt/kubernetes/bin

root@192.168.56.12's password:

cfssl-certinfo                                       % 6441KB   .3MB/s   :

cfssljson

下载CFSSL

2.创建证书存放目录

[root@linux-node1 src]# mkdir -p /usr/local/src/ssl

[root@linux-node1 src]# cd usr/local/src/ssl

bash: cd: usr/local/src/ssl: 没有那个文件或目录

[root@linux-node1 src]# cd /usr/local/src/ssl

[root@linux-node1 ssl]# pwd

/usr/local/src/ssl

mkdir -p /usr/local/src/ssl

3.编辑证书配置文件

{

  "signing": {

    "default": {

      "expiry": "8760h"

    },

    "profiles": {

      "kubernetes": {

        "usages": [

            "signing",

            "key encipherment",

            "server auth",

            "client auth"

        ],

        "expiry": "8760h"

      }

    }

  }

}

vim /usr/local/src/ssl/ca-config.json

4.编辑ca-csr配置文件

{

  "CN": "kubernetes",

  "key": {

    "algo": "rsa",

    "size":

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "System"

    }

  ]

}

vim /usr/local/src/ssl/ca-csr.json

5.生成证书

[root@linux-node1 ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca

// :: [INFO] generating a new CA key and certificate from CSR

// :: [INFO] generate received request

// :: [INFO] received CSR

// :: [INFO] generating key: rsa-

// :: [INFO] encoded CSR

// :: [INFO] signed certificate with serial number

[root@linux-node1 ssl

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

6.把在/usr/local/ssl目录中生成的证书 cp 到本机和其他节点的 /opt/kubernetes/ssl

[root@linux-node1 ssl]#  cp ca.csr ca.pem ca-key.pem ca-config.json /opt/kubernetes/ssl

[root@linux-node1 ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 192.168.56.12:/opt/kubernetes/ssl

root@192.168.56.12's password:

ca.csr                                               %      .0KB/s   :

ca.pem                                               %      .3KB/s   :

ca-key.pem                                           %      .6KB/s   :

ca-config.json                                       %       .3KB/s   :

[root@linux-node1 ssl]#

cp ca.csr ca.pem ca-key.pem ca-config.json /opt/kubernetes/ssl

ETCD集群部署

ETCD集群类似于Hadoop中的zookper分布式协同服务，
也可以是一种分布式的专门为分布式系统设计的K-V存储。

0.准备etcd软件包

wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz

[root@linux-node1 src]# tar zxf etcd-v3.2.18-linux-amd64.tar.gz

[root@linux-node1 src]# cd etcd-v3.2.18-linux-amd64

[root@linux-node1 etcd-v3.2.18-linux-amd64]# cp etcd etcdctl /opt/kubernetes/bin/

[root@linux-node1 etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 192.168.56.12:/opt/kubernetes/bin/

[root@linux-node1 etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 192.168.56.13:/opt/kubernetes/bin/

1.创建 etcd 证书签名配置文件：

etcd集群的通信也需要 CA证书进行认证，所以要使用搭建好的自签名证书给，生成证书，并分发给其他节点；

{

  "CN": "etcd",

  "hosts": [

    "127.0.0.1",

"192.168.56.11",

"192.168.56.12",

"192.168.56.13"

  ],

  "key": {

    "algo": "rsa",

    "size":

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "System"

    }

  ]

}

[root@linux-node1 ~]# vim etcd-csr.json

2.生成 etcd 证书和私钥：

[root@linux-node1 ~]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \

  -ca-key=/opt/kubernetes/ssl/ca-key.pem \

  -config=/opt/kubernetes/ssl/ca-config.json \

  -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

会生成以下证书文件

[root@k8s-master ~]# ls -l etcd*

-rw-r--r--  root root  Mar   : etcd.csr

-rw-r--r--  root root   Mar   : etcd-csr.json

-rw-------  root root  Mar   : etcd-key.pem

-rw-r--r--  root root  Mar   : etcd.pem

cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \ -ca-key=/opt/kubernetes/ssl/ca-key.pem \ -config=/opt/kubernetes/ssl/ca-config.json \ -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

3.将证书移动到/opt/kubernetes/ssl目录下

[root@k8s-master ~]# cp etcd*.pem /opt/kubernetes/ssl

[root@linux-node1 ~]# scp etcd*.pem 192.168.56.12:/opt/kubernetes/ssl

[root@linux-node1 ~]# scp etcd*.pem 192.168.56.13:/opt/kubernetes/ssl

[root@k8s-master ~]# rm -f etcd.csr etcd-csr.json

cp etcd*.pem /opt/kubernetes/ssl

4.设置ETCD配置文件并发发给其他节点

[root@linux-node1 ~]# vim /opt/kubernetes/cfg/etcd.conf

#[member]

ETCD_NAME="etcd-node1"

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

#ETCD_SNAPSHOT_COUNTER=""

#ETCD_HEARTBEAT_INTERVAL=""

#ETCD_ELECTION_TIMEOUT=""

ETCD_LISTEN_PEER_URLS="https://192.168.56.11:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.56.11:2379,https://127.0.0.1:2379"

#ETCD_MAX_SNAPSHOTS=""

#ETCD_MAX_WALS=""

#ETCD_CORS=""

#[cluster]

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.56.11:2380"

# if you use different ETCD_NAME (e.g. test),

# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."

ETCD_INITIAL_CLUSTER="etcd-node1=https://192.168.56.11:2380,etcd-node2=https://192.168.56.12:2380,etcd-node3=https://192.168.56.13:2380"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.56.11:2379"

#[security]

CLIENT_CERT_AUTH="true"

ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"

ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"

ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"

PEER_CLIENT_CERT_AUTH="true"

ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"

ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"

ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"

vim /opt/kubernetes/cfg/etcd.conf

5.创建ETCD系统服务

[root@linux-node1 ~]# vim /etc/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

[Service]

Type=simple

WorkingDirectory=/var/lib/etcd

EnvironmentFile=-/opt/kubernetes/cfg/etcd.conf

# set GOMAXPROCS to number of processors

ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/kubernetes/bin/etcd"

Type=notify

[Install]

WantedBy=multi-user.target

vim /etc/systemd/system/etcd.service

6.重新加载系统服务

[root@linux-node1 ~]# systemctl daemon-reload

[root@linux-node1 ~]# systemctl enable etcd

# scp /opt/kubernetes/cfg/etcd.conf 192.168.56.12:/opt/kubernetes/cfg/

# scp /etc/systemd/system/etcd.service 192.168.56.12:/etc/systemd/system/

# scp /opt/kubernetes/cfg/etcd.conf 192.168.56.13:/opt/kubernetes/cfg/

# scp /etc/systemd/system/etcd.service 192.168.56.13:/etc/systemd/system/

在所有节点上创建etcd存储目录并启动etcd

[root@linux-node1 ~]# mkdir /var/lib/etcd

[root@linux-node1 ~]# systemctl start etcd

[root@linux-node1 ~]# systemctl status etcd

systemctl daemon-reload

7.验证集群

[root@linux-node1 ~]# etcdctl --endpoints=https://192.168.56.11:2379 \

  --ca-file=/opt/kubernetes/ssl/ca.pem \

  --cert-file=/opt/kubernetes/ssl/etcd.pem \

  --key-file=/opt/kubernetes/ssl/etcd-key.pem cluster-health

member 435fb0a8da627a4c is healthy: got healthy result from https://192.168.56.12:2379

member 6566e06d7343e1bb is healthy: got healthy result from https://192.168.56.11:2379

member ce7b884e428b6c8c is healthy: got healthy result from https://192.168.56.13:2379

cluster is healthy

etcdctl --endpoints=https://192.168.56.11:2379 \ --ca-file=/opt/kubernetes/ssl/ca.pem \ --cert-file=/opt/kubernetes/ssl/etcd.pem \ --key-file=/opt/kubernetes/ssl/etcd-key.pem cluster-health

Kubernetes Master节点部署

Kubernetes的Master节点主要包含3个服务：

API Server：Kubernetes组件间的数据交换和通信枢纽，
只有Apiserver才可以操作etcd集群，其他模块只能通过ApiServer间接查询或修改数据；

Scheduler：分配调度Pod（Kubernetes中的逻辑单位，包含容器）到集群的node节点

Controller-manager:有一系列的控制器组成，它通过ApiServer监控整个集群的状态，并确保集群处在预期的工作状态；

部署Kubernetes API服务部署

0.准备软件包

[root@linux-node1 ~]# cd /usr/local/src/kubernetes

[root@linux-node1 kubernetes]# cp server/bin/kube-apiserver /opt/kubernetes/bin/

[root@linux-node1 kubernetes]# cp server/bin/kube-controller-manager /opt/kubernetes/bin/

[root@linux-node1 kubernetes]# cp server/bin/kube-scheduler /opt/kubernetes/bin/

cp server/bin/kube-apiserver /opt/kubernetes/bin/

1.创建生成CSR的 JSON 配置文件

{

  "CN": "kubernetes",

  "hosts": [

    "127.0.0.1",

    "192.168.56.11",

    "10.1.0.1",

    "kubernetes",

    "kubernetes.default",

    "kubernetes.default.svc",

    "kubernetes.default.svc.cluster",

    "kubernetes.default.svc.cluster.local"

  ],

  "key": {

    "algo": "rsa",

    "size":

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "System"

    }

  ]

}

ocal/src/ssl/kubernetes-csr.json

2.生成 kubernetes 证书和私钥

生成kubernetes 证书和私钥存放在/opt/kubernetes/ssl/目录下

 [root@linux-node1 src]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \

   -ca-key=/opt/kubernetes/ssl/ca-key.pem \

   -config=/opt/kubernetes/ssl/ca-config.json \

   -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

[root@linux-node1 src]# cp kubernetes*.pem /opt/kubernetes/ssl/

[root@linux-node1 ~]# scp kubernetes*.pem 192.168.56.12:/opt/kubernetes/ssl/

[root@linux-node1 ~]# scp kubernetes*.pem 192.168.56.13:/opt/kubernetes/ssl/

cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \ -ca-key=/opt/kubernetes/ssl/ca-key.pem \ -config=/opt/kubernetes/ssl/ca-config.json \ -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

3.创建 kube-apiserver 使用的客户端 token 文件

在/opt/kubernetes/ssl/bootstrap-token.csv生成token文件

[root@linux-node1 ~]#  head -c  /dev/urandom | od -An -t x | tr -d ' '

ad6d5bb607a186796d8861557df0d17f

[root@linux-node1 ~]# vim /opt/kubernetes/ssl/ bootstrap-token.csv

ad6d5bb607a186796d8861557df0d17f,kubelet-bootstrap,,"system:kubelet-bootstrap"

vim /opt/kubernetes/ssl/bootstrap-token.csv

4.创建基础用户名/密码认证配置

在/opt/kubernetes/ssl/basic-auth.csv用户密码认证文件

admin,admin,

readonly,readonly,

vim /opt/kubernetes/ssl/basic-auth.csv

5.设置Kubernetes API Server的启动项

[root@linux-node1 ~]# vim /usr/lib/systemd/system/kube-apiserver.service

[Unit]

Description=Kubernetes API Server

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=network.target

[Service]

ExecStart=/opt/kubernetes/bin/kube-apiserver \

  --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \

  --bind-address=192.168.56.11 \

  --insecure-bind-address=127.0.0.1 \

  --authorization-mode=Node,RBAC \

  --runtime-config=rbac.authorization.k8s.io/v1 \

  --kubelet-https=true \

  --anonymous-auth=false \

  --basic-auth-file=/opt/kubernetes/ssl/basic-auth.csv \

  --enable-bootstrap-token-auth \

  --token-auth-file=/opt/kubernetes/ssl/bootstrap-token.csv \

  --service-cluster-ip-range=10.1.0.0/ \

  --service-node-port-range=- \

  --tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \

  --tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \

  --client-ca-file=/opt/kubernetes/ssl/ca.pem \

  --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \

  --etcd-cafile=/opt/kubernetes/ssl/ca.pem \

  --etcd-certfile=/opt/kubernetes/ssl/kubernetes.pem \

  --etcd-keyfile=/opt/kubernetes/ssl/kubernetes-key.pem \

  --etcd-servers=https://192.168.56.11:2379,https://192.168.56.12:2379,https://192.168.56.13:2379 \

  --enable-swagger-ui=true \

  --allow-privileged=true \

  --audit-log-maxage= \

  --audit-log-maxbackup= \

  --audit-log-maxsize= \

  --audit-log-path=/opt/kubernetes/log/api-audit.log \

  --event-ttl=1h \

  --v= \

  --logtostderr=false \

  --log-dir=/opt/kubernetes/log

Restart=on-failure

RestartSec=

Type=notify

LimitNOFILE=

[Install]

WantedBy=multi-user.target

vim /usr/lib/systemd/system/kube-apiserver.service

ps:

192.168.56.11:6443端口是kubernetes-api对外的socket（需要认证）
127.0.0.1:8080端口是给Controller-manager、Scheduler节点内部通信使用；（无需认证）

6.启动API Server服务

[root@linux-node1 ~]# systemctl daemon-reload

[root@linux-node1 ~]# systemctl enable kube-apiserver

[root@linux-node1 ~]# systemctl start kube-apiserver

查看API Server服务状态

[root@linux-node1 ~]# systemctl status kube-apiserver

systemctl start kube-apiserver

部署Controller Manager服务

1.设置 kube-controller-manager的启动项

[root@linux-node1 ~]# vim /usr/lib/systemd/system/kube-controller-manager.service

[Unit]

Description=Kubernetes Controller Manager

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]

ExecStart=/opt/kubernetes/bin/kube-controller-manager \

  --address=127.0.0.1 \

  --master=http://127.0.0.1:8080 \

  --allocate-node-cidrs=true \

  --service-cluster-ip-range=10.1.0.0/ \

  --cluster-cidr=10.2.0.0/ \

  --cluster-name=kubernetes \

  --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \

  --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \

  --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \

  --root-ca-file=/opt/kubernetes/ssl/ca.pem \

  --leader-elect=true \

  --v= \

  --logtostderr=false \

  --log-dir=/opt/kubernetes/log

Restart=on-failure

RestartSec=

[Install]

WantedBy=multi-user.target

vim /usr/lib/systemd/system/kube-controller-manager.service

ps:kube-controller-manager监听在内网127.0.0.1:10252

2.启动Controller Manager服务

[root@linux-node1 ~]# systemctl daemon-reload

[root@linux-node1 scripts]# systemctl enable kube-controller-manager

[root@linux-node1 scripts]# systemctl start kube-controller-manager

systemctl start kube-controller-manager

部署Kubernetes-Scheduler

0.设置Kubernetes-Scheduler的启动项

[root@linux-node1 ~]# vim /usr/lib/systemd/system/kube-scheduler.service

[Unit]

Description=Kubernetes Scheduler

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]

ExecStart=/opt/kubernetes/bin/kube-scheduler \

  --address=127.0.0.1 \

  --master=http://127.0.0.1:8080 \

  --leader-elect=true \

  --v= \

  --logtostderr=false \

  --log-dir=/opt/kubernetes/log

Restart=on-failure

RestartSec=

[Install]

WantedBy=multi-user.target

vim /usr/lib/systemd/system/kube-scheduler.service

2.启动Kubernetes-Scheduler

[root@linux-node1 ~]# systemctl daemon-reload

[root@linux-node1 scripts]# systemctl enable kube-scheduler

[root@linux-node1 scripts]# systemctl start kube-scheduler

[root@linux-node1 scripts]# systemctl status kube-scheduler

systemctl start kube-scheduler

部署kubectl 命令行工具

管理K8S集群除了可以使用程序调用 Api-Server，还可以通过kubectl命令行进行调用

1.准备二进制命令包

[root@linux-node1 ~]# cd /usr/local/src/kubernetes/client/bin

[root@linux-node1 bin]# cp kubectl /opt/kubernetes/bin/

cp kubectl /opt/kubernetes/bin/

2.创建 admin 证书签名请求

[root@linux-node1 ~]# cd /usr/local/src/ssl/

[root@linux-node1 ssl]# vim admin-csr.json

{

  "CN": "admin",

  "hosts": [],

  "key": {

    "algo": "rsa",

    "size":

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "system:masters",

      "OU": "System"

    }

  ]

}

vim admin-csr.json

3.生成 admin 证书和私钥

[root@linux-node1 ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \

   -ca-key=/opt/kubernetes/ssl/ca-key.pem \

   -config=/opt/kubernetes/ssl/ca-config.json \

   -profile=kubernetes admin-csr.json | cfssljson -bare admin

[root@linux-node1 ssl]# ls -l admin*

-rw-r--r--  root root  Mar   : admin.csr

-rw-r--r--  root root   Mar   : admin-csr.json

-rw-------  root root  Mar   : admin-key.pem

-rw-r--r--  root root  Mar   : admin.pem

[root@linux-node1 src]# mv admin*.pem /opt/kubernetes/ssl/

cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \ -ca-key=/opt/kubernetes/ssl/ca-key.pem \ -config=/opt/kubernetes/ssl/ca-config.json \ -profile=kubernetes admin-csr.json | cfssljson -bare admin

以下操作都是为了帮你在当前用户加目录下生成1个config文件，该文件在kuberctl 和api通信就使用这个文件进行加密

apiVersion: v1

clusters:

- cluster:

    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVSTAzZE5wWjExNmVWc0hEdUY2RDZjb29sTWRVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU1TURReU9EQXpNRFF3TUZvWERUSTBNRFF5TmpBek1EUXdNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBeG10WHBRZVpxUUxaRnpyQ0dwRFUKUE9JNDlFclZSd3ZaV2toMmVYSzZsMVpCTXZSWGY0QTczbHU5UkdpanErZ3EwRGNsSmMwWHRoSXNJMlVUNFJ4Qwp5QlpGNzIyVmkrZHlDTHNnUHFYSFFTYUVVVDdQeGhIOUt6c0dMbWV6M0YwUUJCbWNXNU00clJ5U2E1VDYxbkpnCm5QWmQvdjVFZ1VEMDI2TldFcWM2aWp0blVvQ1hFdDFteDRhbWE1YTk1OFBQTm5OSXVJUlFSUnp6Z1U0L3NFVGQKSUpPR2l2N043RysrdWU0Z3pLemZPRFJUU0FDK1FUVnB6c0RNN05sY29ITWpnOGNSL0ZxYWVjQXJoZ05xckxPbQpMamoxUjZDR0d3a2FnUG40SWhGQVkxamJQVXBHSnRQSkN4RlUzY0RQeXQrVEZwblFOVmxCYmMrWE5HTGo0QkFUCkx3SURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVM1ZqU2gyVWp5akFicTgranB5dTM2OThua1lJd0h3WURWUjBqQkJnd0ZvQVUzVmpTaDJVagp5akFicTgranB5dTM2OThua1lJd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMaHdXWExjMmMyREprRW41Y2VrClhxQlJqbXNIVVhKYzhiQWN5aXBrL0Y5OXBKaDRoYjJCMXcvd011aGdTZStRWFptSFVhZUdWbFZJUGhuTkMxM00KallnajZGenM4RGJXbVQ4TWViVHJtVXVjMSttMnQ1clpSdENDeGZocHdhSmJHcURPU29vYUpBVWdvdWdVS00vQQppU2t2N3J6OC9BYjdramFNY2ZFRzJsbmEzdkNXRXhUTW9PL2V3RkR3THZnWUgxMXcybU9ZSjRSV1gxaUFlNVlxCnAzclRscVdQNmM3U1RsNkpyem1EOVUwWkpkMzQ0SmNxcDFORkNpUzJYcGZIdFMySkhxRVVVN1Y4Zi81RzRkeWIKcmRQYVRNNGJsZzlaWUMvcGtlbUZoRjIvRm50Y3hrQVhzWXR4eUVzOVdHeHZyK0JvRnJqeXBpdzlhMDNLeVlDTwo3NGM9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K

    server: https://192.168.56.11:6443

  name: kubernetes

contexts:

- context:

    cluster: kubernetes

    user: admin

  name: kubernetes

current-context: kubernetes

kind: Config

preferences: {}

users:

- name: admin

  user:

    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQzVENDQXNXZ0F3SUJBZ0lVRlNwVzZHcCt6YmtmaloyeDZFMFpGS01mTkpnd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU1TURReU9URTBNall3TUZvWERUSXdNRFF5T0RFME1qWXdNRm93YXpFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RnpBVgpCZ05WQkFvVERuTjVjM1JsYlRwdFlYTjBaWEp6TVE4d0RRWURWUVFMRXdaVGVYTjBaVzB4RGpBTUJnTlZCQU1UCkJXRmtiV2x1TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF6bzFtSEtrdW54OWMKTVFEUlBuTVpMUEd4S250UHFPcjhKKzRGNlRtWmlnTHg5aElXR2lCSHNxQUZPUEhJU3VtTnc1RWE4S29aNDMzRQp6UnM1dnpZUnVRRGE0MDVRcHJrNHhOMzMzeFFSZ204Snk4c2tocUU3WjlqTWxYTGpHVEhtM0txWWIrNXE0SWdZCkUwemNLNVNkdHdvSTVUcVZpYjVZYkNNbVdkLzhEVGlLMW11cGhmZ1dWVytiUWdnLzhrdmVxM0JqbXZhTmJTZTIKRTVrVjBaN01DV2lqQUFML1hkL2YvQjE1MklJOUdwTnNxKy9sQWFFZm9tM0N4bHlPR285YzJMamhqcG9MdUpXTgp5U2hYN3duQjIzUlp0L3ZBb2pqQlUweWYwN0pkNE5DcFVlRGg5TXVUS01FOGhQT21TbzdSaHRPR0hPM2NDK2JkCng3NUFQcE9tRFFJREFRQUJvMzh3ZlRBT0JnTlZIUThCQWY4RUJBTUNCYUF3SFFZRFZSMGxCQll3RkFZSUt3WUIKQlFVSEF3RUdDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIvd1FDTUFBd0hRWURWUjBPQkJZRUZPdDFlNXo3OS9rdwo5QVU4dEhhcm9TOWs4bnpCTUI4R0ExVWRJd1FZTUJhQUZOMVkwb2RsSThvd0c2dlBvNmNydCt2Zko1R0NNQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFDdGpJYWZaeWlYWHhyU2lzM3Z4TWk2MGJDeXRTekkvWEhzcFhyWUR3bnUKRlBxNkJDSzQrbXpvUDREcVhWVTR1NG5PZzhncDFXa0QyVk12YWphUjlUOFRDbUt0NEhFSWR4ai9uSTU0ekJZUAp2eTRrRXVrbUhyWXFPTlRBT1pKMzAzQWxFQUNZRFFQNTFTU3poaThDak15N3NWdmdlUkdCOVo1WVV1TitCaFFpCmxsWm5nVXBHVlBNT0dhMUVYdlY5T01GajVWQ2taa2l1dk5NVzFOWFUvTzNXQVNoWUhhK21NNzJJTHVWdDhJOWMKRlFqTk9rUjNFamU1eHpvOTNEVFMwZGZtbFRXc3dPK0grM3pEcVFKZU9KSzNyYnROeEM2Z0ZLcUl3dHZsOUF0awpkQ2N0YzcveXdTT0NObVc4WDc1azRvaXhNY3RvamVFc3FjMWw5SnBMR1dwMwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBem8xbUhLa3VueDljTVFEUlBuTVpMUEd4S250UHFPcjhKKzRGNlRtWmlnTHg5aElXCkdpQkhzcUFGT1BISVN1bU53NUVhOEtvWjQzM0V6UnM1dnpZUnVRRGE0MDVRcHJrNHhOMzMzeFFSZ204Snk4c2sKaHFFN1o5ak1sWExqR1RIbTNLcVliKzVxNElnWUUwemNLNVNkdHdvSTVUcVZpYjVZYkNNbVdkLzhEVGlLMW11cApoZmdXVlcrYlFnZy84a3ZlcTNCam12YU5iU2UyRTVrVjBaN01DV2lqQUFML1hkL2YvQjE1MklJOUdwTnNxKy9sCkFhRWZvbTNDeGx5T0dvOWMyTGpoanBvTHVKV055U2hYN3duQjIzUlp0L3ZBb2pqQlUweWYwN0pkNE5DcFVlRGgKOU11VEtNRThoUE9tU283Umh0T0dITzNjQytiZHg3NUFQcE9tRFFJREFRQUJBb0lCQUJxbllucm1WZzdRbEN2NgpxQVhBQW9xck1hcUN1UmZhSXVuZ0xFRVpYcmZSZzNtMmdjV2pUcjA5S3c5YkcvYVd4dVZxcnloSk93Z0JMY2t0Cjd6aStlSEVBTEQ3UzExTjhhVmYyTU10SG9xN0xOMTlsK25PcEVLcG83cFdHZXNuQWg4TUgvSjNORFZ1bUZEMUIKV05RQzNJdEhMeml0WTZpZnVIZFQzZG9STGt4aU9TTUZaQ3l0cG9hRkJtc0I4dVBtSXcvajduTHRpWklGRko0eApaZ1pEaXRWTnM5c25xMFNlWVpKZVJvQm5sS0w1QmhpL2VJZDNnd3dONlplZ0ptbnFlVytydXZqUVEzV2pyNXVPCnNncFVWbDZVOW1aN1ZNYXVVWEdZVHA2Qk1UaFFKWlVUQVl5bzZGMzE3blZaRndWVWJ3WnZEU0VXM0I0VzVNS0cKZHRsY3ZpRUNnWUVBNElNOFhxNHozdTlTeVBYZFZEdER4MUNkbkZVak5TMHZ1R0E4ZGpFUGFNbng1STVpTnpFeQpwTjhrUnpaWEVHSW55b3pFVzlzOStHSjJwb3VxZnRJV3pOYUxmZUFRbzBzU3lSTmtuclNsLzlLaERFdWN6alNECkM0dURsb3poTGRmb3VWZi9BdE02SldkWDVZN3ByQXdLcFhkSjJCNU5vQXRxeGxqbnpjeU9zSWtDZ1lFQTY0VlQKMHNHc2tabkVwUUx1ZVIxSkZYSm5qSTA4R01rVFhlU1U0KzJrZzRWdkJTRmxBNzBiOUtaL2l4L3BOcUhveFVYbQpSSTdwUE1GWUJJcU93MFZxdjViN3daQzJnTXgzUWczUUpaRURZb2hZblpFZ3M5UUZyb1NmT2M1dm1seForSG02CmNLN2pkalI5OFFBcHdDd1JVR08vc2lyeFV0UGRnZW16TTFZSUFHVUNnWUFBaFUxbWl0RGorM29kclRST05iVDYKaVYxVU4zNVZhVDFyR0E0TDJDRkpCTzdpc05IWmZ1dTNKaTFYWFBEbXdOT0d6THpIMmNKVENTZHRTM1doeGFyMwozcWVFS3pqZXFCWHJFWGh5UmNqOHh1aEl0d1F1RmtFWGpjTklYaHRIbC9DYVBYSUI5NnR5MnNLQmJjdHM4cm96Cm1Bczd6Ull4QU5YR2ovNDVvL2ZRd1FLQmdRQ0p0Q1MwZjBTVXhPRXkwYW40Nm1TR3c2TkRqSGhzelhRalc3aXEKSTVJaXkrdURobWozYktSaHdNK2wybnlTMHN1MFBCQk1XWHFKYVVvN0xZQVhNQWtnQi9rZXphdkhWc1VMdStQTgpjeUlWaEQ2N3NkVDdENlpheVhRSGFtbmFTTkRaOU9KTGJCWFdVUTZwMER5WS9hc0o0Nmg3Vlp4UG4weE4zd1JGCnFiRjMrUUtCZ1FEYXhTRzgvU1RsNmkyeitIY0F2c25lem1iVXJ6QmFKc3BJM3NOMG9tQ1IvZVJXcTM5dVk5VTEKODk2VXVBQ3RDTnk4VS9EWkY3RnNsakpOK3FkYjYrVXNlN0FZRTVEejlxTDl1SFI2am80dHZoNzFTUkFreE9nRgptSCtDMnJkbDNzdStPU2ZHTlQ4dzFId2dBaDFIdExJRHVsNTFvekduTGFOWWNzNjZ0M0lQbHc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

[root@linux-node1 ~]# ^C

[root@linux-node1 ~]# pwd

/root

[root@linux-node1 ~]# cat /root/.kube/config

apiVersion: v1

clusters:

- cluster:

    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR2akNDQXFhZ0F3SUJBZ0lVSTAzZE5wWjExNmVWc0hEdUY2RDZjb29sTWRVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU1TURReU9EQXpNRFF3TUZvWERUSTBNRFF5TmpBek1EUXdNRm93WlRFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RERBSwpCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBeG10WHBRZVpxUUxaRnpyQ0dwRFUKUE9JNDlFclZSd3ZaV2toMmVYSzZsMVpCTXZSWGY0QTczbHU5UkdpanErZ3EwRGNsSmMwWHRoSXNJMlVUNFJ4Qwp5QlpGNzIyVmkrZHlDTHNnUHFYSFFTYUVVVDdQeGhIOUt6c0dMbWV6M0YwUUJCbWNXNU00clJ5U2E1VDYxbkpnCm5QWmQvdjVFZ1VEMDI2TldFcWM2aWp0blVvQ1hFdDFteDRhbWE1YTk1OFBQTm5OSXVJUlFSUnp6Z1U0L3NFVGQKSUpPR2l2N043RysrdWU0Z3pLemZPRFJUU0FDK1FUVnB6c0RNN05sY29ITWpnOGNSL0ZxYWVjQXJoZ05xckxPbQpMamoxUjZDR0d3a2FnUG40SWhGQVkxamJQVXBHSnRQSkN4RlUzY0RQeXQrVEZwblFOVmxCYmMrWE5HTGo0QkFUCkx3SURBUUFCbzJZd1pEQU9CZ05WSFE4QkFmOEVCQU1DQVFZd0VnWURWUjBUQVFIL0JBZ3dCZ0VCL3dJQkFqQWQKQmdOVkhRNEVGZ1FVM1ZqU2gyVWp5akFicTgranB5dTM2OThua1lJd0h3WURWUjBqQkJnd0ZvQVUzVmpTaDJVagp5akFicTgranB5dTM2OThua1lJd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMaHdXWExjMmMyREprRW41Y2VrClhxQlJqbXNIVVhKYzhiQWN5aXBrL0Y5OXBKaDRoYjJCMXcvd011aGdTZStRWFptSFVhZUdWbFZJUGhuTkMxM00KallnajZGenM4RGJXbVQ4TWViVHJtVXVjMSttMnQ1clpSdENDeGZocHdhSmJHcURPU29vYUpBVWdvdWdVS00vQQppU2t2N3J6OC9BYjdramFNY2ZFRzJsbmEzdkNXRXhUTW9PL2V3RkR3THZnWUgxMXcybU9ZSjRSV1gxaUFlNVlxCnAzclRscVdQNmM3U1RsNkpyem1EOVUwWkpkMzQ0SmNxcDFORkNpUzJYcGZIdFMySkhxRVVVN1Y4Zi81RzRkeWIKcmRQYVRNNGJsZzlaWUMvcGtlbUZoRjIvRm50Y3hrQVhzWXR4eUVzOVdHeHZyK0JvRnJqeXBpdzlhMDNLeVlDTwo3NGM9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K

    server: https://192.168.56.11:6443

  name: kubernetes

contexts:

- context:

    cluster: kubernetes

    user: admin

  name: kubernetes

current-context: kubernetes

kind: Config

preferences: {}

users:

- name: admin

  user:

    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQzVENDQXNXZ0F3SUJBZ0lVRlNwVzZHcCt6YmtmaloyeDZFMFpGS01mTkpnd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1pURUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVQTUEwR0ExVUVDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByCmRXSmxjbTVsZEdWek1CNFhEVEU1TURReU9URTBNall3TUZvWERUSXdNRFF5T0RFME1qWXdNRm93YXpFTE1Ba0cKQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbGFVcHBibWN4RnpBVgpCZ05WQkFvVERuTjVjM1JsYlRwdFlYTjBaWEp6TVE4d0RRWURWUVFMRXdaVGVYTjBaVzB4RGpBTUJnTlZCQU1UCkJXRmtiV2x1TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF6bzFtSEtrdW54OWMKTVFEUlBuTVpMUEd4S250UHFPcjhKKzRGNlRtWmlnTHg5aElXR2lCSHNxQUZPUEhJU3VtTnc1RWE4S29aNDMzRQp6UnM1dnpZUnVRRGE0MDVRcHJrNHhOMzMzeFFSZ204Snk4c2tocUU3WjlqTWxYTGpHVEhtM0txWWIrNXE0SWdZCkUwemNLNVNkdHdvSTVUcVZpYjVZYkNNbVdkLzhEVGlLMW11cGhmZ1dWVytiUWdnLzhrdmVxM0JqbXZhTmJTZTIKRTVrVjBaN01DV2lqQUFML1hkL2YvQjE1MklJOUdwTnNxKy9sQWFFZm9tM0N4bHlPR285YzJMamhqcG9MdUpXTgp5U2hYN3duQjIzUlp0L3ZBb2pqQlUweWYwN0pkNE5DcFVlRGg5TXVUS01FOGhQT21TbzdSaHRPR0hPM2NDK2JkCng3NUFQcE9tRFFJREFRQUJvMzh3ZlRBT0JnTlZIUThCQWY4RUJBTUNCYUF3SFFZRFZSMGxCQll3RkFZSUt3WUIKQlFVSEF3RUdDQ3NHQVFVRkJ3TUNNQXdHQTFVZEV3RUIvd1FDTUFBd0hRWURWUjBPQkJZRUZPdDFlNXo3OS9rdwo5QVU4dEhhcm9TOWs4bnpCTUI4R0ExVWRJd1FZTUJhQUZOMVkwb2RsSThvd0c2dlBvNmNydCt2Zko1R0NNQTBHCkNTcUdTSWIzRFFFQkN3VUFBNElCQVFDdGpJYWZaeWlYWHhyU2lzM3Z4TWk2MGJDeXRTekkvWEhzcFhyWUR3bnUKRlBxNkJDSzQrbXpvUDREcVhWVTR1NG5PZzhncDFXa0QyVk12YWphUjlUOFRDbUt0NEhFSWR4ai9uSTU0ekJZUAp2eTRrRXVrbUhyWXFPTlRBT1pKMzAzQWxFQUNZRFFQNTFTU3poaThDak15N3NWdmdlUkdCOVo1WVV1TitCaFFpCmxsWm5nVXBHVlBNT0dhMUVYdlY5T01GajVWQ2taa2l1dk5NVzFOWFUvTzNXQVNoWUhhK21NNzJJTHVWdDhJOWMKRlFqTk9rUjNFamU1eHpvOTNEVFMwZGZtbFRXc3dPK0grM3pEcVFKZU9KSzNyYnROeEM2Z0ZLcUl3dHZsOUF0awpkQ2N0YzcveXdTT0NObVc4WDc1azRvaXhNY3RvamVFc3FjMWw5SnBMR1dwMwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

    client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBem8xbUhLa3VueDljTVFEUlBuTVpMUEd4S250UHFPcjhKKzRGNlRtWmlnTHg5aElXCkdpQkhzcUFGT1BISVN1bU53NUVhOEtvWjQzM0V6UnM1dnpZUnVRRGE0MDVRcHJrNHhOMzMzeFFSZ204Snk4c2sKaHFFN1o5ak1sWExqR1RIbTNLcVliKzVxNElnWUUwemNLNVNkdHdvSTVUcVZpYjVZYkNNbVdkLzhEVGlLMW11cApoZmdXVlcrYlFnZy84a3ZlcTNCam12YU5iU2UyRTVrVjBaN01DV2lqQUFML1hkL2YvQjE1MklJOUdwTnNxKy9sCkFhRWZvbTNDeGx5T0dvOWMyTGpoanBvTHVKV055U2hYN3duQjIzUlp0L3ZBb2pqQlUweWYwN0pkNE5DcFVlRGgKOU11VEtNRThoUE9tU283Umh0T0dITzNjQytiZHg3NUFQcE9tRFFJREFRQUJBb0lCQUJxbllucm1WZzdRbEN2NgpxQVhBQW9xck1hcUN1UmZhSXVuZ0xFRVpYcmZSZzNtMmdjV2pUcjA5S3c5YkcvYVd4dVZxcnloSk93Z0JMY2t0Cjd6aStlSEVBTEQ3UzExTjhhVmYyTU10SG9xN0xOMTlsK25PcEVLcG83cFdHZXNuQWg4TUgvSjNORFZ1bUZEMUIKV05RQzNJdEhMeml0WTZpZnVIZFQzZG9STGt4aU9TTUZaQ3l0cG9hRkJtc0I4dVBtSXcvajduTHRpWklGRko0eApaZ1pEaXRWTnM5c25xMFNlWVpKZVJvQm5sS0w1QmhpL2VJZDNnd3dONlplZ0ptbnFlVytydXZqUVEzV2pyNXVPCnNncFVWbDZVOW1aN1ZNYXVVWEdZVHA2Qk1UaFFKWlVUQVl5bzZGMzE3blZaRndWVWJ3WnZEU0VXM0I0VzVNS0cKZHRsY3ZpRUNnWUVBNElNOFhxNHozdTlTeVBYZFZEdER4MUNkbkZVak5TMHZ1R0E4ZGpFUGFNbng1STVpTnpFeQpwTjhrUnpaWEVHSW55b3pFVzlzOStHSjJwb3VxZnRJV3pOYUxmZUFRbzBzU3lSTmtuclNsLzlLaERFdWN6alNECkM0dURsb3poTGRmb3VWZi9BdE02SldkWDVZN3ByQXdLcFhkSjJCNU5vQXRxeGxqbnpjeU9zSWtDZ1lFQTY0VlQKMHNHc2tabkVwUUx1ZVIxSkZYSm5qSTA4R01rVFhlU1U0KzJrZzRWdkJTRmxBNzBiOUtaL2l4L3BOcUhveFVYbQpSSTdwUE1GWUJJcU93MFZxdjViN3daQzJnTXgzUWczUUpaRURZb2hZblpFZ3M5UUZyb1NmT2M1dm1seForSG02CmNLN2pkalI5OFFBcHdDd1JVR08vc2lyeFV0UGRnZW16TTFZSUFHVUNnWUFBaFUxbWl0RGorM29kclRST05iVDYKaVYxVU4zNVZhVDFyR0E0TDJDRkpCTzdpc05IWmZ1dTNKaTFYWFBEbXdOT0d6THpIMmNKVENTZHRTM1doeGFyMwozcWVFS3pqZXFCWHJFWGh5UmNqOHh1aEl0d1F1RmtFWGpjTklYaHRIbC9DYVBYSUI5NnR5MnNLQmJjdHM4cm96Cm1Bczd6Ull4QU5YR2ovNDVvL2ZRd1FLQmdRQ0p0Q1MwZjBTVXhPRXkwYW40Nm1TR3c2TkRqSGhzelhRalc3aXEKSTVJaXkrdURobWozYktSaHdNK2wybnlTMHN1MFBCQk1XWHFKYVVvN0xZQVhNQWtnQi9rZXphdkhWc1VMdStQTgpjeUlWaEQ2N3NkVDdENlpheVhRSGFtbmFTTkRaOU9KTGJCWFdVUTZwMER5WS9hc0o0Nmg3Vlp4UG4weE4zd1JGCnFiRjMrUUtCZ1FEYXhTRzgvU1RsNmkyeitIY0F2c25lem1iVXJ6QmFKc3BJM3NOMG9tQ1IvZVJXcTM5dVk5VTEKODk2VXVBQ3RDTnk4VS9EWkY3RnNsakpOK3FkYjYrVXNlN0FZRTVEejlxTDl1SFI2am80dHZoNzFTUkFreE9nRgptSCtDMnJkbDNzdStPU2ZHTlQ4dzFId2dBaDFIdExJRHVsNTFvekduTGFOWWNzNjZ0M0lQbHc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

[root@linux-node1 ~]# cat /root/.kube/config

[root@linux-node1 src]# kubectl config set-cluster kubernetes \

   --certificate-authority=/opt/kubernetes/ssl/ca.pem \

   --embed-certs=true \

   --server=https://192.168.56.11:6443

Cluster "kubernetes" set.

4.设置集群参数

[root@linux-node1 src]# kubectl config set-credentials admin \

   --client-certificate=/opt/kubernetes/ssl/admin.pem \

   --embed-certs=true \

   --client-key=/opt/kubernetes/ssl/admin-key.pem

User "admin" set.

5.设置客户端认证参数

[root@linux-node1 src]# kubectl config set-context kubernetes \

   --cluster=kubernetes \

   --user=admin

Context "kubernetes" created.

6.设置上下文参数

[root@linux-node1 src]# kubectl config use-context kubernetes

Switched to context "kubernetes".

7.设置默认上下文

[root@linux-node1 ~]#  kubectl get cs

NAME                 STATUS      MESSAGE                                                                     ERROR

scheduler            Healthy     ok

controller-manager   Healthy     ok

etcd-               Unhealthy   Get https://192.168.56.12:2379/health: remote error: tls: bad certificate

etcd-               Unhealthy   Get https://192.168.56.13:2379/health: remote error: tls: bad certificate

etcd-               Healthy     {"health": "true"}

[root@linux-node1 ~]#

8.使用kubectl工具

Node节点部署

现在Master节点的组件已经部署完毕，可是光杆司令是无法工作的，所以需要部署node节点；

Node节点部署主要包括：kubelet(master的agent)、Kubernetes Proxy（提供负载均衡服务）

1.二进制包准备将软件包从linux-node1复制到linux-node2中去。

把kubelet、kube-proxy复制到所有node节点

[root@linux-node1 ~]# cd /usr/local/src/kubernetes/server/bin/

[root@linux-node1 bin]# cp kubelet kube-proxy /opt/kubernetes/bin/

[root@linux-node1 bin]# scp kubelet kube-proxy 192.168.56.12:/opt/kubernetes/bin/

[root@linux-node1 bin]# scp kubelet kube-proxy 192.168.56.13:/opt/kubernetes/bin/

scp kubelet kube-proxy 192.168.56.13:/opt/kubernetes/bin/

2.创建角色绑定

[root@linux-node1 ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

clusterrolebinding "kubelet-bootstrap" created

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

切换到/usr/local/src/ssl，一些操作是在 /opt/kubernetes/ssl目录下生成1个bootstrap.kubeconfig 文件，

当node节点的启动kubelet服务时，node节点会携带 bootstrap.kubeconfig 发送post请求进行crs请求验证

所以以后增加Node节点，都需要把该文件复制过去；

3.创建 kubelet bootstrapping kubeconfig 文件设置集群参数

[root@linux-node1 ~]# kubectl config set-cluster kubernetes \

   --certificate-authority=/opt/kubernetes/ssl/ca.pem \

   --embed-certs=true \

   --server=https://192.168.56.11:6443 \

   --kubeconfig=bootstrap.kubeconfig

Cluster "kubernetes" set.

[root@linux-node1 ~]# kubectl config set-context default \

   --cluster=kubernetes \

   --user=kubelet-bootstrap \

   --kubeconfig=bootstrap.kubeconfig

Context "default" created.

设置上下文参数

[root@linux-node1 ~]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

Switched to context "default".

[root@linux-node1 kubernetes]# cp bootstrap.kubeconfig /opt/kubernetes/cfg

[root@linux-node1 kubernetes]# scp bootstrap.kubeconfig 192.168.56.12:/opt/kubernetes/cfg

[root@linux-node1 kubernetes]# scp bootstrap.kubeconfig 192.168.56.13:/opt/kubernetes/cfg

选择默认上下文

4.部署kubelet 1.设置所有Node支持CNI

[root@linux-node2 ~]# mkdir -p /etc/cni/net.d

[root@linux-node2 ~]# vim /etc/cni/net.d/-default.conf

{

        "name": "flannel",

        "type": "flannel",

        "delegate": {

            "bridge": "docker0",

            "isDefaultGateway": true,

            "mtu":

        }

}

vim /etc/cni/net.d/10-default.conf

5.创建kubelet目录

[root@linux-node2 ~]# mkdir /var/lib/kubelet

mkdir /var/lib/kubelet

6.node节点创建kubelet服务配置

[root@k8s-node2 ~]# vim /usr/lib/systemd/system/kubelet.service

[Unit]

Description=Kubernetes Kubelet

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=docker.service

Requires=docker.service

[Service]

WorkingDirectory=/var/lib/kubelet

ExecStart=/opt/kubernetes/bin/kubelet \

  --address=192.168.56.12 \

  --hostname-override=192.168.56.12 \

  --pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0 \

  --experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \

  --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \

  --cert-dir=/opt/kubernetes/ssl \

  --network-plugin=cni \

  --cni-conf-dir=/etc/cni/net.d \

  --cni-bin-dir=/opt/kubernetes/bin/cni \

  --cluster-dns=10.1.0.2 \

  --cluster-domain=cluster.local. \

  --hairpin-mode hairpin-veth \

  --allow-privileged=true \

  --fail-swap-on=false \

  --logtostderr=true \

  --v= \

  --logtostderr=false \

  --log-dir=/opt/kubernetes/log

Restart=on-failure

RestartSec=

node2

[root@k8s-node3 ~]# vim /usr/lib/systemd/system/kubelet.service

[Unit]

Description=Kubernetes Kubelet

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=docker.service

Requires=docker.service

[Service]

WorkingDirectory=/var/lib/kubelet

ExecStart=/opt/kubernetes/bin/kubelet \

  --address=192.168.56.13 \

  --hostname-override=192.168.56.13 \

  --pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0 \

  --experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \

  --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \

  --cert-dir=/opt/kubernetes/ssl \

  --network-plugin=cni \

  --cni-conf-dir=/etc/cni/net.d \

  --cni-bin-dir=/opt/kubernetes/bin/cni \

  --cluster-dns=10.1.0.2 \

  --cluster-domain=cluster.local. \

  --hairpin-mode hairpin-veth \

  --allow-privileged=true \

  --fail-swap-on=false \

  --logtostderr=true \

  --v= \

  --logtostderr=false \

  --log-dir=/opt/kubernetes/log

Restart=on-failure

RestartSec=

node3

7.在node上4.启动Kubelet

[root@linux-node2 ~]# systemctl daemon-reload

[root@linux-node2 ~]# systemctl enable kubelet

[root@linux-node2 ~]# systemctl start kubelet

[root@linux-node2 kubernetes]# systemctl status kubelet

8.在master节点上查看csr请求

注意是在linux-node1（master节点）上执行。

[root@linux-node1 ~]# kubectl get csr

NAME                                                   AGE       REQUESTOR           CONDITION

node-csr-0_w5F1FM_la_SeGiu3Y5xELRpYUjjT2icIFk9gO9KOU   1m        kubelet-bootstrap   Pending

kubectl get csr

可以看到node节点post发送csr请求处在 Pending状态，下面我们批准kubelet 的 TLS 证书请求

[root@linux-node1 ssl]# kubectl get csr|grep 'Pending' | awk 'NR>0{print $1}'| xargs kubectl certificate approve

[root@linux-node1 ssl]# kubectl get csr

NAME                                                   AGE       REQUESTOR           CONDITION

node-csr-YgMzRMD3GrQsJdEUwAKMDp6Zazi-NU_h28DmOlohevc   5m        kubelet-bootstrap   Approved,Issued

node-csr-mu9Ptdy93UCSoGLrk--AVrG1DxKImzgTv5O3kYL1TQE   6m        kubelet-bootstrap   Approved,Issued

kubectl get csr|grep 'Pending' | awk 'NR>0{print $1}'| xargs kubectl certificate approve

csr请求处在由于Pending状态进入Approved状态

与此同时在node节点的/opt/kubernetes/ssl目录下回生成1个名为kubelet-client.crt 的证书

8.查看节点的状态

[root@linux-node1 ssl]# kubectl get node

NAME            STATUS     ROLES     AGE       VERSION

192.168.56.11   NotReady   <none>    8h        v1.10.1

192.168.56.12   Ready      <none>    4m        v1.10.1

192.168.56.13   Ready      <none>    4m        v1.10.1

[root@linux-node1 ssl]#

kubectl get node

部署Kubernetes Proxy

1.配置kube-proxy使用LVS

[root@linux-node2 ~]# yum install -y ipvsadm ipset conntrack

[root@linux-node2 ~]# yum install -y ipvsadm ipset conntrack

2.创建 kube-proxy 证书请求

[root@linux-node1 ~]# cd /usr/local/src/ssl/

[root@linux-node1 ~]# vim kube-proxy-csr.json

{

  "CN": "system:kube-proxy",

  "hosts": [],

  "key": {

    "algo": "rsa",

    "size":

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "System"

    }

  ]

}

vim kube-proxy-csr.json

3.生成证书

[root@linux-node1~]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \

   -ca-key=/opt/kubernetes/ssl/ca-key.pem \

   -config=/opt/kubernetes/ssl/ca-config.json \

   -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy

生成证书

4.分发证书到所有Node节点

[root@linux-node1 ssl]# cp kube-proxy*.pem /opt/kubernetes/ssl/

[root@linux-node1 ssl]# scp kube-proxy*.pem 192.168.56.12:/opt/kubernetes/ssl/

[root@linux-node1 ssl]# scp kube-proxy*.pem 192.168.56.12:/opt/kubernetes/ssl/

5.创建kube-proxy配置文件

[root@linux-node2 ~]# kubectl config set-cluster kubernetes \

   --certificate-authority=/opt/kubernetes/ssl/ca.pem \

   --embed-certs=true \

   --server=https://192.168.56.11:6443 \

   --kubeconfig=kube-proxy.kubeconfig

Cluster "kubernetes" set.

[root@linux-node2 ~]# kubectl config set-credentials kube-proxy \

   --client-certificate=/opt/kubernetes/ssl/kube-proxy.pem \

   --client-key=/opt/kubernetes/ssl/kube-proxy-key.pem \

   --embed-certs=true \

   --kubeconfig=kube-proxy.kubeconfig

User "kube-proxy" set.

[root@linux-node2 ~]# kubectl config set-context default \

   --cluster=kubernetes \

   --user=kube-proxy \

   --kubeconfig=kube-proxy.kubeconfig

Context "default" created.

[root@linux-node2 ~]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

Switched to context "default".

以下操作只为生成1个 kube-proxy.kubeconfig文件

6.分发kubeconfig配置文件

[root@linux-node1 ssl]# cp kube-proxy.kubeconfig /opt/kubernetes/cfg/

[root@linux-node1 ~]# scp kube-proxy.kubeconfig 192.168.56.12:/opt/kubernetes/cfg/

[root@linux-node1 ~]# scp kube-proxy.kubeconfig 192.168.56.13:/opt/kubernetes/cfg/

分发到node

7.创建kube-proxy服务配置

[root@linux-node2 bin]# mkdir /var/lib/kube-proxy

[root@k8s-node2 ~]# vim /usr/lib/systemd/system/kube-proxy.service

[Unit]

Description=Kubernetes Kube-Proxy Server

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=network.target

[Service]

WorkingDirectory=/var/lib/kube-proxy

ExecStart=/opt/kubernetes/bin/kube-proxy \

  --bind-address=192.168.56.12 \

  --hostname-override=192.168.56.12 \

  --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig \

--masquerade-all \

  --feature-gates=SupportIPVSProxyMode=true \

  --proxy-mode=ipvs \

  --ipvs-min-sync-period=5s \

  --ipvs-sync-period=5s \

  --ipvs-scheduler=rr \

  --logtostderr=true \

  --v= \

  --logtostderr=false \

  --log-dir=/opt/kubernetes/log

Restart=on-failure

RestartSec=

LimitNOFILE=

[Install]

WantedBy=multi-user.target

.启动Kubernetes Proxy

[root@linux-node2 ~]# systemctl daemon-reload

[root@linux-node2 ~]# systemctl enable kube-proxy

[root@linux-node2 ~]# systemctl start kube-proxy

9.查看服务状态查看kube-proxy服务状态

[root@linux-node2 cfg]# systemctl status kube-proxy

● kube-proxy.service - Kubernetes Kube-Proxy Server

   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)

   Active: active (running) since Tue -- :: CST; 13min ago

     Docs: https://github.com/GoogleCloudPlatform/kubernetes

 Main PID:  (kube-proxy)

   Memory: 38.1M

   CGroup: /system.slice/kube-proxy.service

           ‣  /opt/kubernetes/bin/kube-proxy --bind-address=192.168.56.12 --hos...

Apr  :: linux-node2.example.com systemd[]: kube-proxy.service holdoff time....

Apr  :: linux-node2.example.com systemd[]: Started Kubernetes Kube-Proxy S....

Apr  :: linux-node2.example.com systemd[]: Starting Kubernetes Kube-Proxy ....

Hint: Some lines were ellipsized, use -l to show in full.

[root@linux-node2 cfg]# ipvsadm -L -n

IP Virtual Server version 1.2. (size=)

Prot LocalAddress:Port Scheduler Flags

  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

TCP  10.1.0.1: rr persistent

  -> 192.168.56.11:           Masq

[root@linux-node2 cfg]#

Flannel网络部署

即使我们部署好了所有Kubernetes集群的所有组件，但是它们没有网络是无法通行的！

Poid：Poid是Kubernetes中1个逻辑的概念，Kubernets直接管理的不是容器而是Poid；
每个Poild里面又包含了1个或者N个容器；

如果你需要的容器之间需要网络互通，你就可以在1个poind里面跑多个容器，它们之间通过host来通行；

每个Poid都要1个IP地址：用来替代docker0网桥，实现Kubernetes中不同容器间的通信

Replication Ctronller（复杂控制）简称RC

RC:是保证Kubernetes中Pod高可用，通过监控运行中的Pod来保证集群中指定Pod的副本数量；

RS(replica set)Kubernetes感觉RC的功能有点单一就对RC升级新增了新功能，RS是RC的升级版

Deployment是1个比RS应用模式更广的API对象：它在保证Pod副本数量的前提下，还可以创建、更新、滚动升级服务；

RC、RS和Deplyment只是保证了支持服务的Pod数量，但没有解决如何高效访问这些服务的问题；
在K8S集群中客户端要访问的服务就是1个server对象，每个servers对象对应集群内有效的虚拟IP

Flunel：就是通过对docker网络的封装实现 Kubernetes中Pod之间的通信，它可以每个Node都可以分配到不同的IP地址段

1.为Flannel生成证书

在/usr/local/src/ssl

{

  "CN": "flanneld",

  "hosts": [],

  "key": {

    "algo": "rsa",

    "size":

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "System"

    }

  ]

vim flanneld-csr.json

2.生成证书

[root@linux-node1 ~]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \

   -ca-key=/opt/kubernetes/ssl/ca-key.pem \

   -config=/opt/kubernetes/ssl/ca-config.json \

   -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

为flannel生成证书

3.分发证书

[root@linux-node1 ~]# cp flanneld*.pem /opt/kubernetes/ssl/

[root@linux-node1 ~]# scp flanneld*.pem 192.168.56.12:/opt/kubernetes/ssl/

[root@linux-node1 ~]# scp flanneld*.pem 192.168.56.13:/opt/kubernetes/ssl/

cp flanneld*.pem /opt/kubernetes/ssl/

4.下载Flannel软件包

[root@linux-node1 ~]# cd /usr/local/src

# wget

 https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz

[root@linux-node1 src]# tar zxf flannel-v0.10.0-linux-amd64.tar.gz

[root@linux-node1 src]# cp flanneld mk-docker-opts.sh /opt/kubernetes/bin/

复制到linux-node2节点

[root@linux-node1 src]# scp flanneld mk-docker-opts.sh 192.168.56.12:/opt/kubernetes/bin/

[root@linux-node1 src]# scp flanneld mk-docker-opts.sh 192.168.56.13:/opt/kubernetes/bin/

复制对应脚本到/opt/kubernetes/bin目录下。

[root@linux-node1 ~]# cd /usr/local/src/kubernetes/cluster/centos/node/bin/

[root@linux-node1 bin]# cp remove-docker0.sh /opt/kubernetes/bin/

[root@linux-node1 bin]# scp remove-docker0.sh 192.168.56.12:/opt/kubernetes/bin/

[root@linux-node1 bin]# scp remove-docker0.sh 192.168.56.13:/opt/kubernetes/bin/

已经下载好了

5.配置Flannel

[root@linux-node1 ~]# vim /opt/kubernetes/cfg/flannel

FLANNEL_ETCD="-etcd-endpoints=https://192.168.56.11:2379,https://192.168.56.12:2379,https://192.168.56.13:2379"

FLANNEL_ETCD_KEY="-etcd-prefix=/kubernetes/network"

FLANNEL_ETCD_CAFILE="--etcd-cafile=/opt/kubernetes/ssl/ca.pem"

FLANNEL_ETCD_CERTFILE="--etcd-certfile=/opt/kubernetes/ssl/flanneld.pem"

FLANNEL_ETCD_KEYFILE="--etcd-keyfile=/opt/kubernetes/ssl/flanneld-key.pem"

复制配置到其它节点上

[root@linux-node1 ~]# scp /opt/kubernetes/cfg/flannel 192.168.56.12:/opt/kubernetes/cfg/

[root@linux-node1 ~]# scp /opt/kubernetes/cfg/flannel 192.168.56.13:/opt/kubernetes/cfg/

vim /opt/kubernetes/cfg/flannel

6.设置Flannel系统服务

[root@linux-node1 ~]# vim /usr/lib/systemd/system/flannel.service

[Unit]

Description=Flanneld overlay address etcd agent

After=network.target

Before=docker.service

[Service]

EnvironmentFile=-/opt/kubernetes/cfg/flannel

ExecStartPre=/opt/kubernetes/bin/remove-docker0.sh

ExecStart=/opt/kubernetes/bin/flanneld ${FLANNEL_ETCD} ${FLANNEL_ETCD_KEY} ${FLANNEL_ETCD_CAFILE} ${FLANNEL_ETCD_CERTFILE} ${FLANNEL_ETCD_KEYFILE}

ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -d /run/flannel/docker

Type=notify

[Install]

WantedBy=multi-user.target

RequiredBy=docker.service

vim /usr/lib/systemd/system/flannel.service

7.复制系统服务脚本到其它节点上

scp /usr/lib/systemd/system/flannel.service 192.168.56.12:/usr/lib/systemd/system/

scp /usr/lib/systemd/system/flannel.service 192.168.56.13:/usr/lib/systemd/system/

8.Flannel 和CNI集成

https://github.com/containernetworking/plugins/releases

wget https://github.com/containernetworking/plugins/releases/download/v0.7.1/cni-plugins-amd64-v0.7.1.tgz

[root@linux-node1 ~]# mkdir /opt/kubernetes/bin/cni

[root@linux-node1 src]# tar zxf cni-plugins-amd64-v0.7.1.tgz -C /opt/kubernetes/bin/cni

# scp -r /opt/kubernetes/bin/cni/* 192.168.56.12:/opt/kubernetes/bin/cni/

# scp -r /opt/kubernetes/bin/cni/* 192.168.56.13:/opt/kubernetes/bin/cni/

CNI插件已经下载好了

创建Etcd的key

/opt/kubernetes/bin/etcdctl --ca-file /opt/kubernetes/ssl/ca.pem --cert-file /opt/kubernetes/ssl/flanneld.pem --key-file /opt/kubernetes/ssl/flanneld-key.pem \

      --no-sync -C https://192.168.56.11:2379,https://192.168.56.12:2379,https://192.168.56.13:2379 \

mk /kubernetes/network/config '{ "Network": "10.2.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}' >/dev/null >&

9.启动flannel

[root@linux-node1 ~]# systemctl daemon-reload

[root@linux-node1 ~]# systemctl enable flannel

[root@linux-node1 ~]# chmod +x /opt/kubernetes/bin/*

[root@linux-node1 ~]# systemctl start flannel
[root@linux-node1 ~]# systemctl status flannel

10.修改docker配置文件，让Docker使用Flannel

[root@linux-node1 ~]# vim /usr/lib/systemd/system/docker.service

[Unit] #在Unit下面修改After和增加Requires

After=network-online.target firewalld.service flannel.service

Wants=network-online.target

Requires=flannel.service

[Service] #增加EnvironmentFile=-/run/flannel/docker

Type=notify

EnvironmentFile=-/run/flannel/docker

ExecStart=/usr/bin/dockerd $DOCKER_OPTS

11.docker配置文件同步

scp /usr/lib/systemd/system/docker.service 192.168.56.12:/usr/lib/systemd/system/

scp /usr/lib/systemd/system/docker.service 192.168.56.13:/usr/lib/systemd/system/

12.重启Docker

[root@linux-node1 ~]# systemctl daemon-reload

[root@linux-node1 ~]# systemctl restart docker

[root@linux-node1 ssl]# kubectl run net-test --image=alpine --replicas= sleep

deployment.apps "net-test" created

[root@linux-node1 ssl]# kubectl get pod -o wide

NAME                        READY     STATUS              RESTARTS   AGE       IP        NODE

net-test-5767cb94df-mkj4p   /       ContainerCreating             10s       <none>    192.168.56.12

net-test-5767cb94df-q8vbt   /       ContainerCreating             10s       <none>    192.168.56.13

[root@linux-node1 ssl]# kubectl get pod -o wide

NAME                        READY     STATUS              RESTARTS   AGE       IP        NODE

net-test-5767cb94df-mkj4p   /       ContainerCreating             27s       <none>    192.168.56.12

net-test-5767cb94df-q8vbt   /       ContainerCreating             27s       <none>    192.168.56.13

[root@linux-node1 ssl]#  kubectl get pod -o wide

NAME                        READY     STATUS    RESTARTS   AGE       IP           NODE

net-test-5767cb94df-mkj4p   /       Running             50s       10.2.101.2   192.168.56.12

net-test-5767cb94df-q8vbt   /       Running             50s       10.2.34.2    192.168.56.13

[root@linux-node1 ssl]# ping 10.2.101.2

PING 10.2.101.2 (10.2.101.2) () bytes of data.

 bytes from 10.2.101.2: icmp_seq= ttl= time=1.88 ms

 bytes from 10.2.101.2: icmp_seq= ttl= time=0.400 ms

^X64 bytes from 10.2.101.2: icmp_seq= ttl= time=0.348 ms

 bytes from 10.2.101.2: icmp_seq= ttl= time=0.406 ms

^C

--- 10.2.101.2 ping statistics ---

 packets transmitted,  received, % packet loss, time 3000ms

rtt min/avg/max/mdev = 0.348/0.760/1.889

测试创建1个K8S应用

如果最后容器处在 running状态说明你的K8S已经搭建完毕；

https://www.bilibili.com/video/av82289390?p=4

码农公寓

前言

什么是Kubernets?

Kuberbetes的架构

实验环境准备

Centos7安装Kubernetes 1.10.1版本

相关文章