CentOS 7.4搭建Kubernetes 1.8.5集群

2023-08-09 17:12:34

环境介绍

角色	操作系统	IP	主机名	Docker版本
master,node	CentOS 7.4	192.168.0.210	node210	17.11.0-ce
node	CentOS 7.4	192.168.0.211	node211	17.11.0-ce
node	CentOS 7.4	192.168.0.212	node212	17.11.0-ce

1.基础环境配置(所有服务器执行)
a.SELinux关闭

sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux

setenforce 0

b.Docker安装

根据官方文档，Docker分为Community Edition (CE)和Enterprise Edition (EE)两个版本，我们作为学习和个人使用，当然选择的是Community Edition (CE)，安装步骤如下：

// 步骤1 - 为了确保没有安装过老的Docker版本，先将已经安装的Docker从宿主机上删除（如果是在使用中的正式服务器，请谨慎执行此步）:

$ sudo yum remove docker \

  docker-common \

  docker-selinux \

  docker-engine

// 步骤2 - 安装Docker所需的包：

$ sudo yum install -y yum-utils \

  device-mapper-persistent-data \

  lvm2

// 步骤3 - 配置到稳定的Docker CE安装库：

$ sudo yum-config-manager \

    --add-repo \

    https://download.docker.com/linux/centos/docker-ce.repo

// 步骤4 - 安装Docker CE：

$ sudo yum install docker-ce

// 步骤5 - 启动Docker服务：

$ sudo systemctl start docker

// 步骤6 - 测试是否安装成功：

// 可以通过查看版本的形式确认安装是否成功：

$ docker --version

// 也可以通过直接运行hello-world容器来确认安装是否成功：

$ docker run hello-world

c.配置国内Docker镜像加速器

curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://e2a6d434.m.daocloud.io

d.开启Docker开机自动启动

systemctl enable docker.service

systemctl restart docker

2.kubernetes证书准备(master执行)
a.为将文件复制到Node节点，节省部署时间，我这里做ssh信任免密复制

ssh-genkey -t rsa

ssh-copy-id 192.168.0.211

ssh-copy-id 192.168.0.212

b.下载证书生成工具

yum -y install wget

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

chmod +x cfssl_linux-amd64

mv cfssl_linux-amd64 /usr/local/bin/cfssl

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

chmod +x cfssljson_linux-amd64

mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

chmod +x cfssl-certinfo_linux-amd64

mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

c.CA证书制作
#目录准备

mkdir /root/ssl

cd /root/ssl

#创建CA证书配置
vim ca-config.json

{

  "signing": {

    "default": {

      "expiry": "87600h"

    },

    "profiles": {

      "kubernetes": {

        "usages": [

            "signing",

            "key encipherment",

            "server auth",

            "client auth"

        ],

        "expiry": "87600h"

      }

    }

  }

}

#创建CA证书请求文件
vim ca-csr.json

{

  "CN": "kubernetes",

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "ST": "JIANGXI",

      "L": "NANCHANG",

      "O": "k8s",

      "OU": "System"

    }

  ]

}

#生成CA证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

#创建kubernetes证书签名请求
vim kubernetes-csr.json

{

    "CN": "kubernetes",

    "hosts": [

      "127.0.0.1",

      "192.168.0.210",      #修改成自己主机的IP

      "192.168.0.211",      #修改成自己主机的IP

      "192.168.0.212",      #修改成自己主机的IP

      "10.254.0.1",

      "kubernetes",

      "node210",      #修改成自己主机的主机名

      "node211",      #修改成自己主机的主机名

      "node212",      #修改成自己主机的主机名

      "kubernetes.default",

      "kubernetes.default.svc",

      "kubernetes.default.svc.cluster",

      "kubernetes.default.svc.cluster.local"

    ],

    "key": {

        "algo": "rsa",

        "size": 2048

    },

    "names": [

        {

            "C": "CN",

            "ST": "JIANGXI",

            "L": "JIANGXI",

            "O": "k8s",

            "OU": "System"

        }

    ]

}

#生成kubernetes证书及私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

#创建admin证书签名请求
vim admin-csr.json

{

  "CN": "admin",

  "hosts": [],

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "ST": "JIANGXI",

      "L": "JIANGXI",

      "O": "system:masters",

      "OU": "System"

    }

  ]

}

#生成admin证书及私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

#创建 kube-proxy 证书签名请求
vim kube-proxy-csr.json

{

  "CN": "system:kube-proxy",

  "hosts": [],

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "ST": "JIANGXI",

      "L": "JIANGXI",

      "O": "k8s",

      "OU": "System"

    }

  ]

}

#生成证书及私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

#分发证书

mkdir -p /etc/kubernetes/ssl

cp -r *.pem /etc/kubernetes/ssl

cd /etc

scp -r kubernetes/ 192.168.0.211:/etc/

scp -r kubernetes/ 192.168.0.212:/etc/

3.etcd集群安装及配置
a.下载etcd，并分发至节点

wget https://github.com/kubernetes/kubernetes/archive/v1.8.5.tar.gz

tar zxf etcd-v3.2.11-linux-amd64.tar.gz

mv etcd-v3.2.11-linux-amd64/etcd* /usr/local/bin

scp -r /usr/local/bin/etc* 192.168.0.211:/usr/local/bin/

scp -r /usr/local/bin/etc* 192.168.0.212:/usr/local/bin/

b.创建etcd服务启动文件
vim /usr/lib/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

Documentation=https://github.com/coreos

[Service]

Type=notify

WorkingDirectory=/var/lib/etcd/

EnvironmentFile=-/etc/etcd/etcd.conf

ExecStart=/usr/local/bin/etcd \

  --name ${ETCD_NAME} \

  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \

  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \

  --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \

  --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \

  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \

  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \

  --initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \

  --listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \

  --listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \

  --advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \

  --initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \

  --initial-cluster infra1=https://192.168.0.210:2380,infra2=https://192.168.0.211:2380,infra3=https://192.168.0.212:2380 \

  --initial-cluster-state new \

  --data-dir=${ETCD_DATA_DIR}

Restart=on-failure

RestartSec=5

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

c.创建必要的目录

mkdir -p /var/lib/etcd/

mkdir /etc/etcd

d.编辑etcd的配置文件
vim /etc/etcd/etcd.conf
node210的配置文件/etc/etcd/etcd.conf为

# [member]

ETCD_NAME=infra1

ETCD_DATA_DIR="/var/lib/etcd"

ETCD_LISTEN_PEER_URLS="https://192.168.0.210:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.0.210:2379"

#[cluster]

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.210:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.210:2379"

node211的配置文件/etc/etcd/etcd.conf为

# [member]

ETCD_NAME=infra2

ETCD_DATA_DIR="/var/lib/etcd"

ETCD_LISTEN_PEER_URLS="https://192.168.0.211:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.0.211:2379"

#[cluster]

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.211:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.211:2379"

node212的配置文件/etc/etcd/etcd.conf为

# [member]

ETCD_NAME=infra3

ETCD_DATA_DIR="/var/lib/etcd"

ETCD_LISTEN_PEER_URLS="https://192.168.0.212:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.0.212:2379"

#[cluster]

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.212:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.212:2379"

#在所有节点执行，启动etcd

systemctl daemon-reload

systemctl enable etcd

systemctl start etcd

systemctl status etcd

如果报错，就需要查看/var/log/messages文件进行排错

e.测试集群是否正常

验证ETCD是否成功启动

etcdctl \

  --ca-file=/etc/kubernetes/ssl/ca.pem \

  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \

  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \

  cluster-health

4.配置kubernetes参数
a.下载kubernetes编译好的二进制文件并进行分发

wget https://dl.k8s.io/v1.8.5/kubernetes-server-linux-amd64.tar.gz

tar zxf kubernetes-server-linux-amd64.tar.gz

cp -rf kubernetes/server/bin/{kube-apiserver,kube-controller-manager,kubectl,kubefed,kubelet,kube-proxy,kube-scheduler} /usr/local/bin/

scp -r kubernetes/server/bin/{kubelet,kube-proxy} 192.168.0.211:/usr/local/bin/

scp -r kubernetes/server/bin/{kubelet,kube-proxy} 192.168.0.212:/usr/local/bin/

#查看kubernetes最新版，可到https://github.com/kubernetes/kubernetes/releases
然后进入 CHANGELOG-x.x.md就可限制二进制的下载地址

b.创建 TLS Bootstrapping Token

cd /etc/kubernetes

export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')

cat > token.csv <<EOF

${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"

EOF

c.创建 kubelet bootstrapping kubeconfig 文件

cd /etc/kubernetes

export KUBE_APISERVER="https://192.168.0.210:6443"

#设置集群参数

kubectl config set-cluster kubernetes \

  --certificate-authority=/etc/kubernetes/ssl/ca.pem \

  --embed-certs=true \

  --server=${KUBE_APISERVER} \

  --kubeconfig=bootstrap.kubeconfig

#设置客户端认证参数

kubectl config set-credentials kubelet-bootstrap \

  --token=${BOOTSTRAP_TOKEN} \

  --kubeconfig=bootstrap.kubeconfig

#设置上下文参数

kubectl config set-context default \

  --cluster=kubernetes \

  --user=kubelet-bootstrap \

  --kubeconfig=bootstrap.kubeconfig

#设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

d.创建 kube-proxy kubeconfig 文件
export KUBE_APISERVER="https://192.168.0.210:6443"
#设置集群参数

kubectl config set-cluster kubernetes \

  --certificate-authority=/etc/kubernetes/ssl/ca.pem \

  --embed-certs=true \

  --server=${KUBE_APISERVER} \

  --kubeconfig=kube-proxy.kubeconfig

#设置客户端认证参数

kubectl config set-credentials kube-proxy \

  --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \

  --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \

  --embed-certs=true \

  --kubeconfig=kube-proxy.kubeconfig

#设置上下文参数

kubectl config set-context default \

  --cluster=kubernetes \

  --user=kube-proxy \

  --kubeconfig=kube-proxy.kubeconfig

#设置默认上下文
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

e.创建 kubectl kubeconfig 文件
export KUBE_APISERVER="https://192.168.0.210:6443"
#设置集群参数

kubectl config set-cluster kubernetes \

  --certificate-authority=/etc/kubernetes/ssl/ca.pem \

  --embed-certs=true \

  --server=${KUBE_APISERVER}

#设置客户端认证参数

kubectl config set-credentials admin \

  --client-certificate=/etc/kubernetes/ssl/admin.pem \

  --embed-certs=true \

  --client-key=/etc/kubernetes/ssl/admin-key.pem

#设置上下文参数

kubectl config set-context kubernetes \

  --cluster=kubernetes \

  --user=admin

#设置默认上下文
kubectl config use-context kubernetes

f.将2个bootstrap.kubeconfig kube-proxy.kubeconfig文件分发至其余服务器

scp -r *.kubeconfig 192.168.0.211:/etc/kubernetes/

scp -r *.kubeconfig 192.168.0.212:/etc/kubernetes/

5.MASTER安装及配置
a.apiserver安装配置
#apiserver服务启动文件
vim /usr/lib/systemd/system/kube-apiserver.service

[Unit]

Description=Kubernetes API Service

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=network.target

After=etcd.service

[Service]

EnvironmentFile=-/etc/kubernetes/config

EnvironmentFile=-/etc/kubernetes/apiserver

ExecStart=/usr/local/bin/kube-apiserver \

        $KUBE_LOGTOSTDERR \

        $KUBE_LOG_LEVEL \

        $KUBE_ETCD_SERVERS \

        $KUBE_API_ADDRESS \

        $KUBE_API_PORT \

        $KUBELET_PORT \

        $KUBE_ALLOW_PRIV \

        $KUBE_SERVICE_ADDRESSES \

        $KUBE_ADMISSION_CONTROL \

        $KUBE_API_ARGS

Restart=on-failure

Type=notify

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

#配置kubernetes默认配置
vim /etc/kubernetes/config

###

# kubernetes system config

#

# The following values are used to configure various aspects of all

# kubernetes services, including

#

#   kube-apiserver.service

#   kube-controller-manager.service

#   kube-scheduler.service

#   kubelet.service

#   kube-proxy.service

# logging to stderr means we get it in the systemd journal

KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug

KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers

KUBE_ALLOW_PRIV="--allow-privileged=true"

# How the controller-manager, scheduler, and proxy find the apiserver

#KUBE_MASTER="--master=http://sz-pg-oam-docker-test-001.tendcloud.com:8080"

KUBE_MASTER="--master=http://192.168.0.210:8080"

#配置apiserver参数
vim /etc/kubernetes/apiserver

###

## kubernetes system config

##

## The following values are used to configure the kube-apiserver

##

#

## The address on the local server to listen to.

#KUBE_API_ADDRESS="--insecure-bind-address=sz-pg-oam-docker-test-001.tendcloud.com"

KUBE_API_ADDRESS="--advertise-address=192.168.0.210 --bind-address=192.168.0.210 --insecure-bind-address=192.168.0.210"

#

## The port on the local server to listen on.

#KUBE_API_PORT="--port=8080"

#

## Port minions listen on

#KUBELET_PORT="--kubelet-port=10250"

#

## Comma separated list of nodes in the etcd cluster

KUBE_ETCD_SERVERS="--etcd-servers=https://192.168.0.210:2379,https://192.168.0.211:2379,https://192.168.0.212:2379"

#

## Address range to use for services

KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

#

## default admission control policies

KUBE_ADMISSION_CONTROL="--admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota"

#

## Add your own!

KUBE_API_ARGS="--authorization-mode=RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --enable-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kuberne

#如果出现错误，查看/var/log/messages

tes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --

event-ttl=1h"

#启动apiserver

systemctl daemon-reload

systemctl enable kube-apiserver

systemctl start kube-apiserver

systemctl status kube-apiserver

#如果出现错误，查看/var/log/messages

b.controller-manager服务配置
#controller-manager服务启动文件
vim /usr/lib/systemd/system/kube-controller-manager.service

[Unit]

Description=Kubernetes Controller Manager

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]

EnvironmentFile=-/etc/kubernetes/config

EnvironmentFile=-/etc/kubernetes/controller-manager

ExecStart=/usr/local/bin/kube-controller-manager \

        $KUBE_LOGTOSTDERR \

        $KUBE_LOG_LEVEL \

        $KUBE_MASTER \

        $KUBE_CONTROLLER_MANAGER_ARGS

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

#配置controller-manager服务配置文件
vim /etc/kubernetes/controller-manager

#The following values are used to configure the kubernetes controller-manager

#defaults from config and apiserver should be adequate

#Add your own!

KUBE_CONTROLLER_MANAGER_ARGS="--address=127.0.0.1 --service-cluster-ip-range=10.254.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem --root-ca-file=/etc/kubernetes/ssl/ca.pem --leader-elect=true"

#启动controller-manager服务
systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl start kube-controller-manager

c.scheduler服务安装及配置
#配置scheduler服务启动文件
vim /usr/lib/systemd/system/kube-scheduler.service

[Unit]

Description=Kubernetes Scheduler Plugin

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]

EnvironmentFile=-/etc/kubernetes/config

EnvironmentFile=-/etc/kubernetes/scheduler

ExecStart=/usr/local/bin/kube-scheduler \

            $KUBE_LOGTOSTDERR \

            $KUBE_LOG_LEVEL \

            $KUBE_MASTER \

            $KUBE_SCHEDULER_ARGS

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

#配置scheduler服务配置文件
vim /etc/kubernetes/scheduler

#kubernetes scheduler config

#default config should be adequate

#Add your own!

KUBE_SCHEDULER_ARGS="--leader-elect=true --address=127.0.0.1"

#启动scheduler服务

systemctl daemon-reload

systemctl enable kube-scheduler

systemctl start kube-scheduler

d.测试master是否正常
kubectl get componentstatuses
#结果如下说明正常

NAME                 STATUS    MESSAGE              ERROR

scheduler            Healthy   ok

controller-manager   Healthy   ok

etcd-0               Healthy   {"health": "true"}

etcd-1               Healthy   {"health": "true"}

etcd-2               Healthy   {"health": "true"}

6.node安装(所有节点)
a.flannel安装及配置(容器网络我们采用flannel)
#yum安装flannel
yum install -y flannel
#检查node节点证书情况
ls /etc/kubernetes/ssl
#修改flannel.service配置文件如下
vi /usr/lib/systemd/system/flanneld.service

[Unit]

Description=Flanneld overlay address etcd agent

After=network.target

After=network-online.target

Wants=network-online.target

After=etcd.service

Before=docker.service

[Service]

Type=notify

EnvironmentFile=/etc/sysconfig/flanneld

EnvironmentFile=-/etc/sysconfig/docker-network

ExecStart=/usr/bin/flanneld-start \

  -etcd-endpoints=${ETCD_ENDPOINTS} \

  -etcd-prefix=${ETCD_PREFIX} \

  $FLANNEL_OPTIONS

ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker

Restart=on-failure

[Install]

WantedBy=multi-user.target

RequiredBy=docker.service

#修改flannel配置文件
vi /etc/sysconfig/flanneld

# Flanneld configuration options

# etcd url location.  Point this to the server where etcd runs

ETCD_ENDPOINTS="https://192.168.0.210:2379,https://192.168.0.211:2379,https://192.168.0.212:2379"

# etcd config key.  This is the configuration key that flannel queries

# For address range assignment

ETCD_PREFIX="/kube-centos/network"

# Any additional options that you want to pass

FLANNEL_OPTIONS="-etcd-cafile=/etc/kubernetes/ssl/ca.pem -etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem -etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem"

#在etcd中创建网络配置

etcdctl --endpoints=https://192.168.0.210:2379,https://192.168.0.211:2379,https://192.168.0.212:2379 \

  --ca-file=/etc/kubernetes/ssl/ca.pem \

  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \

  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \

  mkdir /kube-centos/network

etcdctl --endpoints=https://192.168.0.210:2379,https://192.168.0.211:2379,https://192.168.0.212:2379 \

  --ca-file=/etc/kubernetes/ssl/ca.pem \

  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \

  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \

  mk /kube-centos/network/config '{"Network":"172.30.0.0/16","SubnetLen":24,"Backend":{"Type":"vxlan"}}'

#flannel服务启动
systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld
systemctl status flanneld

b.配置Docker服务启动文件,集成flannel
vim /usr/lib/systemd/system/docker.service

在ExecStart上增加

EnvironmentFile=-/run/flannel/docker

EnvironmentFile=-/run/docker_opts.env

EnvironmentFile=-/run/flannel/subnet.env

修改如下

ExecStart=/usr/bin/dockerd --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}

效果如下:

[Unit]

Description=Docker Application Container Engine

Documentation=https://docs.docker.com

After=network-online.target firewalld.service

Wants=network-online.target

[Service]

Type=notify

# the default is not to use systemd for cgroups because the delegate issues still

# exists and systemd currently does not support the cgroup feature set required

# for containers run by docker

EnvironmentFile=-/run/flannel/docker

EnvironmentFile=-/run/docker_opts.env

EnvironmentFile=-/run/flannel/subnet.env

ExecStart=/usr/bin/dockerd --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}

ExecReload=/bin/kill -s HUP $MAINPID

# Having non-zero Limit*s causes performance problems due to accounting overhead

# in the kernel. We recommend using cgroups to do container-local accounting.

LimitNOFILE=infinity

LimitNPROC=infinity

LimitCORE=infinity

# Uncomment TasksMax if your systemd version supports it.

# Only systemd 226 and above support this version.

#TasksMax=infinity

TimeoutStartSec=0

# set delegate yes so that systemd does not reset the cgroups of docker containers

Delegate=yes

# kill only the docker process, not all processes in the cgroup

KillMode=process

# restart the docker process if it exits prematurely

Restart=on-failure

StartLimitBurst=3

StartLimitInterval=60s

[Install]

WantedBy=multi-user.target

#重启启动Docker服务

systemctl daemon-reload

systemctl restart docker

systemctl status docker

切记要先启动flannel，再启动Docker

c.查询etcd是否分配网络

etcdctl --endpoints=${ETCD_ENDPOINTS} \

  --ca-file=/etc/kubernetes/ssl/ca.pem \

  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \

  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \

  ls /kube-centos/network/subnets

结果大致如下

/kube-centos/network/subnets/172.30.1.0-24

/kube-centos/network/subnets/172.30.54.0-24

/kube-centos/network/subnets/172.30.99.0-24

etcdctl --endpoints=${ETCD_ENDPOINTS} \

  --ca-file=/etc/kubernetes/ssl/ca.pem \

  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \

  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \

  get /kube-centos/network/config

结果大致如下

{"Network":"172.30.0.0/16","SubnetLen":24,"Backend":{"Type":"vxlan"}}

d.安装及配置kubelet
#创建kubelet服务启动文件
vim /usr/lib/systemd/system/kubelet.service

[Unit]

Description=Kubernetes Kubelet Server

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=docker.service

Requires=docker.service

[Service]

WorkingDirectory=/var/lib/kubelet

EnvironmentFile=-/etc/kubernetes/config

EnvironmentFile=-/etc/kubernetes/kubelet

ExecStart=/usr/local/bin/kubelet \

            $KUBE_LOGTOSTDERR \

            $KUBE_LOG_LEVEL \

            $KUBELET_API_SERVER \

            $KUBELET_ADDRESS \

            $KUBELET_PORT \

            $KUBELET_HOSTNAME \

            $KUBE_ALLOW_PRIV \

            $KUBELET_POD_INFRA_CONTAINER \

            $KUBELET_ARGS

Restart=on-failure

[Install]

WantedBy=multi-user.target

#kubelet认证配置文件
vim /etc/kubernetes/kubelet.kubeconfig

apiVersion: v1

kind: Config

clusters:

  - cluster:

      server: http://192.168.0.210:8080

    name: local

contexts:

  - context:

      cluster: local

    name: local

current-context: local

#kubelet配置文件
vim /etc/kubernetes/kubelet

node210下/etc/kubernetes/kubelet内容如下

###

## kubernetes kubelet (minion) config

#

## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)

KUBELET_ADDRESS="--address=192.168.0.210"

#

## The port for the info server to serve on

#KUBELET_PORT="--port=10250"

#

## You may leave this blank to use the actual hostname

KUBELET_HOSTNAME="--hostname-override=192.168.0.210"

#

## location of the api-server

#KUBELET_API_SERVER="--api-servers=http://192.168.0.210:8080"

KUBELET_API_SERVER=" "

#

## pod infrastructure container

#KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=kubernetes/pause"

#

## Add your own!

#KUBELET_ARGS="--cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --require-kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false"

KUBELET_ARGS="--cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false --fail-swap-on=false"

node211里配置文件如下

###

## kubernetes kubelet (minion) config

#

## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)

KUBELET_ADDRESS="--address=192.168.0.211"

#

## The port for the info server to serve on

#KUBELET_PORT="--port=10250"

#

## You may leave this blank to use the actual hostname

KUBELET_HOSTNAME="--hostname-override=192.168.0.211"

#

## location of the api-server

#KUBELET_API_SERVER="--api-servers=http://192.168.0.210:8080"

KUBELET_API_SERVER=" "

#

## pod infrastructure container

#KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=kubernetes/pause"

#

## Add your own!

#KUBELET_ARGS="--cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --require-kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false"

KUBELET_ARGS="--cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false --fail-swap-on=false"

node212里配置文件如下

###

## kubernetes kubelet (minion) config

#

## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)

KUBELET_ADDRESS="--address=192.168.0.212"

#

## The port for the info server to serve on

#KUBELET_PORT="--port=10250"

#

## You may leave this blank to use the actual hostname

KUBELET_HOSTNAME="--hostname-override=192.168.0.212"

#

## location of the api-server

#KUBELET_API_SERVER="--api-servers=http://192.168.0.210:8080"

KUBELET_API_SERVER=" "

#

## pod infrastructure container

#KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"

KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=kubernetes/pause"

#

## Add your own!

#KUBELET_ARGS="--cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --require-kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false"

KUBELET_ARGS="--cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false --fail-swap-on=false"

#启动kubelet服务

mkdir -p /var/lib/kubelet

systemctl daemon-reload

systemctl enable kubelet

systemctl start kubelet

systemctl status kubelet

#这里很容易出错，出错时查看/var/log/messages看日志进行排错

#检查kubelet服务是否正常

kubectl get nodes

NAME            STATUS    ROLES     AGE       VERSION

192.168.0.210   Ready     <none>    14h       v1.8.5

192.168.0.211   Ready     <none>    14h       v1.8.5

192.168.0.212   Ready     <none>    14h       v1.8.5

c.安装及配置kube-proxy
#配置kube-proxy服务启动文件
vim /usr/lib/systemd/system/kube-proxy.service

[Unit]

Description=Kubernetes Kube-Proxy Server

Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=network.target

[Service]

EnvironmentFile=-/etc/kubernetes/config

EnvironmentFile=-/etc/kubernetes/proxy

ExecStart=/usr/local/bin/kube-proxy \

        $KUBE_LOGTOSTDERR \

        $KUBE_LOG_LEVEL \

        $KUBE_MASTER \

        $KUBE_PROXY_ARGS

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

#kube-proxy配置文件如下：
node210:
vim /etc/kubernetes/proxy

###

# kubernetes proxy config

# default config should be adequate

# Add your own!

KUBE_PROXY_ARGS="--bind-address=192.168.0.210 --hostname-override=192.168.0.210 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16"

node211:
vim /etc/kubernetes/proxy

###

# kubernetes proxy config

# default config should be adequate

# Add your own!

KUBE_PROXY_ARGS="--bind-address=192.168.0.211 --hostname-override=192.168.0.211 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16"

node212:
vim /etc/kubernetes/proxy

###

# kubernetes proxy config

# default config should be adequate

# Add your own!

KUBE_PROXY_ARGS="--bind-address=192.168.0.212--hostname-override=192.168.0.212 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16"

#启动kube-proxy服务

systemctl daemon-reload

systemctl enable kube-proxy

systemctl start kube-proxy

systemctl status kube-proxy

d.在所有节点默认开启forward为accept
vim /usr/lib/systemd/system/forward.service

[Unit]

Description=iptables forward

Documentation=http://iptables.org/

After=network.target docker.service

[Service]

Type=forking

ExecStart=/usr/sbin/iptables -P FORWARD ACCEPT

ExecReload=/usr/sbin/iptables -P FORWARD ACCEPT

ExecStop=/usr/sbin/iptables -P FORWARD ACCEPT

PrivateTmp=true

[Install]

WantedBy=multi-user.target

#启动forward服务

systemctl daemon-reload

systemctl enable forward

systemctl start forward

systemctl status forward

7.测试集群是否工作正常
a.创建一个deploy
kubectl run nginx --replicas=2 --labels="run=nginx-service" --image=nginx --port=80
b.映射服务到外网可访问
kubectl expose deployment nginx --type=NodePort --name=nginx-service
c.查看服务状态

kubectl describe svc example-service

Name:                   nginx-service

Namespace:              default

Labels:                 run=nginx-service

Annotations:            <none>

Selector:               run=nginx-service

Type:                   NodePort

IP:                     10.254.84.99

Port:                   <unset> 80/TCP

NodePort:               <unset> 30881/TCP

Endpoints:              172.30.1.2:80,172.30.54.2:80

Session Affinity:       None

Events:                 <none>

d.查看pods启动情况

kubectl get pods

NAME                     READY     STATUS    RESTARTS   AGE

nginx-2317272628-nsfrr   1/1       Running   0          1m

nginx-2317272628-qbbgg   1/1       Running   0          1m

e.在外网通过
http://192.168.0.210:30881
http://192.168.0.211:30881
http://192.168.0.212:30881
都可以访问nginx页面

若无法访问，可通过iptables -nL查看forward链是否开启

本文转自 rong341233 51CTO博客，原文链接:http://blog.51cto.com/fengwan/2049124

码农公寓

相关文章