K8S 安装及部署 Nginx 记录文档

2021-07-16 00:23:50

K8S 部署文档记录

本文旨在对通过基于 Virtual Box 和 Vagrant 安装 CentOS虚拟机，在虚拟机中安装 K8S 集群并部署 Nginx 做一次记录，仅供参考。

K8S 部署文档记录

环境

序号	名称	配置	说明	其他
1	Virtual Box		用来安装虚拟机
2	Vagrant		用来安装 CentOS

虚拟机信息

Tips: 虚拟机与主机之间用 NAT 通信

序号	hostname	ip	配置	其他
1	master	192.168.33.10	cpu:2,memory:2048
2	node1	192.168.33.11	cpu:1,memory:1024

安装基础环境

对于 Virtual Box 和 Vagrant 及基于 Virtual Box 和 Vagrant 安装 CentOS 在这就不细说了，网上有很多教程；在这我提供我的 CentOS 的安装配置。

master

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.box = "centos/7"
  config.vm.network "private_network", ip: "192.168.33.10"
   config.vm.provider "virtualbox" do |vb|
     vb.gui = false
     vb.cpus = 2
     vb.memory = "2048"
   end
end

node1

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.box = "centos/7"
  config.vm.network "private_network", ip: "192.168.33.11"
   config.vm.provider "virtualbox" do |vb|
     vb.gui = false
     vb.memory = "1024"
   end
end

更新 CentOS 主机名

以下步骤 master 和 node1 均需更新

[root@vagrant ~]# su root
Password:
[root@master ~]# hostnamectl set-hostname master

IP 映射

以下步骤 master 和 node1 均需更新

[root@master ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.33.10   master
192.168.33.11   node1
[root@master ~]# reboot

K8S 安装前期准备

以下步骤 master 和 node1 均需更新

1. 执行脚本preENV.sh

[root@master ~]# cat preENV.sh
#!/bin/bash
# 关闭防火墙
systemctl stop firewalld && systemctl disable firewalld
# 关闭SELINUX
setenforce 0 && sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
# 关闭Swap
swapoff -a && sed -i "s/\/dev\/mapper\/centos-swap/\#\/dev\/mapper\/centos-swap/g" /etc/fstab

2. 创建 /etc/sysctl.d/k8s.conf

[root@master ~]# cat /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

3. 执行命令使修改生效

[root@master ~]# modprobe br_netfilter
[root@master ~]# sysctl -p /etc/sysctl.d/k8s.conf

4. kube-proxy 开启 ipvs 的前置条件

[root@master ~]# cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
[root@master ~]# chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

安装 Docker

以下步骤 master 和 node1 均需更新

[root@master ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
[root@master ~]# wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@master ~]# yum makecache fast
[root@master ~]# yum install -y docker-ce
[root@master ~]# systemctl start docker
[root@master ~]# systemctl enable docker

K8S 安装

配置镜像

以下步骤 master 和 node1 均需更新

[root@master ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
[root@master ~]# yum makecache fast
[root@master ~]# yum install -y kubelet kubeadm kubectl
[root@master ~]# systemctl enable kubelet

Master 节点处理

Master 节点镜像映射

[root@master ~]# vim k8sMasterImages.sh
#!/bin/bash

set -e

KUBE_VERSION=v1.14.1
KUBE_PAUSE_VERSION=3.1
ETCD_VERSION=3.3.10
CORE_DNS_VERSION=1.3.1

GCR_URL=k8s.gcr.io
ALIYUN_URL=registry.cn-hangzhou.aliyuncs.com/google_containers

images=(kube-proxy:${KUBE_VERSION}
kube-scheduler:${KUBE_VERSION}
kube-controller-manager:${KUBE_VERSION}
kube-apiserver:${KUBE_VERSION}
pause:${KUBE_PAUSE_VERSION}
etcd:${ETCD_VERSION}
coredns:${CORE_DNS_VERSION})


for imageName in ${images[@]} ; do
  docker pull $ALIYUN_URL/$imageName
  docker tag  $ALIYUN_URL/$imageName $GCR_URL/$imageName
  docker rmi $ALIYUN_URL/$imageName
done
[root@master ~]# chmod +x k8sMasterImages.sh
[root@master ~]# ./k8sMasterImages.sh

Master 节点初始化

以下内容复制到物理机上保存

[root@master ~]# kubeadm init --kubernetes-version=v1.14.1 --apiserver-advertise-address=192.168.33.10 --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.14.1
[preflight] Running pre-flight checks
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
    [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master localhost] and IPs [192.168.33.10 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master localhost] and IPs [192.168.33.10 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.33.10]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 17.502789 seconds
[upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --experimental-upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 6ftxi9.gov5rsp9syw1fect
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.33.10:6443 --token 6ftxi9.gov5rsp9syw1fect \
    --discovery-token-ca-cert-hash sha256:e474c065e6c7c69c12ec37b24cedb5745f941231c532032921553766789a4a5e

说明

--kubernetes-version=v1.14.1 : 指定K8S版本
--apiserver-advertise-address=192.168.33.10 : 指定apiserver 的地址，一般为主节点的地址，也就是虚拟机的地址
--pod-network-cidr=10.244.0.0/16 : Pod的地址范围，其值为CIDR格式的网络地址；计划使用flannel网络插件；使用flannel网络插件时，其默认地址为10.244.0.0/16.

执行如下命令

[root@master ~]# mkdir -p $HOME/.kube
[root@master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

查看集群状态

[root@master ~]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health":"true"}

结果显示和上面的输出一致就 ok 了。

查看 Pod 信息

[root@master ~]# kubectl get pod -A -o wide
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE     IP              NODE     NOMINATED NODE   READINESS GATES
kube-system   coredns-fb8b8dccf-88hfr          0/1     Pending   0          6m33s   <none>          <none>   <none>           <none>
kube-system   coredns-fb8b8dccf-jmw7m          0/1     Pending   0          6m33s   <none>          <none>   <none>           <none>
kube-system   etcd-master                      1/1     Running   0          5m42s   192.168.33.10   master   <none>           <none>
kube-system   kube-apiserver-master            1/1     Running   0          5m46s   192.168.33.10   master   <none>           <none>
kube-system   kube-controller-manager-master   1/1     Running   0          5m47s   192.168.33.10   master   <none>           <none>
kube-system   kube-proxy-46wth                 1/1     Running   0          6m32s   192.168.33.10   master   <none>           <none>
kube-system   kube-scheduler-master            1/1     Running   0          5m40s   192.168.33.10   master   <none>           <none>

到这里，会发现除了 coredns 未 ready，其他 pod 都正常 Running这是正常的，因为还没有网络插件，接下来将 node1 加入集群后在主节点安装 flannel 后就正常了，稍后操作。

将主节点也设置为工作节点,参与负载

[root@master ~]# kubectl describe node master | grep Taint
Taints:             node-role.kubernetes.io/master:NoSchedule
[root@master ~]# kubectl taint nodes master node-role.kubernetes.io/master-
node "master" untainted

node1 节点处理

在上面的步骤中，我们完成了关闭防火墙、关闭SELINUX、创建k8s.conf 配置文件，kube-proxy 开启 ipvs 和 Docker、kubelet、kubectl、kubeadm 等的安装工作，接下来我们将拉取 node1 上用到的镜像，并将其加入到 master 节点上，形成 2 个节点的集群。

安装镜像

[vagrant@node1 ~]$ cat k8sNodeImages.sh
#!/bin/bash

set -e

KUBE_VERSION=v1.14.1
KUBE_PAUSE_VERSION=3.1

GCR_URL=k8s.gcr.io
ALIYUN_URL=registry.cn-hangzhou.aliyuncs.com/google_containers

images=(kube-proxy-amd64:${KUBE_VERSION}
pause:${KUBE_PAUSE_VERSION})


for imageName in ${images[@]} ; do
  docker pull $ALIYUN_URL/$imageName
  docker tag  $ALIYUN_URL/$imageName $GCR_URL/$imageName
  docker rmi $ALIYUN_URL/$imageName
done
[root@node1 ~]# chmod +x k8sNodeImages.sh
[root@node1 ~]# ./k8sNodeImages.sh

执行 master 上初始化生成的将 node 加入 master 的命令

[root@node1 ~]# kubeadm join 192.168.33.10:6443 --token 6ftxi9.gov5rsp9syw1fect \
>     --discovery-token-ca-cert-hash sha256:e474c065e6c7c69c12ec37b24cedb5745f941231c532032921553766789a4a5e
[preflight] Running pre-flight checks
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
[root@node1 ~]#

master 节点处理

查看集群信息

在此之前 master 节点的状态可能为 NotReady, 可以忽略，在安装 flannel 之后就会变为 Ready。

[root@master ~]# kubectl get nodes
NAME     STATUS     ROLES    AGE   VERSION
master   Ready      master   58m   v1.14.1
node1    NotReady   <none>   13s   v1.14.1

刚加入集群的 node1 状态是 NotReady 状态，等几分钟就 Ready 了，可以通过如下命令来监控

[root@master ~]# watch kubectl get nodes
Every 2.0s: kubectl get node                                                                                                                            Sun May 12 02:48:00 2019

NAME     STATUS   ROLES    AGE   VERSION
master   Ready    master   12h   v1.14.1
node1    Ready    <none>   11h   v1.14.1

部署 flannel

推荐等到 node1 状态 Ready 之后再安装 flannel
因为 flannel 中的有些镜像在 quay-mirror.qiniu.com，国内无法访问，故可以转到阿里云镜像中心获取，然后再将其重新打 tag 为 quay.io/coreos/flannel:v0.11.0-amd64 中的名称即可。

获取阿里云镜像仓库中的镜像

[root@master ~]# cat processFlannelImage.sh
#!/bin/bash
docker pull quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64
docker tag quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64
[root@master ~]# chmod +x processFlannelImage.sh
[root@master ~]# ./processFlannelImage.sh

安装 flannel

[root@master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
podsecuritypolicy.extensions/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

查看 pod 信息

[root@master ~]# kubectl get pod -A -o wide
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE    IP              NODE     NOMINATED NODE   READINESS GATES
kube-system   coredns-fb8b8dccf-88hfr            1/1     Running   0          12h    10.244.0.3      master   <none>           <none>
kube-system   coredns-fb8b8dccf-jmw7m            1/1     Running   0          12h    10.244.0.2      master   <none>           <none>
kube-system   etcd-master                        1/1     Running   0          12h    192.168.33.10   master   <none>           <none>
kube-system   kube-apiserver-master              1/1     Running   0          12h    192.168.33.10   master   <none>           <none>
kube-system   kube-controller-manager-master     1/1     Running   0          12h    192.168.33.10   master   <none>           <none>
kube-system   kube-flannel-ds-amd64-5jtfd        1/1     Running   0          11h    192.168.33.11   node1    <none>           <none>
kube-system   kube-flannel-ds-amd64-bgv9j        1/1     Running   0          11h    192.168.33.10   master   <none>           <none>
kube-system   kube-proxy-46wth                   1/1     Running   0          12h    192.168.33.10   master   <none>           <none>
kube-system   kube-proxy-r86jk                   1/1     Running   0          11h    192.168.33.11   node1    <none>           <none>
kube-system   kube-scheduler-master              1/1     Running   0          12h    192.168.33.10   master   <none>           <none>

现在可以发现所有的 pod 都正常启动了。

部署 Nginx

部署 Nginx pod

[root@master ~]# cat nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
[root@master ~]# kubectl apply -f nginx-deployment.yaml
deployment.apps/nginx-deployment created
[root@master ~]# kubectl get pods
NAME                               READY   STATUS              RESTARTS   AGE
nginx-deployment-6dd86d77d-fb9bp   0/1     ContainerCreating   0          101s
nginx-deployment-6dd86d77d-lzc56   1/1     Running             0          101s
nginx-deployment-6dd86d77d-p7pd8   1/1     Running             0          101s

创建 service, 并让外部访问

[root@master ~]# cat nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  labels:
    app: nginx
spec:
  ports:
  - port: 88
    targetPort: 80
  selector:
    app: nginx
  type: NodePort
[root@master ~]# kubectl create -f nginx-service.yaml
service/nginx-service created
[root@master ~]# kubectl get service/nginx-service
NAME            TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
nginx-service   NodePort   10.99.114.248   <none>        88:31345/TCP   14s

service 说明

为了让主机访问 k8s 提供的nginx 服务，我们需要在 nginx-service.yaml 规范 spec 中指定 type 为 NodePort。

在浏览器中访问

访问地址格式为 MasterIP:NodePort, NodePort 为kubectl get service/nginx-servic 后 88:31345/TCP中 31345这个端口；本次为http://192.168.33.10:31345/

总结

不需要配置 ssh, 有些文档中写需要配置，但是没必要
部署前一定要对部署的环境变量设置
配置信息尽量脚本化
master 初始化时的参数一定要指定正确并知道其含义
在初始化好 master 节点后，先不要部署网络相关的内容，先将 node 加入集群，并等到 node 的状态为 Ready 后再部署网络相关的组件

码农公寓

K8S 部署文档记录

环境

虚拟机信息

安装基础环境

master

node1

更新 CentOS 主机名

IP 映射

K8S 安装前期准备

1. 执行脚本preENV.sh

2. 创建 /etc/sysctl.d/k8s.conf

3. 执行命令使修改生效

4. kube-proxy 开启 ipvs 的前置条件

安装 Docker

K8S 安装

配置镜像

Master 节点处理

Master 节点镜像映射

Master 节点初始化

说明

执行如下命令

查看集群状态

查看 Pod 信息

将主节点也设置为工作节点,参与负载

node1 节点处理

安装镜像

执行 master 上初始化生成的将 node 加入 master 的命令

master 节点处理

查看集群信息

部署 flannel

获取阿里云镜像仓库中的镜像

安装 flannel

查看 pod 信息

部署 Nginx

部署 Nginx pod

创建 service, 并让外部访问

service 说明

在浏览器中访问

总结

相关文章