Daemonset方式部署ingress-nginx结合keepalived实现高可用

2021-07-07 01:17:08

概述：

官方原始文件使用的是deployment，replicate 为 1，这样将会在某一台节点上启动对应的nginx-ingress-controller pod。外部流量访问至该节点，由该节点负载分担至内部的service。考虑到单点故障的问题，改为DaemonSet然后删掉replicate ，配合亲和性部署在指定节点上启动nginx-ingress-controller pod，确保有多个节点启动nginx-ingress-controller pod，后续将这些节点加入到外部硬件负载均衡组实现高可用性。

 

介绍：

1.使用daemonset方式将ingress-controller部署在相应节点，一般在k8s-node节点上，master节点为了集群稳定不建议部署。

2.如果是云服务器，可直接申请使用负载均衡高可用IP对应解析访问域名，虚拟IP绑定后端node服务器，达到高可用目的。

　　

　　下面针对的是企业自建服务器部署高可用架构：

 

大致结构：

在Kubernetes中添加了ingress后，在企业机房网络DNS中添加A记录，域名为你的ingress中host的内容(备案域名)，IP为你的keepalived的VIP(内网IP)，

这样集群外部就可以通过域名来访问你的服务，也解决了单点故障。

选择Kubernetes部署了ingress的三个node作为节点，都安装keepalived。

 

修改配置文件/etc/keepalived/keepalived.conf

除了priority优先级不一样，其他三个node节点都一样

注意：修改如下所示

Daemonset方式部署ingress-nginx结合keepalived实现高可用

 

启动keeplived

systemctl start keepalived
systemctl enable keepalived

 

创建deployment和service

$ vim deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment 
metadata: 
  name: deployment 
spec: 
  replicas: 2 
  template: 
    metadata: 
      labels: 
        name: nginx 
    spec: 
      containers: 
      - name: nginx 
        image: wangyanglinux/myapp:v3 
        imagePullPolicy: IfNotPresent 
        ports: 
        - containerPort: 80

---
apiVersion: v1 
kind: Service 
metadata: 
  name: svc-1 
spec: 
  ports: 
  - port: 80 
    targetPort: 80 
    protocol: TCP 
  selector: 
    name: nginx #当name=nginx时匹配

$ kubectl apply -f deployment.yaml

 

 

安装ingress-nginx-controller

1.给要部署的ingress节点打标签

nginx-ingress-controller会随意选择一个node节点运行pod，为此需要我们把nginx-ingress-controller运行到指定的node节点上。
首先需要给需要运行nginx-ingress-controller的node节点打标签

kubectl label nodes k8s-node01 edgenode=true
kubectl label nodes k8s-node02 edgenode=true
kubectl label nodes k8s-node03 edgenode=true

查看node标签

kubectl get node --show-labels

 

2.daemonset形式安装ingress-nginx-controller(修改原来ingress部署的yaml文件，注意修改标红处)

(deployment改为daemonset、注释replicate、添加hostNetwork、添加亲和性属性)

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend 
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: registry.cn-hangzhou.aliyuncs.com/google_containers/defaultbackend:1.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---
 
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: ingress-nginx
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend
---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true
      nodeSelector:
        edgenode: 'true'
      containers:
        - name: nginx-ingress-controller
          image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.20.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          # livenessProbe:
          #   failureThreshold: 3
          #   httpGet:
          #     path: /healthz
          #     port: 10254
          #     scheme: HTTP
          #   initialDelaySeconds: 10
          #   periodSeconds: 10
          #   successThreshold: 1
          #   timeoutSeconds: 1
          # readinessProbe:
          #   failureThreshold: 3
          #   httpGet:
          #     path: /healthz
          #     port: 10254
          #     scheme: HTTP
          #   periodSeconds: 10
          #   successThreshold: 1
          #   timeoutSeconds: 1
---

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
 
---
 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update
 
---
 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get
 
---
 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx
 
---
 
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
---

kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx

 

应用资源清单

kubectl apply -f ingress-nginx.yaml

 

查看安装是否成功

kubectl get ds -n ingress-nginx
kubectl get pods -n ingress-nginx -o wide

 

 

Ingress HTTPS 代理访问

创建https证书的secret

kubectl create secret tls tls-secret --key tls.key --cert tls.crt

 

创建ingress策略

$ vim https_ingress.yaml

apiVersion: extensions/v1beta1 
kind: Ingress 
metadata: 
  name: https 
spec: 
  tls:
  - hosts: 
    - www.test.com 
    secretName: tls-secret #上面创建时保存的secret名称
  rules: 
    - host: www.test.com 
      http: 
        paths: 
        - path: / 
          backend: 
            serviceName: svc-1
            servicePort: 80

$ kubectl apply -f https_ingress.yaml

 

测试

 

上一篇：WinUI get Launched Param, args.UWPLaunchActivatedEventArgs.Kind is aways returning as 'launch


下一篇：Canal数据同步