七，ingress及ingress cluster

2024-03-18 12:14:34

Service 类型
namespace 名称空间
Ingress Controller
Ingress
- Ingress-nginx 进行测试
- 模拟测试 Ingress 后端 Tomcat 访问
模拟测试 Https

Service 类型

ClusterIP 只在集群内部访问，无法跨越集群边界
NodePort 提供集群外部流量进入到集群内部，是在clusterIP前提之上增强的
LoadBalancer 集群部署在公有云或私有云之上，提供给公有云或私有云的负载均衡中间使用
ExternelName 把集群外部的服务映射给集群内部使用，是一个FQDN的名称，一个CNAME名称，这个CNAM指向公网的FQDN

No ClusterIP 成为 Headless Sercie

ServiceName -> PodIP

Service 是一个四层调度器

也就是说，当请求为HTTPS请求的时候，则无法卸载证书。

namespace 名称空间

；
创建 namespace 如下：

[root@master manifests]# kubectl create namespace dev
namespace/dev created
[root@master manifests]# kubectl get ns
NAME              STATUS   AGE
default           Active   20d
dev               Active   4s       # 新创建的namespace
kube-node-lease   Active   20d
kube-public       Active   20d
kube-system       Active   20d

删除 namespace 如下：

[root@master manifests]# kubectl delete ns/dev
namespace "dev" deleted
[root@master manifests]# kubectl get ns
NAME              STATUS   AGE
default           Active   20d
kube-node-lease   Active   20d
kube-public       Active   20d
kube-system       Active   20d

Ingress Controller

举例

当kubernetes集群有上千甚至跟多个节点的时候，此时需要特有的web七层代理
如在集群其中的四个节点上打上污点，这四个节点上只运行web七层代理所对应的Pod
由此Pod来代理集群内部的Service，Service再把流量转发给集群内部对应的Pod。

这就叫做 Ingress Controller

Ingress Controller 是基于 DaemonSet 控制器来实现。

DaemonSet 是实现保证集群内部每个节点上都运行一个指定的Pod。

Kubernetes 中三种常用七层代理:

Nginx # 常规代理
Traefik # 微服务前段负载均衡
Envoy # 微服务代理负载均衡，使用的多

Ingress

Ingress 和 Ingress Controller 是两回事。

Ingress 原理

Ingress 启动一个独立的Pod来运行七层代理，可以是 Nginx 、 Traefik 或者是 Envoy，此时 ingress Pod 会直接代理后端运行服务的 Pod，为了能监听后端Pod的变化，需要一个无头Service Headless Sercie 通过标签选择器来选择后端指定的Pod，并收集到后端Pod 对应的IP，而此Service不会被使用，它主要是用于被Ingress Pod 来监听后端Pod的变化，而一旦后端Pod产生变化，无头Service 会知道，此时被Ingress Pod 发现后，会自动根据变化来更改配置文件，并重载。

如果此时使用的是 Nginx 类型的 Ingress Pod ，则每次变化后修改配置文件后都会自动重新 reload。
但如果使用 Traefik 或者是 Envoy 天生就是为微服务开发的，更友好的支持。

Ingress-nginx 进行测试

可以在 GitHub 上找到Ingress-nginx

本次使用的是相对简单部署 ingress-nginx 的配置清单。

ingress-nginx 单独的详情页面

因为是直接在物理机上直接安装的kubernetes集群，所以需要两个配置清单。

[root@master ingress]#  kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created

还需要一个Ingress-name名称空间中创建的NodePort Service，以便引入集群外流量

需要根据实际情况修改一下：

[root@master ingress]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml
[root@master ingress]# cat service-nodeport.yaml 
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
      nodePort: 30080       # 手动指定node对外的http端口
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
      nodePort: 30443       # 手动指定node对外的https端口
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

[root@master ingress]#kubectl apply -f service-nodeport.yaml
service/ingress-nginx created

查看创建的ingress-nginx Pod

[root@master ingress]# kubectl get pods -n ingress-nginx -o wide
NAME                                        READY   STATUS    RESTARTS   AGE   IP            NODE                NOMINATED NODE   READINESS GATES
nginx-ingress-controller-7995bd9c47-b4fzh   1/1     Running   0          34s   10.244.1.27   node03.kubernetes   <none>           <none>
[root@master ingress]# kubectl get svc -n ingress-nginx     # 这里是第二次执行在Ingress-name名称空间中创建的NodePort Service，以便引入集群外流量
NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx   NodePort   10.111.201.108   <none>        80:30080/TCP,443:30443/TCP   44s

创建对应的后端Pod和Service

查看创建的ingress-myapp

[root@master ingress]# cat myapp-deploy.yaml 
apiVersion: v1
kind: Service
metadata:
  name: myapp       # Service 名称
  namespace: default
spec:
  selector:
    app: myapp
    release: canary
  ports:
  - name: http
    targetPort: 80      # 指定容器端口
    port: 80            # Service 自己开房的端口
    
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-deploy
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      release: canary
  template:
    metadata:
      labels:
        app: myapp
        release: canary
    spec:
      containers:
      - name: myapp
        image: ikubernetes/myapp:v2
        ports:
        - name: http
          containerPort: 80
[root@master ingress]# kubectl apply -f myapp-deplay.yaml 
service/myapp created
deployment.apps/myapp-deploy unchanged

查看

[root@master ingress]# kubectl get pods
NAME                            READY   STATUS    RESTARTS   AGE
myapp-deploy-55b78d8548-b4c2c   1/1     Running   0          42h
myapp-deploy-55b78d8548-gpdw8   1/1     Running   0          42h
myapp-deploy-55b78d8548-hr6vx   1/1     Running   0          42h
[root@master ingress]# kubectl get svc
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1     <none>        443/TCP   22d
myapp        ClusterIP   10.111.44.7   <none>        80/TCP    6s

创建 Ingress

[root@master ingress]# cat ingress-myapp.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-myapp
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"        # 自动生成 nginx 相关的匹配和修改规则
spec:
  rules:
  - host: myapp.beijingfc.com       # 使用 host 主机模式来访问
    http:
      paths:
      - path:
        backend:
          serviceName: myapp        # 这里指定刚刚创建的后端Pod对应的Service名称
          servicePort: 80
[root@master ingress]# kubectl apply -f ingress-myapp.yaml 
ingress.extensions/ingress-myapp created
[root@master ingress]# kubectl get ing
NAME            HOSTS                 ADDRESS   PORTS   AGE
ingress-myapp   myapp.beijingfc.com             80      6s

查看 Ingress-controller 对应的Pod配置信息

上面一系列都配置完成之后，此时在看查看 Ingress-controller 的Pod中nginx的配置信息包含如下:

[root@master ingress]# kubectl exec -n ingress-nginx -it nginx-ingress-controller-7995bd9c47-b4fzh -- /bin/sh
    
   $  cat nginx.conf 
        server {
            server_name myapp.beijingfc.com ;
            listen 80;

            set $proxy_upstream_name "-";
            set $pass_access_scheme $scheme;
            set $pass_server_port $server_port;
            set $best_http_host $http_host;
            set $pass_port $pass_server_port;
            location / {

                set $namespace      "default";
                set $ingress_name   "ingress-myapp";
                set $service_name   "myapp";
                set $service_port   "80";
                set $location_path  "/";

    ... ...
    ... ...

访问测试

此时在集群外部访问测试：

注意，临时测试，上面使用的host模式来访问，所以在测试的集群外部机器上手动添加hosts记录

[root@xinguan-test-node1 ~]# grep myapp /etc/hosts
10.0.20.20 myapp.beijingfc.com
[root@xinguan-test-node1 ~]# curl myapp.beijingfc.com:30080
Hello MyApp | Version: v2 | <a href="hostname.html">Pod Name</a>
[root@xinguan-test-node1 ~]# curl myapp.beijingfc.com:30080/hostname.html
myapp-deploy-55b78d8548-hr6vx

模拟测试 Ingress 后端 Tomcat 访问

还是基于上面的配置，一切都不动，增加tomcat后端Pod和Service 以及 Ingress

创建 tomcat 的 Pod 和Service

[root@master ingress]# cat tomcat-deploy.yaml 
apiVersion: v1
kind: Service
metadata:
  name: tomcat
  namespace: default
spec:
  selector:
    app: tomcat
    release: canary
  ports:
  - name: http
    targetPort: 8080
    port: 8080
  - name: ajp
    targetPort: 8009
    port: 8009
    
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-depoly
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tomcat
      release: canary
  template:
    metadata:
      labels:
        app: tomcat
        release: canary
    spec:
      containers:
      - name: tomcat
        image: tomcat:8.5.32-jre8-alpine
        ports:
        - name: http
          containerPort: 8080
        - name: ajp
          containerPort: 8009

创建和查看

[root@master ingress]# kubectl apply -f tomcat-deploy.yaml 
service/tomcat created
deployment.apps/tomcat-depoly created
[root@master ingress]# kubectl get pods -o wide
NAME                             READY   STATUS    RESTARTS   AGE   IP            NODE                NOMINATED NODE   READINESS GATES
myapp-deploy-55b78d8548-b4c2c    1/1     Running   0          43h   10.244.3.29   node01.kubernetes   <none>           <none>
myapp-deploy-55b78d8548-gpdw8    1/1     Running   0          43h   10.244.1.26   node03.kubernetes   <none>           <none>
myapp-deploy-55b78d8548-hr6vx    1/1     Running   0          43h   10.244.2.22   node02.kubernetes   <none>           <none>
tomcat-depoly-579d97b849-66wrh   1/1     Running   0          7s    10.244.3.30   node01.kubernetes   <none>           <none>
tomcat-depoly-579d97b849-rh5r6   1/1     Running   0          7s    10.244.1.28   node03.kubernetes   <none>           <none>
tomcat-depoly-579d97b849-sd49w   1/1     Running   0          7s    10.244.2.23   node02.kubernetes   <none>           <none>
[root@master ingress]# kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP             22d
myapp        ClusterIP   10.111.44.7    <none>        80/TCP              76m
tomcat       ClusterIP   10.106.39.78   <none>        8080/TCP,8009/TCP   3m18s

创建 tomcat-ingress

[root@master ingress]# cat ingress-tomcat.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-tomcat
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: tomcat.beijingfc.com
    http:
      paths:
      - path:
        backend:
          serviceName: tomcat
          servicePort: 8080

创建和查看

[root@master ingress]# kubectl apply -f ingress-tomcat.yaml 
ingress.extensions/ingress-tomcat created
[root@master ingress]# kubectl get ing
NAME             HOSTS                  ADDRESS   PORTS   AGE
ingress-myapp    myapp.beijingfc.com              80      53m
ingress-tomcat   tomcat.beijingfc.com             80      4s

测试访问 tomcat

注意，同样写Hosts 之后进行测试

[root@xinguan-test-node1 ~]# grep tomcat /etc/hosts
10.0.20.20 myapp.beijingfc.com tomcat.beijingfc.com
[root@xinguan-test-node1 ~]# curl tomcat.beijingfc.com:30080



<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8" />
        <title>Apache Tomcat/8.5.32</title>
        <link href="favicon.ico" rel="icon" type="image/x-icon" />
        <link href="favicon.ico" rel="shortcut icon" type="image/x-icon" />
        <link href="tomcat.css" rel="stylesheet" type="text/css" />
    </head>

    <body>
        <div id="wrapper">
            <div id="navigation" class="curved container">
                <span id="nav-home"><a href="http://tomcat.apache.org/">Home</a></span>
                <span id="nav-hosts"><a href="/docs/">Documentation</a></span>
                <span id="nav-config"><a href="/docs/config/">Configuration</a></span>
                <span id="nav-examples"><a href="/examples/">Examples</a></span>
                <span id="nav-wiki"><a href="http://wiki.apache.org/tomcat/FrontPage">Wiki</a></span>
                <span id="nav-lists"><a href="http://tomcat.apache.org/lists.html">Mailing Lists</a></span>
                <span id="nav-help"><a href="http://tomcat.apache.org/findhelp.html">Find Help</a></span>
                <br class="separator" />
            </div>
            <div id="asf-box">
                <h1>Apache Tomcat/8.5.32</h1>
            </div>
... ...
... ...

模拟测试 Https

模拟测试通过Ingress 的七层代理，访问Https，通过Ingress卸载证书后访问后端

所有先需要手动自签SSL证书

自签SSL证书

[root@master https]# openssl genrsa -out tls.key 2048 
Generating RSA private key, 2048 bit long modulus
...............................+++
.....................................................................................................................................+++
e is 65537 (0x10001)
[root@master https]# openssl req -new -x509 -key tls.key -out tls.crt -subj /C=CN/ST=BeiJing/L=BeiJing/O=DefOps/CN=tomcat.beijingfc.com
[root@master https]# ll
total 8
-rw-r--r-- 1 root root 1302 Aug  1 11:33 tls.crt
-rw-r--r-- 1 root root 1675 Aug  1 11:31 tls.key

自签证书完成

创建secret

注意，此证书不能直接帖进去，需要先创建成secret，才能应用到Ingress中

[root@master https]# kubectl create secret tls tomcat-ingress-secret --cert=tls.crt --key=tls.key 
secret/tomcat-ingress-secret created
[root@master https]# kubectl get secret
NAME                    TYPE                                  DATA   AGE
default-token-bc86p     kubernetes.io/service-account-token   3      22d
tomcat-ingress-secret   kubernetes.io/tls                     2      10s
[root@master https]# kubectl describe secret tomcat-ingress-secret
Name:         tomcat-ingress-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.key:  1675 bytes
tls.crt:  1302 bytes

创建tls的https清单文件

[root@master https]# cat ingress-tomcat-tls.yaml 
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-tomcat-tls
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - tomcat.beijingfc.com
    secretName: tomcat-ingress-secret       # 这里指定刚刚创建的secret名称
  rules:
  - host: tomcat.beijingfc.com
    http:
      paths:
      - path:
        backend:
          serviceName: tomcat
          servicePort: 8080

创建和查看

[root@master https]# kubectl apply -f ingress-tomcat-tls.yaml 
ingress.extensions/ingress-tomcat-tls created
[root@master https]# kubectl get ing
NAME                 HOSTS                  ADDRESS   PORTS     AGE
ingress-myapp        myapp.beijingfc.com              80        174m
ingress-tomcat       tomcat.beijingfc.com             80        121m
ingress-tomcat-tls   tomcat.beijingfc.com             80, 443   4s          # 这里看到被创建成功

[root@master https]# kubectl describe ingress ingress-tomcat-tls
Name:             ingress-tomcat-tls
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<none>)
TLS:
  tomcat-ingress-secret terminates tomcat.beijingfc.com
Rules:
  Host                  Path  Backends
  ----                  ----  --------
  tomcat.beijingfc.com  
                           tomcat:8080 (10.244.1.28:8080,10.244.2.23:8080,10.244.3.30:8080)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx"},"name":"ingress-tomcat-tls","namespace":"default"},"spec":{"rules":[{"host":"tomcat.beijingfc.com","http":{"paths":[{"backend":{"serviceName":"tomcat","servicePort":8080},"path":null}]}}],"tls":[{"hosts":["tomcat.beijingfc.com"],"secretName":"tomcat-ingress-secret"}]}}

  kubernetes.io/ingress.class:  nginx
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  CREATE  26s   nginx-ingress-controller  Ingress default/ingress-tomcat-tls

测试访问

此时就已经完成，可以测试访问

因前面测试hosts已经写好

码农公寓

Service 类型

namespace 名称空间

Ingress Controller

Ingress

Ingress-nginx 进行测试

创建对应的后端Pod和Service

创建 Ingress

查看 Ingress-controller 对应的Pod配置信息

访问测试

模拟测试 Ingress 后端 Tomcat 访问

创建 tomcat 的 Pod 和Service

创建 tomcat-ingress

测试访问 tomcat

模拟测试 Https

自签SSL证书

创建secret

创建tls的https清单文件

测试访问

相关文章