Istio的流量管理(实操二)(istio 系列四)

2023-07-25 12:49:58

Istio的流量管理(实操二)(istio 系列四)

涵盖官方文档Traffic Management章节中的inrgess部分。

Istio的流量管理(实操二)(istio 系列四)

Ingress网关

在kubernetes环境中，kubernetes ingress资源用于指定暴露到集群外的服务。在istio服务网格中，使用了一种不同的配置模型，称为istio网关。一个网关允许将istio的特性，如镜像和路由规则应用到进入集群的流量上。

本节描述了如何使用istio网关将一个服务暴露到服务网格外。

环境准备

使用如下命令创建httpbin服务

$ kubectl apply -f samples/httpbin/httpbin.yaml

确定ingress的IP和端口

由于本环境中没有配置对外的负载均衡，因此此处的EXTERNAL-IP为空，使用node port进行访问

# kubectl get svc istio-ingressgateway -n istio-system

NAME                    TYPE           CLUSTER-IP    EXTERNAL-IP  PORT(S)    AGE

istio-ingressgateway   LoadBalancer   10.84.93.45   <pending>     ...        11d

获取ingressgateway service的http2和https对应的端口

$ export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')

$ export SECURE_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}')

$ export TCP_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="tcp")].nodePort}')

下面是istio-system命名空间的istio-ingressgateway service中的一部分端口信息，可以看到http2和https的nodeport分别为31194和31785，对应上面的INGRESS_PORT和SECURE_INGRESS_PORT

{

    "name": "http2",

    "nodePort": 31194,

    "port": 80,

    "protocol": "TCP",

    "targetPort": 80

},

{

    "name": "https",

    "nodePort": 31785,

    "port": 443,

    "protocol": "TCP",

    "targetPort": 443

},

获取istio-system命名空间中ingressgateway pod 的hostIP

$ export INGRESS_HOST=$(kubectl get po -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].status.hostIP}')

使用istio网关配置ingress

一个ingress网关描述了在网格边缘用于接收入站HTTP/TCP连接的负载均衡，配置了暴露的端口，协议等。kubernetes ingress资源不包括任何流量路由配置。ingress 流量的路由使用istio路由规则，与内部服务请求相同：

创建istio Gateway，将来自httpbin.example.com的流量导入网格的80端口(即默认的ingressgatewaypod)

kubectl apply -f - <<EOF

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:

  name: httpbin-gateway

spec:

  selector:

    istio: ingressgateway # use Istio default gateway implementation

  servers:

  - port:

      number: 80

      name: http

      protocol: HTTP

    hosts:

    - "httpbin.example.com"

EOF

通过Gateway配置进入的流量路由，将来自httpbin.example.com，且目的地为/status或/delay的请求分发到httpbin服务的8000端口，其他请求会返回404响应。
```
kubectl apply -f - <<EOF

apiVersion: networking.istio.io/v1alpha3

kind: VirtualService

metadata:

  name: httpbin

spec:

  hosts:

  - "httpbin.example.com"

  gateways:

  - httpbin-gateway

  http:

  - match:

    - uri:

        prefix: /status

    - uri:

        prefix: /delay

    route:

    - destination:

        port:

          number: 8000

        host: httpbin

EOF
```
可以看到流量被导入了httpbin service暴露的8000端口上
```
$ oc get svc |grep httpbin

httpbin       ClusterIP      10.84.222.69   <none>    8000/TCP   19h
```
来自网格内部其他服务的请求则不受此规则的约束，会使用默认的轮询路由进行请求分发。为了限制内部的调用规则，可以将特定的值mesh添加到gateways列表中。由于内部服务的主机名可能与外部不同，因此需要将主机名添加到hosts列表中。

使用curl命令访问httpbin服务，此时通过-H选项修改了HTTP请求首部的Host字段，使用http2的nodeport方式访问：

$ curl -s -I -HHost:httpbin.example.com http://$INGRESS_HOST:$INGRESS_PORT/status/200

HTTP/1.1 200 OK

server: istio-envoy

date: Thu, 21 May 2020 03:22:50 GMT

content-type: text/html; charset=utf-8

access-control-allow-origin: *

access-control-allow-credentials: true

content-length: 0

x-envoy-upstream-service-time: 21

访问其他URL路径则返回404错误

$ curl -s -I -HHost:httpbin.example.com http://$INGRESS_HOST:$INGRESS_PORT/headers

HTTP/1.1 404 Not Found

date: Thu, 21 May 2020 03:25:20 GMT

server: istio-envoy

transfer-encoding: chunked

使用浏览器访问ingress服务

由于无法像使用curl一样修改请求的Host首部字段，因此无法使用浏览器访问httpbin服务。为了解决这个问题，可以在Gateway和VirtualService中的host字段使用通配符*。

$ kubectl apply -f - <<EOF

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:

  name: httpbin-gateway

spec:

  selector:

    istio: ingressgateway # use Istio default gateway implementation

  servers:

  - port:

      number: 80

      name: http

      protocol: HTTP

    hosts:

    - "*"  #指定通配符，不限制外部流量的地址

---

apiVersion: networking.istio.io/v1alpha3

kind: VirtualService

metadata:

  name: httpbin

spec:

  hosts:

  - "*"

  gateways:

  - httpbin-gateway

  http:

  - match:

    - uri:

        prefix: /headers

    route:

    - destination:

        port:

          number: 8000

        host: httpbin

EOF

问题定位

检查环境变量INGRESS_HOST和INGRESS_PORT，保证这两个值是有效的

$ kubectl get svc -n istio-system

$ echo INGRESS_HOST=$INGRESS_HOST, INGRESS_PORT=$INGRESS_PORT

校验相同的端口上没有其他istio ingress网格
```
$ kubectl get gateway --all-namespaces
```
校验没有在相同的IP和端口上定义kubernetes ingress资源
```
$ kubectl get ingress --all-namespaces
```
如果没有负载均衡，可以参照上面步骤使用node port方式

卸载

$ kubectl delete gateway httpbin-gateway

$ kubectl delete virtualservice httpbin

$ kubectl delete --ignore-not-found=true -f samples/httpbin/httpbin.yaml

Ingress(kubernetes)

执行ingress流量控制中的Before you begin 的Before you begin 和Determining the ingress IP and ports小节的操作，部署httpbin服务。本节介绍如何通过kubernetes的Ingress(非istio的gateway)进行访问。

下面展示如何配置一个80端口的Ingress，用于HTTP流量：

创建一个istio Gateway，将来自httpbin.example.com:80/status/*的流量分发到service httpbin的8000端口

$ kubectl apply -f - <<EOF

apiVersion: networking.k8s.io/v1beta1

kind: Ingress

metadata:

  annotations:

    kubernetes.io/ingress.class: istio

  name: ingress

spec:

  rules:

  - host: httpbin.example.com

    http:

      paths:

      - path: /status/*

        backend:

          serviceName: httpbin

          servicePort: 8000

EOF

注意需要使用 kubernetes.io/ingress.class annotation来告诉istio网关控制器处理该ingress，否则会忽略该ingress。

使用curl命令访问httpbin服务。Ingress的流量也需要经过istio ingressgateway。

$ curl -I -HHost:httpbin.example.com http://$INGRESS_HOST:$INGRESS_PORT/status/200

HTTP/1.1 200 OK

server: istio-envoy

date: Fri, 22 May 2020 06:12:56 GMT

content-type: text/html; charset=utf-8

access-control-allow-origin: *

access-control-allow-credentials: true

content-length: 0

x-envoy-upstream-service-time: 20

httpbin服务的发现是通过EDS实现的，使用如下命令查看：

$ istioctl proxy-config cluster istio-ingressgateway-569669bb67-b6p5r|grep 8000

httpbin.default.svc.cluster.local                   8000    -    outbound      EDS

outbound_.8000_._.httpbin.default.svc.cluster.local   -     -       -          EDS

访问其他未暴露的路径，返回HTTP 404错误：

$ curl -I -HHost:httpbin.example.com http://$INGRESS_HOST:$INGRESS_PORT/headers

HTTP/1.1 404 Not Found

date: Fri, 22 May 2020 06:24:30 GMT

server: istio-envoy

transfer-encoding: chunked

下一步

TLS

Ingress支持TLS设置。Istio也支持TLS设置，但相关的secret必须存在istio-ingressgateway deployment所在的命名空间中。可以使用cert-manager生成这些证书。

指定路径类型

默认情况下，Istio会将路径视为完全匹配，如果路径使用/*或*结尾，则该路径为前缀匹配。不支持其他正则匹配。

kubernetes 1.18中新增了一个字段pathType，允许声明为Exact或Prefix。

指定`IngressClass`

kubernetes 1.18中新增了一个资源IngressClass，替换了Ingress资源的 kubernetes.io/ingress.class annotation。如果使用该资源，则需要将controller设置为istio.io/ingress-controller：

apiVersion: networking.k8s.io/v1beta1

kind: IngressClass

metadata:

  name: istio

spec:

  controller: istio.io/ingress-controller

---

apiVersion: networking.k8s.io/v1beta1

kind: Ingress

metadata:

  name: ingress

spec:

  ingressClassName: istio

  ...

卸载

$ kubectl delete ingress ingress

$ kubectl delete --ignore-not-found=true -f samples/httpbin/httpbin.yaml

安全网关

本节讲述如何使用simple或mutual TLS暴露安全的HTTPS服务。证书是通过SDS进行密钥发现的。

TLS需要的私钥，服务端证书，根证书是使用基于文件装载的方法配置的。

执行ingress流量控制中的Before you begin 的Before you begin 和Determining the ingress IP and ports小节的操作，部署httpbin服务，并获取 INGRESS_HOST 和SECURE_INGRESS_PORT变量。

生成服务端证书和私钥

下面使用openssl生成需要的证书和密钥

生成一个根证书和一个私钥，用于签名服务的证书

$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example.com.key -out example.com.crt

为 httpbin.example.com生成一个证书和私钥

$ openssl req -out httpbin.example.com.csr -newkey rsa:2048 -nodes -keyout httpbin.example.com.key -subj "/CN=httpbin.example.com/O=httpbin organization"

$ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in httpbin.example.com.csr -out httpbin.example.com.crt

单主机配置TLS ingress网关

为ingree网关创建一个secret

secret的名字不能以istio或prometheus开头，且secret不应该包含token字段

$ kubectl create -n istio-system secret tls httpbin-credential --key=httpbin.example.com.key --cert=httpbin.example.com.crt

在server部分定义一个443端口的Gateway，将credentialName指定为httpbin-credential，与secret的名字相同，TLS的mode为SIMPLE。

$ cat <<EOF | kubectl apply -f -

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:

  name: mygateway

spec:

  selector:

    istio: ingressgateway # use istio default ingress gateway

  servers:

  - port:

      number: 443

      name: https

      protocol: HTTPS

    tls:

      mode: SIMPLE

      credentialName: httpbin-credential # must be the same as secret

    hosts:

    - httpbin.example.com

EOF

配置进入Gateway的流量路由。与上一节中的VirtualService相同

$ kubectl apply -f - <<EOF

apiVersion: networking.istio.io/v1alpha3

kind: VirtualService

metadata:

  name: httpbin

spec:

  hosts:

  - "httpbin.example.com"

  gateways:

  - httpbin-gateway

  http:

  - match:

    - uri:

        prefix: /status

    - uri:

        prefix: /delay

    route:

    - destination:

        port:

          number: 8000  #可以看到tls只是这是在gateway上的，当进入网格之后就不需要了

        host: httpbin

EOF

使用curl向SECURE_INGRESS_PORT发送HTTPS请求访问httpbin服务，请求中携带了公钥example.com.crt 。--resolve标记可以在使用curl访问TLS的网关IP时，在SNI中支持httpbin.example.com。--cacert选择支持使用生成的证书校验服务。

-HHost:httpbin.example.com 选项仅在SECURE_INGRESS_PORT不同于实际的网关端口(443)时才会需要，例如，通过映射的NodePort方式访问服务时。

通过将请求发送到/status/418 URL路径时，可以看到httpbin确实被访问了，httpbin服务会返回418 I’m a Teapot代码。.

$ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \

--cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

> --cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

* Added httpbin.example.com:31967:172.20.127.211 to DNS cache

* About to connect() to httpbin.example.com port 31967 (#0)

*   Trying 172.20.127.211...

* Connected to httpbin.example.com (172.20.127.211) port 31967 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

*   CAfile: example.com.crt

  CApath: none

* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

* Server certificate:

*       subject: O=httpbin organization,CN=httpbin.example.com

*       start date: May 22 09:03:01 2020 GMT

*       expire date: May 22 09:03:01 2021 GMT

*       common name: httpbin.example.com

*       issuer: CN=example.com,O=example Inc.

> GET /status/418 HTTP/1.1

> User-Agent: curl/7.29.0

> Accept: */*

> Host:httpbin.example.com

>

< HTTP/1.1 418 Unknown

< server: istio-envoy

< date: Fri, 22 May 2020 09:08:29 GMT

< x-more-info: http://tools.ietf.org/html/rfc2324

< access-control-allow-origin: *

< access-control-allow-credentials: true

< content-length: 135

< x-envoy-upstream-service-time: 2

<

    -=[ teapot ]=-

       _...._

     .'  _ _ `.

    | ."` ^ `". _,

    \_;`"---"`|//

      |       ;/

      \_     _/

        `"""`

* Connection #0 to host httpbin.example.com left intact

查看curl输出中的Server certificate中的信息，上述返回值的最后有一个茶壶的图片，说明运行成功。

删除老的网关secret，创建一个新的secret，并使用该secret修改ingress网关的凭据

$ kubectl -n istio-system delete secret httpbin-credential

$ mkdir new_certificates

$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout new_certificates/example.com.key -out new_certificates/example.com.crt

$ openssl req -out new_certificates/httpbin.example.com.csr -newkey rsa:2048 -nodes -keyout new_certificates/httpbin.example.com.key -subj "/CN=httpbin.example.com/O=httpbin organization"

$ openssl x509 -req -days 365 -CA new_certificates/example.com.crt -CAkey new_certificates/example.com.key -set_serial 0 -in new_certificates/httpbin.example.com.csr -out new_certificates/httpbin.example.com.crt

$ kubectl create -n istio-system secret tls httpbin-credential \

--key=new_certificates/httpbin.example.com.key \

--cert=new_certificates/httpbin.example.com.crt

使用新的证书链访问httpbin服务

$ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \

--cacert new_certificates/example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

如果使用老的证书访问，则返回错误

$ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \

> --cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

* Added httpbin.example.com:31967:172.20.127.211 to DNS cache

* About to connect() to httpbin.example.com port 31967 (#0)

*   Trying 172.20.127.211...

* Connected to httpbin.example.com (172.20.127.211) port 31967 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

*   CAfile: example.com.crt

  CApath: none

* Server certificate:

*       subject: O=httpbin organization,CN=httpbin.example.com

*       start date: May 22 09:24:07 2020 GMT

*       expire date: May 22 09:24:07 2021 GMT

*       common name: httpbin.example.com

*       issuer: CN=example.com,O=example Inc.

* NSS error -8182 (SEC_ERROR_BAD_SIGNATURE)

* Peer's certificate has an invalid signature.

* Closing connection 0

curl: (60) Peer's certificate has an invalid signature.

多主机配置TLS ingress网关

本节会为多个主机(httpbin.example.com和helloworld-v1.example.com)配置一个ingress网关。ingress网关会在credentialName中查找唯一的凭据。

删除之前创建的secret并为httpbin重建凭据

$ kubectl -n istio-system delete secret httpbin-credential

$ kubectl create -n istio-system secret tls httpbin-credential \

--key=httpbin.example.com.key \

--cert=httpbin.example.com.crt

启动helloworld-v1

$ cat <<EOF | kubectl apply -f -

apiVersion: v1

kind: Service

metadata:

  name: helloworld-v1

  labels:

    app: helloworld-v1

spec:

  ports:

  - name: http

    port: 5000

  selector:

    app: helloworld-v1

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: helloworld-v1

spec:

  replicas: 1

  selector:

    matchLabels:

      app: helloworld-v1

      version: v1

  template:

    metadata:

      labels:

        app: helloworld-v1

        version: v1

    spec:

      containers:

      - name: helloworld

        image: istio/examples-helloworld-v1

        resources:

          requests:

            cpu: "100m"

        imagePullPolicy: IfNotPresent #Always

        ports:

        - containerPort: 5000

EOF

为helloworld-v1.example.com创建证书和私钥

$ openssl req -out helloworld-v1.example.com.csr -newkey rsa:2048 -nodes -keyout helloworld-v1.example.com.key -subj "/CN=helloworld-v1.example.com/O=helloworld organization"

$ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 1 -in helloworld-v1.example.com.csr -out helloworld-v1.example.com.crt

为helloworld-credential创建secret

$ kubectl create -n istio-system secret tls helloworld-credential --key=helloworld-v1.example.com.key --cert=helloworld-v1.example.com.crt

定义两个网关，网关端口为443。在credentialName字段分别设置httpbin-credential和helloworld-credential，TLS模式为SIMPLE。

$ cat <<EOF | kubectl apply -f -

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:

  name: mygateway

spec:

  selector:

    istio: ingressgateway # use istio default ingress gateway

  servers:

  - port:

      number: 443

      name: https-httpbin #httpbin的gateway配置

      protocol: HTTPS

    tls:

      mode: SIMPLE

      credentialName: httpbin-credential

    hosts:

    - httpbin.example.com

  - port:

      number: 443

      name: https-helloworld #https-helloword的gateway配置

      protocol: HTTPS

    tls:

      mode: SIMPLE

      credentialName: helloworld-credential

    hosts:

    - helloworld-v1.example.com

EOF

配置gateway的流量路由，为新应用添加对应的virtual service

$ cat <<EOF | kubectl apply -f -

apiVersion: networking.istio.io/v1alpha3

kind: VirtualService

metadata:

  name: helloworld-v1

spec:

  hosts:

  - helloworld-v1.example.com

  gateways:

  - mygateway

  http:

  - match:

    - uri:

        exact: /hello

    route:

    - destination:

        host: helloworld-v1

        port:

          number: 5000

EOF

向helloworld-v1.example.com发送HTTPS请求

$ curl -v -HHost:helloworld-v1.example.com --resolve "helloworld-v1.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \

--cacert example.com.crt "https://helloworld-v1.example.com:$SECURE_INGRESS_PORT/hello"

...

< HTTP/1.1 200 OK

< content-type: text/html; charset=utf-8

< content-length: 60

< server: istio-envoy

< date: Sat, 23 May 2020 07:38:40 GMT

< x-envoy-upstream-service-time: 143

<

Hello version: v1, instance: helloworld-v1-5dfcf5d5cd-2l44c

* Connection #0 to host helloworld-v1.example.com left intact

向 httpbin.example.com发送请求

$ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \

--cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

 ...

    -=[ teapot ]=-

       _...._

     .'  _ _ `.

    | ."` ^ `". _,

    \_;`"---"`|//

      |       ;/

      \_     _/

        `"""`

* Connection #0 to host httpbin.example.com left intact

配置一个mutual TLS ingress网关

删除之前的secreting创建一个新的secret，server会使用该CA证书校验client，使用cacert保存CA证书。

$ kubectl -n istio-system delete secret httpbin-credential

$ kubectl create -n istio-system secret generic httpbin-credential --from-file=tls.key=httpbin.example.com.key \

--from-file=tls.crt=httpbin.example.com.crt --from-file=ca.crt=example.com.crt

将gateway的TLS模式设置为MUTUAL

$ cat <<EOF | kubectl apply -f -

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:

 name: mygateway

spec:

 selector:

   istio: ingressgateway # use istio default ingress gateway

 servers:

 - port:

     number: 443

     name: https

     protocol: HTTPS

   tls:

     mode: MUTUAL #TLS模式设置为MUTUAL

     credentialName: httpbin-credential # must be the same as secret

   hosts:

   - httpbin.example.com

EOF

使用先前的方式发送HTTPS请求，可以看到访问失败

$ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" \

> --cacert example.com.crt "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

* Added httpbin.example.com:31967:172.20.127.211 to DNS cache

* About to connect() to httpbin.example.com port 31967 (#0)

*   Trying 172.20.127.211...

* Connected to httpbin.example.com (172.20.127.211) port 31967 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

*   CAfile: example.com.crt

  CApath: none

* NSS: client certificate not found (nickname not specified)

* NSS error -12227 (SSL_ERROR_HANDSHAKE_FAILURE_ALERT)

* SSL peer was unable to negotiate an acceptable set of security parameters.

* Closing connection 0

curl: (35) NSS: client certificate not found (nickname not specified)

生成client的证书和私钥。在curl中传入客户端的证书和私钥，使用--cert传入客户端证书，使用--key传入私钥

$ curl -v -HHost:httpbin.example.com --resolve "httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST" --cacert example.com.crt --cert ./client.example.com.crt --key ./client.example.com.key "https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418"

...

    -=[ teapot ]=-

       _...._

     .'  _ _ `.

    | ."` ^ `". _,

    \_;`"---"`|//

      |       ;/

      \_     _/

        `"""`

* Connection #0 to host httpbin.example.com left intact

istio支持几种不同的Secret格式，来支持与多种工具的集成，如cert-manager:

一个TLS Secret使用tls.key和tls.crt；对于mutual TLS，会用到ca.crt
一个generic Secret会用到key和cert；对于mutual TLS，会用到cacert
一个generic Secret会用到key和cert；对于mutual TLS，会用到一个单独的名为 <secret>-cacert的generic Secret，以及一个cacert key。例如httpbin-credential 包含key 和cert，httpbin-credential-cacert 包含cacert

问题定位

检查INGRESS_HOST和SECURE_INGRESS_PORT环境变量

$ kubectl get svc -n istio-system

$ echo INGRESS_HOST=$INGRESS_HOST, SECURE_INGRESS_PORT=$SECURE_INGRESS_PORT

检查istio-ingressgateway控制器的错误日志

$ kubectl logs -n istio-system "$(kubectl get pod -l istio=ingressgateway \

-n istio-system -o jsonpath='{.items[0].metadata.name}')"

校验istio-system命名空间中成功创建了secret。上例中应该存在httpbin-credential和helloworld-credential
```
$ kubectl -n istio-system get secrets
```

校验ingress网关agent将密钥/证书对上传到了ingress网关

$ kubectl logs -n istio-system "$(kubectl get pod -l istio=ingressgateway \

-n istio-system -o jsonpath='{.items[0].metadata.name}')"

定位mutul TLS问题

校验CA加载到了 istio-ingressgateway pod中，查看是否存在example.com.crt

$ kubectl exec -it -n istio-system $(kubectl -n istio-system get pods -l istio=ingressgateway -o jsonpath='{.items[0].metadata.name}') -- ls -al /etc/istio/ingressgateway-ca-certs

如果创建了istio-ingressgateway-ca-certs secret，但没有加载CA证书，删除ingress网关pod，强制加载该证书
```
$ kubectl delete pod -n istio-system -l istio=ingressgateway
```
校验CA证书的Subject字段是否正确
```
$ kubectl exec -i -n istio-system $(kubectl get pod -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].metadata.name}')  -- cat /etc/istio/ingressgateway-ca-certs/example.com.crt | openssl x509 -text -noout | grep 'Subject:'

        Subject: O=example Inc., CN=example.com
```
log中可以看到添加了httpbin-credential secret。如果使用mutual TLS，那么也会出现 httpbin-credential-cacert secret。校验log中显示了网关agent从ingress网关接收到了SDS请求，资源的名称为 httpbin-credential，且ingress网关获取到了密钥/证书对。如果使用了mutual TLS，日志应该显示将密钥/证书发送到ingress网关，网关agent接收到了带 httpbin-credential-cacert 资源名称的SDS请求，并回去到了根证书。

卸载

删除Gateway配置，VirtualService和secret

$ kubectl delete gateway mygateway

$ kubectl delete virtualservice httpbin

$ kubectl delete --ignore-not-found=true -n istio-system secret httpbin-credential \

helloworld-credential

$ kubectl delete --ignore-not-found=true virtualservice helloworld-v1

删除证书目录

$ rm -rf example.com.crt example.com.key httpbin.example.com.crt httpbin.example.com.key httpbin.example.com.csr helloworld-v1.example.com.crt helloworld-v1.example.com.key helloworld-v1.example.com.csr client.example.com.crt client.example.com.csr client.example.com.key ./new_certificates

停止httpbin和helloworld-v1 服务：

$ kubectl delete deployment --ignore-not-found=true httpbin helloworld-v1

$ kubectl delete service --ignore-not-found=true httpbin helloworld-v1

不终止TLS的ingress网关

上一节中描述了如何配置HTTPS ingree来访问一个HTTP服务。本节中描述如何配置HTTPS ingrss来访问HTTPS服务等。通过配置ingress网关来执行SNI方式的访问，而不会在请求进入ingress时终止TLS。

本例中使用一个NGINX服务器作为HTTPS服务。

生成客户端和服务端的证书和密钥

生成一个根证书和私钥，用于签名服务

$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example.com.key -out example.com.crt

为 nginx.example.com创建证书和私钥

$ openssl req -out nginx.example.com.csr -newkey rsa:2048 -nodes -keyout nginx.example.com.key -subj "/CN=nginx.example.com/O=some organization"

$ openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in nginx.example.com.csr -out nginx.example.com.crt

部署NGINX服务

创建kubernetes Secret保存服务的证书

$ kubectl create secret tls nginx-server-certs --key nginx.example.com.key --cert nginx.example.com.crt

为NGINX服务创建配置文件

$ cat <<EOF > ./nginx.conf

events {

}

http {

  log_format main '$remote_addr - $remote_user [$time_local]  $status '

  '"$request" $body_bytes_sent "$http_referer" '

  '"$http_user_agent" "$http_x_forwarded_for"';

  access_log /var/log/nginx/access.log main;

  error_log  /var/log/nginx/error.log;

  server {

    listen 443 ssl;

    root /usr/share/nginx/html;

    index index.html;

    server_name nginx.example.com;

    ssl_certificate /etc/nginx-server-certs/tls.crt;

    ssl_certificate_key /etc/nginx-server-certs/tls.key;

  }

}

EOF

为NGINX服务创建kubernetes configmap

$ kubectl create configmap nginx-configmap --from-file=nginx.conf=./nginx.conf

部署NGINX服务

$ cat <<EOF | istioctl kube-inject -f - | kubectl apply -f -

apiVersion: v1

kind: Service

metadata:

  name: my-nginx

  labels:

    run: my-nginx

spec:

  ports:

  - port: 443

    protocol: TCP

  selector:

    run: my-nginx

---

apiVersion: apps/v1

kind: Deployment

metadata:

  name: my-nginx

spec:

  selector:

    matchLabels:

      run: my-nginx

  replicas: 1

  template:

    metadata:

      labels:

        run: my-nginx

    spec:

      containers:

      - name: my-nginx

        image: nginx

        ports:

        - containerPort: 443

        volumeMounts:

        - name: nginx-config

          mountPath: /etc/nginx

          readOnly: true

        - name: nginx-server-certs

          mountPath: /etc/nginx-server-certs

          readOnly: true

      volumes:

      - name: nginx-config

        configMap:

          name: nginx-configmap

      - name: nginx-server-certs

        secret:

          secretName: nginx-server-certs #保存了NGINX服务的证书和私钥

EOF

为了测试NGINX服务部署成功，向服务发送不使用证书的方式请求，并校验打印信息是否正确：

$ kubectl exec -it $(kubectl get pod  -l run=my-nginx -o jsonpath={.items..metadata.name}) -c istio-proxy -- curl -v -k --resolve

...

* Server certificate:

*  subject: CN=nginx.example.com; O=some organization

*  start date: May 25 02:09:02 2020 GMT

*  expire date: May 25 02:09:02 2021 GMT

*  issuer: O=example Inc.; CN=example.com

*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

...

配置一个ingress gateway

定义一个gateway，端口为443.注意TLS的模式为PASSTHROUGH，表示gateway会放行ingress流量，不会终止TLS

$ kubectl apply -f - <<EOF

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:

  name: mygateway

spec:

  selector:

    istio: ingressgateway # use istio default ingress gateway

  servers:

  - port:

      number: 443

      name: https

      protocol: HTTPS

    tls:

      mode: PASSTHROUGH #不终止TLS

    hosts:

    - nginx.example.com

EOF

配置经过Gateway的流量路由

$ kubectl apply -f - <<EOF

apiVersion: networking.istio.io/v1alpha3

kind: VirtualService

metadata:

  name: nginx

spec:

  hosts:

  - nginx.example.com

  gateways:

  - mygateway

  tls:

  - match:

    - port: 443 #将gateway的流量导入kubernetes的my-nginx service

      sniHosts:

      - nginx.example.com

    route:

    - destination:

        host: my-nginx

        port:

          number: 443

EOF

根据指导配置SECURE_INGRESS_PORT和INGRESS_HOST环境变量

通过ingress访问nginx，可以看到访问成功

$ curl -v --resolve nginx.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST --cacert example.com.crt https://nginx.example.com:$SECURE_INGRESS_PORT

...

* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

* Server certificate:

*       subject: O=some organization,CN=nginx.example.com

*       start date: May 25 02:09:02 2020 GMT

*       expire date: May 25 02:09:02 2021 GMT

*       common name: nginx.example.com

*       issuer: CN=example.com,O=example Inc.

...

<title>Welcome to nginx!</title>

...

卸载

移除kubernetes资源

$ kubectl delete secret nginx-server-certs

$ kubectl delete configmap nginx-configmap

$ kubectl delete service my-nginx

$ kubectl delete deployment my-nginx

$ kubectl delete gateway mygateway

$ kubectl delete virtualservice nginx

删除证书和密钥

$ rm example.com.crt example.com.key nginx.example.com.crt nginx.example.com.key nginx.example.com.csr

删除生成的配置文件
```
$ rm ./nginx.conf
```

码农公寓

Istio的流量管理(实操二)(istio 系列四)

Ingress网关

环境准备

确定ingress的IP和端口

使用istio网关配置ingress

使用浏览器访问ingress服务

问题定位

卸载

Ingress(kubernetes)

下一步

TLS

指定路径类型

指定IngressClass

卸载

安全网关

生成服务端证书和私钥

单主机配置TLS ingress网关

多主机配置TLS ingress网关

配置一个mutual TLS ingress网关

问题定位

定位mutul TLS问题

卸载

不终止TLS的ingress网关

生成客户端和服务端的证书和密钥

部署NGINX服务

配置一个ingress gateway

卸载

相关文章

指定`IngressClass`