Cilium路由模式(一)

Encapsulation 封装

官方文档

缺省模式,Cilium 会自动运行在封装模式下,因为这种模式对底层网络基础设施(特别是云厂商)依赖性比较低,只要2层网络通城,执行封装技术即可达到Pod通信目的

在这种模式下,所有集群节点使用基于 UDP 的封装协议 VXLAN 或 Geneve 形成隧道网状结构。 Cilium 节点之间的所有流量都被封装

二种封装技术

  1. VXLAN
  2. Geneve

如果你的网络有设置防火墙,注意了,这几个端口

Encapsulation Mode Port Range / Protocol
VXLAN (Default) 8472/UDP
Geneve 6081/UDP

封装优势

  • 简单  连接集群节点的网络不需要知道 PodCIDR。 集群节点可以产生多个路由或链路层域。 只要集群节点可以使用 IP/UDP 相互访问,底层网络的拓扑结构就无关紧要
  • 寻址空间 由于不依赖于任何底层网络限制,如果 PodCIDR 大小相应地配置,可用寻址空间可能要大得多,并且允许每个节点运行任意数量的 Pod
  • 自动配置 当与编排系统(例如 Kubernetes)一起运行时,集群中所有节点的列表(包括其关联的分配前缀节点)将自动提供给每个cilium-agent, 加入集群的新节点将自动合并到网格中
  • identity context 封装协议允许将元数据与网络数据包一起携带。 Cilium 利用这种能力来传输元数据,例如源安全身份。 身份传输是一种优化,旨在避免在远程节点上进行一次身份查找

封装劣势 

  • MTU Overhead 由于添加了封装header,MTU太大引起的弊端低于本机路由(VXLAN 的每个网络数据包 50 字节)。 这导致特定网络连接影响低吞吐量。 当然可以通过启用巨型帧(每 1500 字节有 50 字节的开销,而每 9000 字节有 50 字节的开销)来很大程度上缓解。

配置过程

  1. 配置过程如下
    helm install cilium cilium/cilium --version 1.9.9     --namespace kube-system     --set tunnel=vxlan     --set kubeProxyReplacement=strict     --set ipam.mode=kubernetes     --set ipam.operator.clusterPoolIPv4PodCIDR=172.21.0.0/20     --set ipam.operator.clusterPoolIPv4MaskSize=26     --set k8sServiceHost=apiserver.qiangyun.com     --set k8sServicePort=6443

     

  2. 观察节点的路由,如下
    <root@PROD-FE-K8S-WN1 ~># netstat -rn
    Kernel IP routing table
    Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
    0.0.0.0         10.1.16.253     0.0.0.0         UG        0 0          0 eth0
    10.1.16.0       0.0.0.0         255.255.255.0   U         0 0          0 eth0
    169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
    172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
    172.21.0.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.1.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.2.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.3.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.4.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.5.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.6.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.7.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.8.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.9.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.9.225    0.0.0.0         255.255.255.255 UH        0 0          0 cilium_host
    172.21.10.0     172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.11.0     172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.12.0     172.21.9.225    255.255.255.192 UG        0 0          0 cilium_host
    172.21.12.64    172.21.9.225    255.255.255.192 UG        0 0          0 cilium_host
    172.21.12.128   172.21.9.225    255.255.255.192 UG        0 0          0 cilium_host
    172.21.12.192   172.21.9.225    255.255.255.192 UG        0 0          0 cilium_host
    172.21.13.0     172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.14.0     172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
    172.21.15.0     172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host

    # 说明
    每个节点都有一个CiliumInternal地址 172.21.9.225
    每个节点都有一个IPAM
    每个节点都有一个healthcheck的地址
    以下地址在下方体现
  3. CRD CiliumNode
    spec:
      addresses:
        - ip: 10.1.16.221
          type: InternalIP
        - ip: 172.21.9.225
          type: CiliumInternalIP
      azure: {}
      encryption: {}
      eni: {}
      health:
        ipv4: 172.21.9.190
      ipam:
        podCIDRs:
          - 172.21.9.0/24
  4. 简单说下,通信原理
    <root@PROD-FE-K8S-WN1 ~># ifconfig 
    cilium_host: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST>  mtu 1500
            inet 172.21.9.225  netmask 255.255.255.255  broadcast 0.0.0.0
            ether 22:cb:9d:23:d8:48  txqueuelen 1000  (Ethernet)
            RX packets 4665  bytes 356292 (347.9 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 273  bytes 19841 (19.3 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    cilium_net: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST>  mtu 1500
            ether 26:7f:1f:99:b5:db  txqueuelen 1000  (Ethernet)
            RX packets 273  bytes 19841 (19.3 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 4665  bytes 356292 (347.9 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    cilium_vxlan: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
            ether 02:16:be:c2:2f:2f  txqueuelen 1000  (Ethernet)
            RX packets 10023  bytes 634132 (619.2 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 9979  bytes 629067 (614.3 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    # 说明
    cilium_host 类似于一个路由器,网关设备
    cilium_net & cilium_host成对出现,像是一个veth,一端接容器一端接主机
    cilium_vxlan 就是虚拟二层网络,用来提供Pod跨节点通信MTU封装

     

  5. 虽然cilium默认使用的封装技术,但是host routing模式还是使用到BPF模式,如下
    root@PROD-FE-K8S-WN1:/home/cilium# cilium status --verbose
    KVStore:                Ok   Disabled
    Kubernetes:             Ok   1.18 (v1.18.5) [linux/amd64]
    Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
    KubeProxyReplacement:   Strict   [eth0 (Direct Routing)]
    Cilium:                 Ok   1.9.9 (v1.9.9-5bcf83c)
    NodeMonitor:            Listening for events on 2 CPUs with 64x4096 of shared memory
    Cilium health daemon:   Ok   
    IPAM:                   IPv4: 2/255 allocated from 172.21.9.0/24, 
    Allocated addresses:
      172.21.9.225 (router)
      172.21.9.26 (health)
    BandwidthManager:       Disabled
    Host Routing:           BPF
    Masquerading:           BPF   [eth0]   172.21.9.0/24
    Clock Source for BPF:   ktime
    Controller Status:      18/18 healthy
      Name                                  Last success   Last error   Count   Message
      cilium-health-ep                      33s ago        never        0       no error   
      dns-garbage-collector-job             41s ago        never        0       no error   
      endpoint-2159-regeneration-recovery   never          never        0       no error   
      endpoint-3199-regeneration-recovery   never          never        0       no error   
      k8s-heartbeat                         11s ago        never        0       no error   
      mark-k8s-node-as-available            48m34s ago     never        0       no error   
      metricsmap-bpf-prom-sync              6s ago         never        0       no error   
      neighbor-table-refresh                3m34s ago      never        0       no error   
      resolve-identity-2159                 3m34s ago      never        0       no error   
      resolve-identity-3199                 3m33s ago      never        0       no error   
      sync-endpoints-and-host-ips           34s ago        never        0       no error   
      sync-lb-maps-with-k8s-services        48m34s ago     never        0       no error   
      sync-policymap-2159                   31s ago        never        0       no error   
      sync-policymap-3199                   31s ago        never        0       no error   
      sync-to-k8s-ciliumendpoint (2159)     4s ago         never        0       no error   
      sync-to-k8s-ciliumendpoint (3199)     13s ago        never        0       no error   
      template-dir-watcher                  never          never        0       no error   
      update-k8s-node-annotations           48m40s ago     never        0       no error   
    Proxy Status:   OK, ip 172.21.9.225, 0 redirects active on ports 10000-20000
    Hubble:         Ok   Current/Max Flows: 4096/4096 (100.00%), Flows/s: 5.39   Metrics: Disabled
    KubeProxyReplacement Details:
      Status:              Strict
      Protocols:           TCP, UDP
      Devices:             eth0 (Direct Routing)
      Mode:                SNAT  (kuber-proxy的模式,默认就是SNAT)
      Backend Selection:   Random
      Session Affinity:    Enabled
      XDP Acceleration:    Disabled
      Services:
      - ClusterIP:      Enabled
      - NodePort:       Enabled (Range: 30000-32767) 
      - LoadBalancer:   Enabled 
      - externalIPs:    Enabled 
      - HostPort:       Enabled
    BPF Maps:   dynamic sizing: on (ratio: 0.002500)
      Name                          Size
      Non-TCP connection tracking   65536
      TCP connection tracking       131072
      Endpoint policy               65535
      Events                        2
      IP cache                      512000
      IP masquerading agent         16384
      IPv4 fragmentation            8192
      IPv4 service                  65536
      IPv6 service                  65536
      IPv4 service backend          65536
      IPv6 service backend          65536
      IPv4 service reverse NAT      65536
      IPv6 service reverse NAT      65536
      Metrics                       1024
      NAT                           131072
      Neighbor table                131072
      Global policy                 16384
      Per endpoint policy           65536
      Session affinity              65536
      Signal                        2
      Sockmap                       65535
      Sock reverse NAT              65536
      Tunnel                        65536
    Cluster health:                 19/19 reachable   (2021-08-27T17:54:39Z)
      Name                          IP                Node        Endpoints
      prod-fe-k8s-wn1 (localhost)   10.1.16.221       reachable   reachable
      prod-be-k8s-wn1               10.1.17.231       reachable   reachable
      prod-be-k8s-wn2               10.1.17.232       reachable   reachable
      prod-be-k8s-wn6               10.1.17.236       reachable   reachable
      prod-be-k8s-wn7               10.1.17.237       reachable   reachable
      prod-be-k8s-wn8               10.1.17.238       reachable   reachable
      prod-data-k8s-wn1             10.1.18.50        reachable   reachable
      prod-data-k8s-wn2             10.1.18.49        reachable   reachable
      prod-data-k8s-wn3             10.1.18.51        reachable   reachable
      prod-fe-k8s-wn2               10.1.16.222       reachable   reachable
      prod-fe-k8s-wn3               10.1.16.223       reachable   reachable
      prod-k8s-cp1                  10.1.0.5          reachable   reachable
      prod-k8s-cp2                  10.1.0.7          reachable   reachable
      prod-k8s-cp3                  10.1.0.6          reachable   reachable
      prod-sys-k8s-wn1              10.1.0.8          reachable   reachable
      prod-sys-k8s-wn2              10.1.0.9          reachable   reachable
      prod-sys-k8s-wn3              10.1.0.11         reachable   reachable
      prod-sys-k8s-wn4              10.1.0.10         reachable   reachable
      prod-sys-k8s-wn5              10.1.0.12         reachable   reachable
  6. cilium-agent 具体启动输出日志
    <root@PROD-FE-K8S-WN1 ~># dps
    6fdce5c6148b    Up 51 minutes   k8s_pord-ingress_prod-ingress-b76597794-tmrtc_ingress-nginx_a63d92fe-5c99-4948-89ca-fd70d2298f99_3
    43686a967be8    Up 52 minutes   k8s_cilium-agent_cilium-cgrdw_kube-system_14e0fb48-cc56-46d8-b929-64f66b36c6b7_2
    <root@PROD-FE-K8S-WN1 ~># dok^C
    <root@PROD-FE-K8S-WN1 ~># docker logs -f 43686a967be8
    level=info msg="Skipped reading configuration file" reason="Config File \"ciliumd\" Not Found in \"[/root]\"" subsys=config
    level=info msg="Started gops server" address="127.0.0.1:9890" subsys=daemon
    level=info msg="Memory available for map entries (0.003% of 3976814592B): 9942036B" subsys=config
    level=info msg="option bpf-ct-global-tcp-max set by dynamic sizing to 131072" subsys=config
    level=info msg="option bpf-ct-global-any-max set by dynamic sizing to 65536" subsys=config
    level=info msg="option bpf-nat-global-max set by dynamic sizing to 131072" subsys=config
    level=info msg="option bpf-neigh-global-max set by dynamic sizing to 131072" subsys=config
    level=info msg="option bpf-sock-rev-map-max set by dynamic sizing to 65536" subsys=config
    level=info msg="  --agent-health-port=‘9876‘" subsys=daemon
    level=info msg="  --agent-labels=‘‘" subsys=daemon
    level=info msg="  --allow-icmp-frag-needed=‘true‘" subsys=daemon
    level=info msg="  --allow-localhost=‘auto‘" subsys=daemon
    level=info msg="  --annotate-k8s-node=‘true‘" subsys=daemon
    level=info msg="  --api-rate-limit=‘map[]‘" subsys=daemon
    level=info msg="  --arping-refresh-period=‘5m0s‘" subsys=daemon
    level=info msg="  --auto-create-cilium-node-resource=‘true‘" subsys=daemon
    level=info msg="  --auto-direct-node-routes=‘false‘" subsys=daemon  DSR 模式关闭状态(因为只能运行于Native-Routing)
    level=info msg="  --blacklist-conflicting-routes=‘false‘" subsys=daemon
    level=info msg="  --bpf-compile-debug=‘false‘" subsys=daemon
    level=info msg="  --bpf-ct-global-any-max=‘262144‘" subsys=daemon
    level=info msg="  --bpf-ct-global-tcp-max=‘524288‘" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-any=‘1m0s‘" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-tcp=‘6h0m0s‘" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-tcp-fin=‘10s‘" subsys=daemon
    level=info msg="  --bpf-ct-timeout-regular-tcp-syn=‘1m0s‘" subsys=daemon
    level=info msg="  --bpf-ct-timeout-service-any=‘1m0s‘" subsys=daemon
    level=info msg="  --bpf-ct-timeout-service-tcp=‘6h0m0s‘" subsys=daemon
    level=info msg="  --bpf-fragments-map-max=‘8192‘" subsys=daemon
    level=info msg="  --bpf-lb-acceleration=‘disabled‘" subsys=daemon
    level=info msg="  --bpf-lb-algorithm=‘random‘" subsys=daemon
    level=info msg="  --bpf-lb-maglev-hash-seed=‘JLfvgnHc2kaSUFaI‘" subsys=daemon
    level=info msg="  --bpf-lb-maglev-table-size=‘16381‘" subsys=daemon
    level=info msg="  --bpf-lb-map-max=‘65536‘" subsys=daemon
    level=info msg="  --bpf-lb-mode=‘snat‘" subsys=daemon
    level=info msg="  --bpf-map-dynamic-size-ratio=‘0.0025‘" subsys=daemon
    level=info msg="  --bpf-nat-global-max=‘524288‘" subsys=daemon
    level=info msg="  --bpf-neigh-global-max=‘524288‘" subsys=daemon
    level=info msg="  --bpf-policy-map-max=‘16384‘" subsys=daemon
    level=info msg="  --bpf-root=‘‘" subsys=daemon
    level=info msg="  --bpf-sock-rev-map-max=‘262144‘" subsys=daemon
    level=info msg="  --certificates-directory=‘/var/run/cilium/certs‘" subsys=daemon
    level=info msg="  --cgroup-root=‘/run/cilium/cgroupv2‘" subsys=daemon
    level=info msg="  --cluster-id=‘‘" subsys=daemon
    level=info msg="  --cluster-name=‘default‘" subsys=daemon
    level=info msg="  --clustermesh-config=‘/var/lib/cilium/clustermesh/‘" subsys=daemon
    level=info msg="  --cmdref=‘‘" subsys=daemon
    level=info msg="  --config=‘‘" subsys=daemon
    level=info msg="  --config-dir=‘/tmp/cilium/config-map‘" subsys=daemon
    level=info msg="  --conntrack-gc-interval=‘0s‘" subsys=daemon
    level=info msg="  --crd-wait-timeout=‘5m0s‘" subsys=daemon
    level=info msg="  --datapath-mode=‘veth‘" subsys=daemon
    level=info msg="  --debug=‘false‘" subsys=daemon
    level=info msg="  --debug-verbose=‘‘" subsys=daemon
    level=info msg="  --device=‘‘" subsys=daemon
    level=info msg="  --devices=‘‘" subsys=daemon
    level=info msg="  --direct-routing-device=‘‘" subsys=daemon
    level=info msg="  --disable-cnp-status-updates=‘true‘" subsys=daemon
    level=info msg="  --disable-conntrack=‘false‘" subsys=daemon
    level=info msg="  --disable-endpoint-crd=‘false‘" subsys=daemon
    level=info msg="  --disable-envoy-version-check=‘false‘" subsys=daemon
    level=info msg="  --disable-iptables-feeder-rules=‘‘" subsys=daemon
    level=info msg="  --dns-max-ips-per-restored-rule=‘1000‘" subsys=daemon
    level=info msg="  --egress-masquerade-interfaces=‘‘" subsys=daemon 伪装模式的一种(基于iptables-base),还有一种是基于eBPF-base,二者并存优先取eBPF-base
    level=info msg="  --egress-multi-home-ip-rule-compat=‘false‘" subsys=daemon
    level=info msg="  --enable-auto-protect-node-port-range=‘true‘" subsys=daemon
    level=info msg="  --enable-bandwidth-manager=‘false‘" subsys=daemon
    level=info msg="  --enable-bpf-clock-probe=‘true‘" subsys=daemon
    level=info msg="  --enable-bpf-masquerade=‘true‘" subsys=daemon
    level=info msg="  --enable-bpf-tproxy=‘false‘" subsys=daemon
    level=info msg="  --enable-endpoint-health-checking=‘true‘" subsys=daemon
    level=info msg="  --enable-endpoint-routes=‘false‘" subsys=daemon  # Native-Routing的另一种路由模式,后面会有介绍,与前者不能并存,否则会报错
    level=info msg="  --enable-external-ips=‘true‘" subsys=daemon
    level=info msg="  --enable-health-check-nodeport=‘true‘" subsys=daemon
    level=info msg="  --enable-health-checking=‘true‘" subsys=daemon
    level=info msg="  --enable-host-firewall=‘false‘" subsys=daemon
    level=info msg="  --enable-host-legacy-routing=‘false‘" subsys=daemon
    level=info msg="  --enable-host-port=‘true‘" subsys=daemon
    level=info msg="  --enable-host-reachable-services=‘false‘" subsys=daemon
    level=info msg="  --enable-hubble=‘true‘" subsys=daemon
    level=info msg="  --enable-identity-mark=‘true‘" subsys=daemon
    level=info msg="  --enable-ip-masq-agent=‘false‘" subsys=daemon
    level=info msg="  --enable-ipsec=‘false‘" subsys=daemon
    level=info msg="  --enable-ipv4=‘true‘" subsys=daemon
    level=info msg="  --enable-ipv4-fragment-tracking=‘true‘" subsys=daemon
    level=info msg="  --enable-ipv6=‘false‘" subsys=daemon
    level=info msg="  --enable-ipv6-ndp=‘false‘" subsys=daemon
    level=info msg="  --enable-k8s-api-discovery=‘false‘" subsys=daemon
    level=info msg="  --enable-k8s-endpoint-slice=‘true‘" subsys=daemon
    level=info msg="  --enable-k8s-event-handover=‘false‘" subsys=daemon
    level=info msg="  --enable-l7-proxy=‘true‘" subsys=daemon
    level=info msg="  --enable-local-node-route=‘true‘" subsys=daemon
    level=info msg="  --enable-local-redirect-policy=‘false‘" subsys=daemon
    level=info msg="  --enable-monitor=‘true‘" subsys=daemon
    level=info msg="  --enable-node-port=‘false‘" subsys=daemon
    level=info msg="  --enable-policy=‘default‘" subsys=daemon
    level=info msg="  --enable-remote-node-identity=‘true‘" subsys=daemon
    level=info msg="  --enable-selective-regeneration=‘true‘" subsys=daemon
    level=info msg="  --enable-session-affinity=‘true‘" subsys=daemon
    level=info msg="  --enable-svc-source-range-check=‘true‘" subsys=daemon
    level=info msg="  --enable-tracing=‘false‘" subsys=daemon
    level=info msg="  --enable-well-known-identities=‘false‘" subsys=daemon
    level=info msg="  --enable-xt-socket-fallback=‘true‘" subsys=daemon
    level=info msg="  --encrypt-interface=‘‘" subsys=daemon
    level=info msg="  --encrypt-node=‘false‘" subsys=daemon
    level=info msg="  --endpoint-interface-name-prefix=‘lxc+‘" subsys=daemon
    level=info msg="  --endpoint-queue-size=‘25‘" subsys=daemon
    level=info msg="  --endpoint-status=‘‘" subsys=daemon
    level=info msg="  --envoy-log=‘‘" subsys=daemon
    level=info msg="  --exclude-local-address=‘‘" subsys=daemon
    level=info msg="  --fixed-identity-mapping=‘map[]‘" subsys=daemon
    level=info msg="  --flannel-master-device=‘‘" subsys=daemon
    level=info msg="  --flannel-uninstall-on-exit=‘false‘" subsys=daemon
    level=info msg="  --force-local-policy-eval-at-source=‘true‘" subsys=daemon
    level=info msg="  --gops-port=‘9890‘" subsys=daemon
    level=info msg="  --host-reachable-services-protos=‘tcp,udp‘" subsys=daemon
    level=info msg="  --http-403-msg=‘‘" subsys=daemon
    level=info msg="  --http-idle-timeout=‘0‘" subsys=daemon
    level=info msg="  --http-max-grpc-timeout=‘0‘" subsys=daemon
    level=info msg="  --http-normalize-path=‘true‘" subsys=daemon
    level=info msg="  --http-request-timeout=‘3600‘" subsys=daemon
    level=info msg="  --http-retry-count=‘3‘" subsys=daemon
    level=info msg="  --http-retry-timeout=‘0‘" subsys=daemon
    level=info msg="  --hubble-disable-tls=‘false‘" subsys=daemon
    level=info msg="  --hubble-event-queue-size=‘0‘" subsys=daemon
    level=info msg="  --hubble-flow-buffer-size=‘4095‘" subsys=daemon
    level=info msg="  --hubble-listen-address=‘:4244‘" subsys=daemon
    level=info msg="  --hubble-metrics=‘‘" subsys=daemon
    level=info msg="  --hubble-metrics-server=‘‘" subsys=daemon
    level=info msg="  --hubble-socket-path=‘/var/run/cilium/hubble.sock‘" subsys=daemon
    level=info msg="  --hubble-tls-cert-file=‘/var/lib/cilium/tls/hubble/server.crt‘" subsys=daemon
    level=info msg="  --hubble-tls-client-ca-files=‘/var/lib/cilium/tls/hubble/client-ca.crt‘" subsys=daemon
    level=info msg="  --hubble-tls-key-file=‘/var/lib/cilium/tls/hubble/server.key‘" subsys=daemon
    level=info msg="  --identity-allocation-mode=‘crd‘" subsys=daemon
    level=info msg="  --identity-change-grace-period=‘5s‘" subsys=daemon
    level=info msg="  --install-iptables-rules=‘true‘" subsys=daemon
    level=info msg="  --ip-allocation-timeout=‘2m0s‘" subsys=daemon
    level=info msg="  --ip-masq-agent-config-path=‘/etc/config/ip-masq-agent‘" subsys=daemon
    level=info msg="  --ipam=‘kubernetes‘" subsys=daemon CRD的模式,有好几种,官方默认是cluster-pool,kubernetes模式代表从node到拿IP地址,基于controller-manager 启动参数--allocate-node-cidrs
    level=info msg="  --ipsec-key-file=‘‘" subsys=daemon
    level=info msg="  --iptables-lock-timeout=‘5s‘" subsys=daemon
    level=info msg="  --iptables-random-fully=‘false‘" subsys=daemon
    level=info msg="  --ipv4-node=‘auto‘" subsys=daemon
    level=info msg="  --ipv4-pod-subnets=‘‘" subsys=daemon
    level=info msg="  --ipv4-range=‘auto‘" subsys=daemon
    level=info msg="  --ipv4-service-loopback-address=‘169.254.42.1‘" subsys=daemon
    level=info msg="  --ipv4-service-range=‘auto‘" subsys=daemon
    level=info msg="  --ipv6-cluster-alloc-cidr=‘f00d::/64‘" subsys=daemon
    level=info msg="  --ipv6-mcast-device=‘‘" subsys=daemon
    level=info msg="  --ipv6-node=‘auto‘" subsys=daemon
    level=info msg="  --ipv6-pod-subnets=‘‘" subsys=daemon
    level=info msg="  --ipv6-range=‘auto‘" subsys=daemon
    level=info msg="  --ipv6-service-range=‘auto‘" subsys=daemon
    level=info msg="  --ipvlan-master-device=‘undefined‘" subsys=daemon
    level=info msg="  --join-cluster=‘false‘" subsys=daemon
    level=info msg="  --k8s-api-server=‘‘" subsys=daemon
    level=info msg="  --k8s-force-json-patch=‘false‘" subsys=daemon
    level=info msg="  --k8s-heartbeat-timeout=‘30s‘" subsys=daemon
    level=info msg="  --k8s-kubeconfig-path=‘‘" subsys=daemon
    level=info msg="  --k8s-namespace=‘kube-system‘" subsys=daemon
    level=info msg="  --k8s-require-ipv4-pod-cidr=‘false‘" subsys=daemon
    level=info msg="  --k8s-require-ipv6-pod-cidr=‘false‘" subsys=daemon
    level=info msg="  --k8s-service-cache-size=‘128‘" subsys=daemon
    level=info msg="  --k8s-service-proxy-name=‘‘" subsys=daemon
    level=info msg="  --k8s-sync-timeout=‘3m0s‘" subsys=daemon
    level=info msg="  --k8s-watcher-endpoint-selector=‘metadata.name!=kube-scheduler,metadata.name!=kube-controller-manager,metadata.name!=etcd-operator,metadata.name!=gcp-controller-manager‘" subsys=daemon
    level=info msg="  --k8s-watcher-queue-size=‘1024‘" subsys=daemon
    level=info msg="  --keep-config=‘false‘" subsys=daemon
    level=info msg="  --kube-proxy-replacement=‘strict‘" subsys=daemon
    level=info msg="  --kube-proxy-replacement-healthz-bind-address=‘‘" subsys=daemon
    level=info msg="  --kvstore=‘‘" subsys=daemon
    level=info msg="  --kvstore-connectivity-timeout=‘2m0s‘" subsys=daemon
    level=info msg="  --kvstore-lease-ttl=‘15m0s‘" subsys=daemon
    level=info msg="  --kvstore-opt=‘map[]‘" subsys=daemon
    level=info msg="  --kvstore-periodic-sync=‘5m0s‘" subsys=daemon
    level=info msg="  --label-prefix-file=‘‘" subsys=daemon
    level=info msg="  --labels=‘‘" subsys=daemon
    level=info msg="  --lib-dir=‘/var/lib/cilium‘" subsys=daemon
    level=info msg="  --log-driver=‘‘" subsys=daemon
    level=info msg="  --log-opt=‘map[]‘" subsys=daemon
    level=info msg="  --log-system-load=‘false‘" subsys=daemon
    level=info msg="  --masquerade=‘true‘" subsys=daemon
    level=info msg="  --max-controller-interval=‘0‘" subsys=daemon
    level=info msg="  --metrics=‘‘" subsys=daemon
    level=info msg="  --monitor-aggregation=‘medium‘" subsys=daemon
    level=info msg="  --monitor-aggregation-flags=‘all‘" subsys=daemon
    level=info msg="  --monitor-aggregation-interval=‘5s‘" subsys=daemon
    level=info msg="  --monitor-queue-size=‘0‘" subsys=daemon
    level=info msg="  --mtu=‘0‘" subsys=daemon
    level=info msg="  --nat46-range=‘0:0:0:0:0:FFFF::/96‘" subsys=daemon
    level=info msg="  --native-routing-cidr=‘‘" subsys=daemon
    level=info msg="  --node-port-acceleration=‘disabled‘" subsys=daemon
    level=info msg="  --node-port-algorithm=‘random‘" subsys=daemon
    level=info msg="  --node-port-bind-protection=‘true‘" subsys=daemon
    level=info msg="  --node-port-mode=‘snat‘" subsys=daemon
    level=info msg="  --node-port-range=‘30000,32767‘" subsys=daemon
    level=info msg="  --policy-audit-mode=‘false‘" subsys=daemon
    level=info msg="  --policy-queue-size=‘100‘" subsys=daemon
    level=info msg="  --policy-trigger-interval=‘1s‘" subsys=daemon
    level=info msg="  --pprof=‘false‘" subsys=daemon
    level=info msg="  --preallocate-bpf-maps=‘false‘" subsys=daemon
    level=info msg="  --prefilter-device=‘undefined‘" subsys=daemon
    level=info msg="  --prefilter-mode=‘native‘" subsys=daemon
    level=info msg="  --prepend-iptables-chains=‘true‘" subsys=daemon
    level=info msg="  --prometheus-serve-addr=‘‘" subsys=daemon
    level=info msg="  --proxy-connect-timeout=‘1‘" subsys=daemon
    level=info msg="  --proxy-prometheus-port=‘0‘" subsys=daemon
    level=info msg="  --read-cni-conf=‘‘" subsys=daemon
    level=info msg="  --restore=‘true‘" subsys=daemon
    level=info msg="  --sidecar-istio-proxy-image=‘cilium/istio_proxy‘" subsys=daemon
    level=info msg="  --single-cluster-route=‘false‘" subsys=daemon
    level=info msg="  --skip-crd-creation=‘false‘" subsys=daemon
    level=info msg="  --socket-path=‘/var/run/cilium/cilium.sock‘" subsys=daemon
    level=info msg="  --sockops-enable=‘false‘" subsys=daemon
    level=info msg="  --state-dir=‘/var/run/cilium‘" subsys=daemon
    level=info msg="  --tofqdns-dns-reject-response-code=‘refused‘" subsys=daemon
    level=info msg="  --tofqdns-enable-dns-compression=‘true‘" subsys=daemon
    level=info msg="  --tofqdns-endpoint-max-ip-per-hostname=‘50‘" subsys=daemon
    level=info msg="  --tofqdns-idle-connection-grace-period=‘0s‘" subsys=daemon
    level=info msg="  --tofqdns-max-deferred-connection-deletes=‘10000‘" subsys=daemon
    level=info msg="  --tofqdns-min-ttl=‘0‘" subsys=daemon
    level=info msg="  --tofqdns-pre-cache=‘‘" subsys=daemon
    level=info msg="  --tofqdns-proxy-port=‘0‘" subsys=daemon
    level=info msg="  --tofqdns-proxy-response-max-delay=‘100ms‘" subsys=daemon
    level=info msg="  --trace-payloadlen=‘128‘" subsys=daemon
    level=info msg="  --tunnel=‘vxlan‘" subsys=daemon
    level=info msg="  --version=‘false‘" subsys=daemon
    level=info msg="  --write-cni-conf-when-ready=‘‘" subsys=daemon
    level=info msg="     _ _ _" subsys=daemon
    level=info msg=" ___|_| |_|_ _ _____" subsys=daemon
    level=info msg="|  _| | | | | |     |" subsys=daemon
    level=info msg="|___|_|_|_|___|_|_|_|" subsys=daemon
    level=info msg="Cilium 1.9.9 5bcf83c 2021-07-19T16:45:00-07:00 go version go1.15.14 linux/amd64" subsys=daemon
    level=info msg="cilium-envoy  version: 82a70d56bf324287ced3129300db609eceb21d10/1.17.3/Distribution/RELEASE/BoringSSL" subsys=daemon
    level=info msg="clang (10.0.0) and kernel (5.11.1) versions: OK!" subsys=linux-datapath
    level=info msg="linking environment: OK!" subsys=linux-datapath
    level=info msg="Detected mounted BPF filesystem at /sys/fs/bpf" subsys=bpf
    level=info msg="Mounted cgroupv2 filesystem at /run/cilium/cgroupv2" subsys=cgroups
    level=info msg="Parsing base label prefixes from default label list" subsys=labels-filter
    level=info msg="Parsing additional label prefixes from user inputs: []" subsys=labels-filter
    level=info msg="Final label prefixes to be used for identity evaluation:" subsys=labels-filter
    level=info msg=" - reserved:.*" subsys=labels-filter
    level=info msg=" - :io.kubernetes.pod.namespace" subsys=labels-filter
    level=info msg=" - :io.cilium.k8s.namespace.labels" subsys=labels-filter
    level=info msg=" - :app.kubernetes.io" subsys=labels-filter
    level=info msg=" - !:io.kubernetes" subsys=labels-filter
    level=info msg=" - !:kubernetes.io" subsys=labels-filter
    level=info msg=" - !:.*beta.kubernetes.io" subsys=labels-filter
    level=info msg=" - !:k8s.io" subsys=labels-filter
    level=info msg=" - !:pod-template-generation" subsys=labels-filter
    level=info msg=" - !:pod-template-hash" subsys=labels-filter
    level=info msg=" - !:controller-revision-hash" subsys=labels-filter
    level=info msg=" - !:annotation.*" subsys=labels-filter
    level=info msg=" - !:etcd_node" subsys=labels-filter
    level=info msg="Auto-disabling \"enable-bpf-clock-probe\" feature since KERNEL_HZ cannot be determined" error="Cannot probe CONFIG_HZ" subsys=daemon
    level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=10.221.0.0/16
    level=info msg="Initializing daemon" subsys=daemon
    level=info msg="Establishing connection to apiserver" host="https://apiserver.qiangyun.com:6443" subsys=k8s
    level=info msg="Connected to apiserver" subsys=k8s
    level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"enable-host-reachable-services\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
    level=info msg="Inheriting MTU from external network interface" device=eth0 ipAddr=10.1.16.221 mtu=1500 subsys=mtu
    level=info msg="Restored services from maps" failed=0 restored=11 subsys=service
    level=info msg="Reading old endpoints..." subsys=daemon
    level=info msg="No old endpoints found." subsys=daemon
    level=info msg="Envoy: Starting xDS gRPC server listening on /var/run/cilium/xds.sock" subsys=envoy-manager
    level=info msg="Waiting until all Cilium CRDs are available" subsys=k8s
    level=info msg="All Cilium CRDs have been found and are available" subsys=k8s
    level=info msg="Retrieved node information from kubernetes node" nodeName=prod-fe-k8s-wn1 subsys=k8s
    level=info msg="Received own node information from API server" ipAddr.ipv4=10.1.16.221 ipAddr.ipv6="<nil>" k8sNodeIP=10.1.16.221 labels="map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/env:prod kubernetes.io/hostname:prod-fe-k8s-wn1 kubernetes.io/ingress:prod kubernetes.io/os:linux kubernetes.io/resource:prod-fe node-role.kubernetes.io/worker:worker topology.diskplugin.csi.alibabacloud.com/zone:cn-hangzhou-h]" nodeName=prod-fe-k8s-wn1 subsys=k8s v4Prefix=172.21.9.0/24 v6Prefix="<nil>"
    level=info msg="Restored router IPs from node information" ipv4=172.21.9.225 ipv6="<nil>" subsys=k8s
    level=info msg="k8s mode: Allowing localhost to reach local endpoints" subsys=daemon
    level=info msg="Using auto-derived devices to attach Loadbalancer, Host Firewall or Bandwidth Manager program" devices="[eth0]" directRoutingDevice=eth0 subsys=daemon
    level=info msg="Enabling k8s event listener" subsys=k8s-watcher
    level=info msg="Removing stale endpoint interfaces" subsys=daemon
    level=info msg="Skipping kvstore configuration" subsys=daemon
    level=info msg="Restored router address from node_config" file=/var/run/cilium/state/globals/node_config.h ipv4=172.21.9.225 ipv6="<nil>" subsys=node
    level=info msg="Initializing node addressing" subsys=daemon
    level=info msg="Initializing kubernetes IPAM" subsys=ipam v4Prefix=172.21.9.0/24 v6Prefix="<nil>"
    level=info msg="Restoring endpoints..." subsys=daemon
    level=info msg="Endpoints restored" failed=0 restored=0 subsys=daemon
    level=info msg="Addressing information:" subsys=daemon
    level=info msg="  Cluster-Name: default" subsys=daemon
    level=info msg="  Cluster-ID: 0" subsys=daemon
    level=info msg="  Local node-name: prod-fe-k8s-wn1" subsys=daemon
    level=info msg="  Node-IPv6: <nil>" subsys=daemon
    level=info msg="  External-Node IPv4: 10.1.16.221" subsys=daemon
    level=info msg="  Internal-Node IPv4: 172.21.9.225" subsys=daemon
    level=info msg="  IPv4 allocation prefix: 172.21.9.0/24" subsys=daemon
    level=info msg="  Loopback IPv4: 169.254.42.1" subsys=daemon
    level=info msg="  Local IPv4 addresses:" subsys=daemon
    level=info msg="  - 10.1.16.221" subsys=daemon
    level=info msg="Creating or updating CiliumNode resource" node=prod-fe-k8s-wn1 subsys=nodediscovery
    level=info msg="Waiting until all pre-existing resources related to policy have been received" subsys=k8s-watcher
    level=info msg="Adding local node to cluster" node="{prod-fe-k8s-wn1 default [{InternalIP 10.1.16.221} {CiliumInternalIP 172.21.9.225}] 172.21.9.0/24 <nil> 172.21.9.26 <nil> 0 local 0 map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/env:prod kubernetes.io/hostname:prod-fe-k8s-wn1 kubernetes.io/ingress:prod kubernetes.io/os:linux kubernetes.io/resource:prod-fe node-role.kubernetes.io/worker:worker topology.diskplugin.csi.alibabacloud.com/zone:cn-hangzhou-h] 6}" subsys=nodediscovery
    level=info msg="Annotating k8s node" subsys=daemon v4CiliumHostIP.IPv4=172.21.9.225 v4Prefix=172.21.9.0/24 v4healthIP.IPv4=172.21.9.26 v6CiliumHostIP.IPv6="<nil>" v6Prefix="<nil>" v6healthIP.IPv6="<nil>"
    level=info msg="Initializing identity allocator" subsys=identity-cache
    level=info msg="Cluster-ID is not specified, skipping ClusterMesh initialization" subsys=daemon
    level=info msg="Setting up BPF datapath" bpfClockSource=ktime bpfInsnSet=v3 subsys=datapath-loader
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=net.core.bpf_jit_enable sysParamValue=1
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=net.ipv4.conf.all.rp_filter sysParamValue=0
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=kernel.unprivileged_bpf_disabled sysParamValue=1
    level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=kernel.timer_migration sysParamValue=0
    level=info msg="regenerating all endpoints" reason="one or more identities created or deleted" subsys=endpoint-manager
    level=info msg="All pre-existing resources related to policy have been received; continuing" subsys=k8s-watcher
    level=info msg="Adding new proxy port rules for cilium-dns-egress:37581" proxy port name=cilium-dns-egress subsys=proxy
    level=info msg="Serving cilium node monitor v1.2 API at unix:///var/run/cilium/monitor1_2.sock" subsys=monitor-agent
    level=info msg="Validating configured node address ranges" subsys=daemon
    level=info msg="Starting connection tracking garbage collector" subsys=daemon
    level=info msg="Datapath signal listener running" subsys=signal
    level=info msg="Starting IP identity watcher" subsys=ipcache
    level=info msg="Initial scan of connection tracking completed" subsys=ct-gc
    level=info msg="Regenerating restored endpoints" numRestored=0 subsys=daemon
    level=info msg="Creating host endpoint" subsys=daemon
    level=info msg="New endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2159 ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Resolving identity labels (blocking)" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2159 identityLabels="k8s:node-role.kubernetes.io/worker=worker,k8s:topology.diskplugin.csi.alibabacloud.com/zone=cn-hangzhou-h,reserved:host" ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Identity of endpoint changed" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2159 identity=1 identityLabels="k8s:node-role.kubernetes.io/worker=worker,k8s:topology.diskplugin.csi.alibabacloud.com/zone=cn-hangzhou-h,reserved:host" ipv4= ipv6= k8sPodName=/ oldIdentity="no identity" subsys=endpoint
    level=info msg="Launching Cilium health daemon" subsys=daemon
    level=info msg="Finished regenerating restored endpoints" regenerated=0 subsys=daemon total=0
    level=info msg="Launching Cilium health endpoint" subsys=daemon
    level=info msg="Started healthz status API server" address="127.0.0.1:9876" subsys=daemon
    level=info msg="Initializing Cilium API" subsys=daemon
    level=info msg="Daemon initialization completed" bootstrapTime=8.140788349s subsys=daemon
    level=info msg="Serving cilium API at unix:///var/run/cilium/cilium.sock" subsys=daemon
    level=info msg="Configuring Hubble server" eventQueueSize=2048 maxFlows=4095 subsys=hubble
    level=info msg="Starting local Hubble server" address="unix:///var/run/cilium/hubble.sock" subsys=hubble
    level=info msg="Beginning to read perf buffer" startTime="2021-08-27 17:06:36.376614944 +0000 UTC m=+8.225871854" subsys=monitor-agent
    level=info msg="New endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=3199 ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Resolving identity labels (blocking)" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=3199 identityLabels="reserved:health" ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Identity of endpoint changed" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=3199 identity=4 identityLabels="reserved:health" ipv4= ipv6= k8sPodName=/ oldIdentity="no identity" subsys=endpoint
    level=info msg="Compiled new BPF template" BPFCompilationTime=1.833919045s file-path=/var/run/cilium/state/templates/532a69347dd40c75334a195185011bc79bd07ca7/bpf_host.o subsys=datapath-loader
    level=info msg="Rewrote endpoint BPF program" containerID= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=2159 identity=1 ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Compiled new BPF template" BPFCompilationTime=1.88288814s file-path=/var/run/cilium/state/templates/fb6dc13c1055d6e188939f7cb8ae5c7e8ed3fe25/bpf_lxc.o subsys=datapath-loader
    level=info msg="Rewrote endpoint BPF program" containerID= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=3199 identity=4 ipv4= ipv6= k8sPodName=/ subsys=endpoint
    level=info msg="Serving cilium health API at unix:///var/run/cilium/health.sock" subsys=health-server
    level=info msg="Waiting for Hubble server TLS certificate and key files to be created" subsys=hubble
    level=info msg="Conntrack garbage collector interval recalculated" deleteRatio=0.008514404296875 newInterval=7m30s subsys=map-ct
    level=info msg="Conntrack garbage collector interval recalculated" deleteRatio=0.013427734375 newInterval=11m15s subsys=map-ct
    level=info msg="Conntrack garbage collector interval recalculated" deleteRatio=0.0150604248046875 newInterval=16m53s subsys=map-ct
    level=info msg="Conntrack garbage collector interval recalculated" deleteRatio=0.0257110595703125 newInterval=25m20s subsys=map-ct

     

Cilium路由模式(一)

上一篇:3Hello world


下一篇:02-playbook