Cilium路由模式（一）

2023-11-09 21:43:28

Encapsulation 封装

官方文档

缺省模式，Cilium 会自动运行在封装模式下，因为这种模式对底层网络基础设施（特别是云厂商）依赖性比较低，只要2层网络通城，执行封装技术即可达到Pod通信目的

在这种模式下，所有集群节点使用基于 UDP 的封装协议 VXLAN 或 Geneve 形成隧道网状结构。 Cilium 节点之间的所有流量都被封装

二种封装技术

VXLAN
Geneve

如果你的网络有设置防火墙，注意了，这几个端口

Encapsulation Mode	Port Range / Protocol
VXLAN (Default)	8472/UDP
Geneve	6081/UDP

封装优势

简单连接集群节点的网络不需要知道 PodCIDR。集群节点可以产生多个路由或链路层域。只要集群节点可以使用 IP/UDP 相互访问，底层网络的拓扑结构就无关紧要
寻址空间 由于不依赖于任何底层网络限制，如果 PodCIDR 大小相应地配置，可用寻址空间可能要大得多，并且允许每个节点运行任意数量的 Pod
自动配置 当与编排系统（例如 Kubernetes）一起运行时，集群中所有节点的列表（包括其关联的分配前缀节点）将自动提供给每个cilium-agent，加入集群的新节点将自动合并到网格中
identity context 封装协议允许将元数据与网络数据包一起携带。 Cilium 利用这种能力来传输元数据，例如源安全身份。身份传输是一种优化，旨在避免在远程节点上进行一次身份查找

封装劣势

MTU Overhead 由于添加了封装header，MTU太大引起的弊端低于本机路由（VXLAN 的每个网络数据包 50 字节）。这导致特定网络连接影响低吞吐量。当然可以通过启用巨型帧（每 1500 字节有 50 字节的开销，而每 9000 字节有 50 字节的开销）来很大程度上缓解。

配置过程

配置过程如下

helm install cilium cilium/cilium --version 1.9.9     --namespace kube-system     --set tunnel=vxlan     --set kubeProxyReplacement=strict     --set ipam.mode=kubernetes     --set ipam.operator.clusterPoolIPv4PodCIDR=172.21.0.0/20     --set ipam.operator.clusterPoolIPv4MaskSize=26     --set k8sServiceHost=apiserver.qiangyun.com     --set k8sServicePort=6443

观察节点的路由，如下

<root@PROD-FE-K8S-WN1 ~># netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.1.16.253     0.0.0.0         UG        0 0          0 eth0
10.1.16.0       0.0.0.0         255.255.255.0   U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
172.21.0.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
172.21.1.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
172.21.2.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
172.21.3.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
172.21.4.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
172.21.5.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
172.21.6.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
172.21.7.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
172.21.8.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
172.21.9.0      172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
172.21.9.225    0.0.0.0         255.255.255.255 UH        0 0          0 cilium_host
172.21.10.0     172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
172.21.11.0     172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
172.21.12.0     172.21.9.225    255.255.255.192 UG        0 0          0 cilium_host
172.21.12.64    172.21.9.225    255.255.255.192 UG        0 0          0 cilium_host
172.21.12.128   172.21.9.225    255.255.255.192 UG        0 0          0 cilium_host
172.21.12.192   172.21.9.225    255.255.255.192 UG        0 0          0 cilium_host
172.21.13.0     172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
172.21.14.0     172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host
172.21.15.0     172.21.9.225    255.255.255.0   UG        0 0          0 cilium_host

# 说明
每个节点都有一个CiliumInternal地址 172.21.9.225
每个节点都有一个IPAM 
每个节点都有一个healthcheck的地址
以下地址在下方体现

CRD CiliumNode

spec:
  addresses:
    - ip: 10.1.16.221
      type: InternalIP
    - ip: 172.21.9.225
      type: CiliumInternalIP
  azure: {}
  encryption: {}
  eni: {}
  health:
    ipv4: 172.21.9.190
  ipam:
    podCIDRs:
      - 172.21.9.0/24

简单说下，通信原理

<root@PROD-FE-K8S-WN1 ~># ifconfig 
cilium_host: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 172.21.9.225  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 22:cb:9d:23:d8:48  txqueuelen 1000  (Ethernet)
        RX packets 4665  bytes 356292 (347.9 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 273  bytes 19841 (19.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

cilium_net: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST>  mtu 1500
        ether 26:7f:1f:99:b5:db  txqueuelen 1000  (Ethernet)
        RX packets 273  bytes 19841 (19.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4665  bytes 356292 (347.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

cilium_vxlan: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 02:16:be:c2:2f:2f  txqueuelen 1000  (Ethernet)
        RX packets 10023  bytes 634132 (619.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9979  bytes 629067 (614.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
# 说明
cilium_host 类似于一个路由器，网关设备
cilium_net & cilium_host成对出现，像是一个veth，一端接容器一端接主机
cilium_vxlan 就是虚拟二层网络，用来提供Pod跨节点通信MTU封装

虽然cilium默认使用的封装技术，但是host routing模式还是使用到BPF模式，如下

root@PROD-FE-K8S-WN1:/home/cilium# cilium status --verbose
KVStore:                Ok   Disabled
Kubernetes:             Ok   1.18 (v1.18.5) [linux/amd64]
Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:   Strict   [eth0 (Direct Routing)]
Cilium:                 Ok   1.9.9 (v1.9.9-5bcf83c)
NodeMonitor:            Listening for events on 2 CPUs with 64x4096 of shared memory
Cilium health daemon:   Ok   
IPAM:                   IPv4: 2/255 allocated from 172.21.9.0/24, 
Allocated addresses:
  172.21.9.225 (router)
  172.21.9.26 (health)
BandwidthManager:       Disabled
Host Routing:           BPF
Masquerading:           BPF   [eth0]   172.21.9.0/24
Clock Source for BPF:   ktime
Controller Status:      18/18 healthy
  Name                                  Last success   Last error   Count   Message
  cilium-health-ep                      33s ago        never        0       no error   
  dns-garbage-collector-job             41s ago        never        0       no error   
  endpoint-2159-regeneration-recovery   never          never        0       no error   
  endpoint-3199-regeneration-recovery   never          never        0       no error   
  k8s-heartbeat                         11s ago        never        0       no error   
  mark-k8s-node-as-available            48m34s ago     never        0       no error   
  metricsmap-bpf-prom-sync              6s ago         never        0       no error   
  neighbor-table-refresh                3m34s ago      never        0       no error   
  resolve-identity-2159                 3m34s ago      never        0       no error   
  resolve-identity-3199                 3m33s ago      never        0       no error   
  sync-endpoints-and-host-ips           34s ago        never        0       no error   
  sync-lb-maps-with-k8s-services        48m34s ago     never        0       no error   
  sync-policymap-2159                   31s ago        never        0       no error   
  sync-policymap-3199                   31s ago        never        0       no error   
  sync-to-k8s-ciliumendpoint (2159)     4s ago         never        0       no error   
  sync-to-k8s-ciliumendpoint (3199)     13s ago        never        0       no error   
  template-dir-watcher                  never          never        0       no error   
  update-k8s-node-annotations           48m40s ago     never        0       no error   
Proxy Status:   OK, ip 172.21.9.225, 0 redirects active on ports 10000-20000
Hubble:         Ok   Current/Max Flows: 4096/4096 (100.00%), Flows/s: 5.39   Metrics: Disabled
KubeProxyReplacement Details:
  Status:              Strict
  Protocols:           TCP, UDP
  Devices:             eth0 (Direct Routing)
  Mode:                SNAT  （kuber-proxy的模式，默认就是SNAT）
  Backend Selection:   Random
  Session Affinity:    Enabled
  XDP Acceleration:    Disabled
  Services:
  - ClusterIP:      Enabled
  - NodePort:       Enabled (Range: 30000-32767) 
  - LoadBalancer:   Enabled 
  - externalIPs:    Enabled 
  - HostPort:       Enabled
BPF Maps:   dynamic sizing: on (ratio: 0.002500)
  Name                          Size
  Non-TCP connection tracking   65536
  TCP connection tracking       131072
  Endpoint policy               65535
  Events                        2
  IP cache                      512000
  IP masquerading agent         16384
  IPv4 fragmentation            8192
  IPv4 service                  65536
  IPv6 service                  65536
  IPv4 service backend          65536
  IPv6 service backend          65536
  IPv4 service reverse NAT      65536
  IPv6 service reverse NAT      65536
  Metrics                       1024
  NAT                           131072
  Neighbor table                131072
  Global policy                 16384
  Per endpoint policy           65536
  Session affinity              65536
  Signal                        2
  Sockmap                       65535
  Sock reverse NAT              65536
  Tunnel                        65536
Cluster health:                 19/19 reachable   (2021-08-27T17:54:39Z)
  Name                          IP                Node        Endpoints
  prod-fe-k8s-wn1 (localhost)   10.1.16.221       reachable   reachable
  prod-be-k8s-wn1               10.1.17.231       reachable   reachable
  prod-be-k8s-wn2               10.1.17.232       reachable   reachable
  prod-be-k8s-wn6               10.1.17.236       reachable   reachable
  prod-be-k8s-wn7               10.1.17.237       reachable   reachable
  prod-be-k8s-wn8               10.1.17.238       reachable   reachable
  prod-data-k8s-wn1             10.1.18.50        reachable   reachable
  prod-data-k8s-wn2             10.1.18.49        reachable   reachable
  prod-data-k8s-wn3             10.1.18.51        reachable   reachable
  prod-fe-k8s-wn2               10.1.16.222       reachable   reachable
  prod-fe-k8s-wn3               10.1.16.223       reachable   reachable
  prod-k8s-cp1                  10.1.0.5          reachable   reachable
  prod-k8s-cp2                  10.1.0.7          reachable   reachable
  prod-k8s-cp3                  10.1.0.6          reachable   reachable
  prod-sys-k8s-wn1              10.1.0.8          reachable   reachable
  prod-sys-k8s-wn2              10.1.0.9          reachable   reachable
  prod-sys-k8s-wn3              10.1.0.11         reachable   reachable
  prod-sys-k8s-wn4              10.1.0.10         reachable   reachable
  prod-sys-k8s-wn5              10.1.0.12         reachable   reachable

cilium-agent 具体启动输出日志

<root@PROD-FE-K8S-WN1 ~># dps
6fdce5c6148b    Up 51 minutes   k8s_pord-ingress_prod-ingress-b76597794-tmrtc_ingress-nginx_a63d92fe-5c99-4948-89ca-fd70d2298f99_3
43686a967be8    Up 52 minutes   k8s_cilium-agent_cilium-cgrdw_kube-system_14e0fb48-cc56-46d8-b929-64f66b36c6b7_2
<root@PROD-FE-K8S-WN1 ~># dok^C
<root@PROD-FE-K8S-WN1 ~># docker logs -f 43686a967be8
level=info msg="Skipped reading configuration file" reason="Config File \"ciliumd\" Not Found in \"[/root]\"" subsys=config
level=info msg="Started gops server" address="127.0.0.1:9890" subsys=daemon
level=info msg="Memory available for map entries (0.003% of 3976814592B): 9942036B" subsys=config
level=info msg="option bpf-ct-global-tcp-max set by dynamic sizing to 131072" subsys=config
level=info msg="option bpf-ct-global-any-max set by dynamic sizing to 65536" subsys=config
level=info msg="option bpf-nat-global-max set by dynamic sizing to 131072" subsys=config
level=info msg="option bpf-neigh-global-max set by dynamic sizing to 131072" subsys=config
level=info msg="option bpf-sock-rev-map-max set by dynamic sizing to 65536" subsys=config
level=info msg="  --agent-health-port=‘9876‘" subsys=daemon
level=info msg="  --agent-labels=‘‘" subsys=daemon
level=info msg="  --allow-icmp-frag-needed=‘true‘" subsys=daemon
level=info msg="  --allow-localhost=‘auto‘" subsys=daemon
level=info msg="  --annotate-k8s-node=‘true‘" subsys=daemon
level=info msg="  --api-rate-limit=‘map[]‘" subsys=daemon
level=info msg="  --arping-refresh-period=‘5m0s‘" subsys=daemon
level=info msg="  --auto-create-cilium-node-resource=‘true‘" subsys=daemon
level=info msg="  --auto-direct-node-routes=‘false‘" subsys=daemon  DSR 模式关闭状态（因为只能运行于Native-Routing）
level=info msg="  --blacklist-conflicting-routes=‘false‘" subsys=daemon
level=info msg="  --bpf-compile-debug=‘false‘" subsys=daemon
level=info msg="  --bpf-ct-global-any-max=‘262144‘" subsys=daemon
level=info msg="  --bpf-ct-global-tcp-max=‘524288‘" subsys=daemon
level=info msg="  --bpf-ct-timeout-regular-any=‘1m0s‘" subsys=daemon
level=info msg="  --bpf-ct-timeout-regular-tcp=‘6h0m0s‘" subsys=daemon
level=info msg="  --bpf-ct-timeout-regular-tcp-fin=‘10s‘" subsys=daemon
level=info msg="  --bpf-ct-timeout-regular-tcp-syn=‘1m0s‘" subsys=daemon
level=info msg="  --bpf-ct-timeout-service-any=‘1m0s‘" subsys=daemon
level=info msg="  --bpf-ct-timeout-service-tcp=‘6h0m0s‘" subsys=daemon
level=info msg="  --bpf-fragments-map-max=‘8192‘" subsys=daemon
level=info msg="  --bpf-lb-acceleration=‘disabled‘" subsys=daemon
level=info msg="  --bpf-lb-algorithm=‘random‘" subsys=daemon
level=info msg="  --bpf-lb-maglev-hash-seed=‘JLfvgnHc2kaSUFaI‘" subsys=daemon
level=info msg="  --bpf-lb-maglev-table-size=‘16381‘" subsys=daemon
level=info msg="  --bpf-lb-map-max=‘65536‘" subsys=daemon
level=info msg="  --bpf-lb-mode=‘snat‘" subsys=daemon
level=info msg="  --bpf-map-dynamic-size-ratio=‘0.0025‘" subsys=daemon
level=info msg="  --bpf-nat-global-max=‘524288‘" subsys=daemon
level=info msg="  --bpf-neigh-global-max=‘524288‘" subsys=daemon
level=info msg="  --bpf-policy-map-max=‘16384‘" subsys=daemon
level=info msg="  --bpf-root=‘‘" subsys=daemon
level=info msg="  --bpf-sock-rev-map-max=‘262144‘" subsys=daemon
level=info msg="  --certificates-directory=‘/var/run/cilium/certs‘" subsys=daemon
level=info msg="  --cgroup-root=‘/run/cilium/cgroupv2‘" subsys=daemon
level=info msg="  --cluster-id=‘‘" subsys=daemon
level=info msg="  --cluster-name=‘default‘" subsys=daemon
level=info msg="  --clustermesh-config=‘/var/lib/cilium/clustermesh/‘" subsys=daemon
level=info msg="  --cmdref=‘‘" subsys=daemon
level=info msg="  --config=‘‘" subsys=daemon
level=info msg="  --config-dir=‘/tmp/cilium/config-map‘" subsys=daemon
level=info msg="  --conntrack-gc-interval=‘0s‘" subsys=daemon
level=info msg="  --crd-wait-timeout=‘5m0s‘" subsys=daemon
level=info msg="  --datapath-mode=‘veth‘" subsys=daemon
level=info msg="  --debug=‘false‘" subsys=daemon
level=info msg="  --debug-verbose=‘‘" subsys=daemon
level=info msg="  --device=‘‘" subsys=daemon
level=info msg="  --devices=‘‘" subsys=daemon
level=info msg="  --direct-routing-device=‘‘" subsys=daemon
level=info msg="  --disable-cnp-status-updates=‘true‘" subsys=daemon
level=info msg="  --disable-conntrack=‘false‘" subsys=daemon
level=info msg="  --disable-endpoint-crd=‘false‘" subsys=daemon
level=info msg="  --disable-envoy-version-check=‘false‘" subsys=daemon
level=info msg="  --disable-iptables-feeder-rules=‘‘" subsys=daemon
level=info msg="  --dns-max-ips-per-restored-rule=‘1000‘" subsys=daemon
level=info msg="  --egress-masquerade-interfaces=‘‘" subsys=daemon 伪装模式的一种（基于iptables-base），还有一种是基于eBPF-base，二者并存优先取eBPF-base
level=info msg="  --egress-multi-home-ip-rule-compat=‘false‘" subsys=daemon
level=info msg="  --enable-auto-protect-node-port-range=‘true‘" subsys=daemon
level=info msg="  --enable-bandwidth-manager=‘false‘" subsys=daemon
level=info msg="  --enable-bpf-clock-probe=‘true‘" subsys=daemon
level=info msg="  --enable-bpf-masquerade=‘true‘" subsys=daemon
level=info msg="  --enable-bpf-tproxy=‘false‘" subsys=daemon
level=info msg="  --enable-endpoint-health-checking=‘true‘" subsys=daemon
level=info msg="  --enable-endpoint-routes=‘false‘" subsys=daemon  # Native-Routing的另一种路由模式，后面会有介绍，与前者不能并存，否则会报错
level=info msg="  --enable-external-ips=‘true‘" subsys=daemon
level=info msg="  --enable-health-check-nodeport=‘true‘" subsys=daemon
level=info msg="  --enable-health-checking=‘true‘" subsys=daemon
level=info msg="  --enable-host-firewall=‘false‘" subsys=daemon
level=info msg="  --enable-host-legacy-routing=‘false‘" subsys=daemon
level=info msg="  --enable-host-port=‘true‘" subsys=daemon
level=info msg="  --enable-host-reachable-services=‘false‘" subsys=daemon
level=info msg="  --enable-hubble=‘true‘" subsys=daemon
level=info msg="  --enable-identity-mark=‘true‘" subsys=daemon
level=info msg="  --enable-ip-masq-agent=‘false‘" subsys=daemon
level=info msg="  --enable-ipsec=‘false‘" subsys=daemon
level=info msg="  --enable-ipv4=‘true‘" subsys=daemon
level=info msg="  --enable-ipv4-fragment-tracking=‘true‘" subsys=daemon
level=info msg="  --enable-ipv6=‘false‘" subsys=daemon
level=info msg="  --enable-ipv6-ndp=‘false‘" subsys=daemon
level=info msg="  --enable-k8s-api-discovery=‘false‘" subsys=daemon
level=info msg="  --enable-k8s-endpoint-slice=‘true‘" subsys=daemon
level=info msg="  --enable-k8s-event-handover=‘false‘" subsys=daemon
level=info msg="  --enable-l7-proxy=‘true‘" subsys=daemon
level=info msg="  --enable-local-node-route=‘true‘" subsys=daemon
level=info msg="  --enable-local-redirect-policy=‘false‘" subsys=daemon
level=info msg="  --enable-monitor=‘true‘" subsys=daemon
level=info msg="  --enable-node-port=‘false‘" subsys=daemon
level=info msg="  --enable-policy=‘default‘" subsys=daemon
level=info msg="  --enable-remote-node-identity=‘true‘" subsys=daemon
level=info msg="  --enable-selective-regeneration=‘true‘" subsys=daemon
level=info msg="  --enable-session-affinity=‘true‘" subsys=daemon
level=info msg="  --enable-svc-source-range-check=‘true‘" subsys=daemon
level=info msg="  --enable-tracing=‘false‘" subsys=daemon
level=info msg="  --enable-well-known-identities=‘false‘" subsys=daemon
level=info msg="  --enable-xt-socket-fallback=‘true‘" subsys=daemon
level=info msg="  --encrypt-interface=‘‘" subsys=daemon
level=info msg="  --encrypt-node=‘false‘" subsys=daemon
level=info msg="  --endpoint-interface-name-prefix=‘lxc+‘" subsys=daemon
level=info msg="  --endpoint-queue-size=‘25‘" subsys=daemon
level=info msg="  --endpoint-status=‘‘" subsys=daemon
level=info msg="  --envoy-log=‘‘" subsys=daemon
level=info msg="  --exclude-local-address=‘‘" subsys=daemon
level=info msg="  --fixed-identity-mapping=‘map[]‘" subsys=daemon
level=info msg="  --flannel-master-device=‘‘" subsys=daemon
level=info msg="  --flannel-uninstall-on-exit=‘false‘" subsys=daemon
level=info msg="  --force-local-policy-eval-at-source=‘true‘" subsys=daemon
level=info msg="  --gops-port=‘9890‘" subsys=daemon
level=info msg="  --host-reachable-services-protos=‘tcp,udp‘" subsys=daemon
level=info msg="  --http-403-msg=‘‘" subsys=daemon
level=info msg="  --http-idle-timeout=‘0‘" subsys=daemon
level=info msg="  --http-max-grpc-timeout=‘0‘" subsys=daemon
level=info msg="  --http-normalize-path=‘true‘" subsys=daemon
level=info msg="  --http-request-timeout=‘3600‘" subsys=daemon
level=info msg="  --http-retry-count=‘3‘" subsys=daemon
level=info msg="  --http-retry-timeout=‘0‘" subsys=daemon
level=info msg="  --hubble-disable-tls=‘false‘" subsys=daemon
level=info msg="  --hubble-event-queue-size=‘0‘" subsys=daemon
level=info msg="  --hubble-flow-buffer-size=‘4095‘" subsys=daemon
level=info msg="  --hubble-listen-address=‘:4244‘" subsys=daemon
level=info msg="  --hubble-metrics=‘‘" subsys=daemon
level=info msg="  --hubble-metrics-server=‘‘" subsys=daemon
level=info msg="  --hubble-socket-path=‘/var/run/cilium/hubble.sock‘" subsys=daemon
level=info msg="  --hubble-tls-cert-file=‘/var/lib/cilium/tls/hubble/server.crt‘" subsys=daemon
level=info msg="  --hubble-tls-client-ca-files=‘/var/lib/cilium/tls/hubble/client-ca.crt‘" subsys=daemon
level=info msg="  --hubble-tls-key-file=‘/var/lib/cilium/tls/hubble/server.key‘" subsys=daemon
level=info msg="  --identity-allocation-mode=‘crd‘" subsys=daemon
level=info msg="  --identity-change-grace-period=‘5s‘" subsys=daemon
level=info msg="  --install-iptables-rules=‘true‘" subsys=daemon
level=info msg="  --ip-allocation-timeout=‘2m0s‘" subsys=daemon
level=info msg="  --ip-masq-agent-config-path=‘/etc/config/ip-masq-agent‘" subsys=daemon
level=info msg="  --ipam=‘kubernetes‘" subsys=daemon CRD的模式，有好几种，官方默认是cluster-pool,kubernetes模式代表从node到拿IP地址，基于controller-manager 启动参数--allocate-node-cidrs
level=info msg="  --ipsec-key-file=‘‘" subsys=daemon
level=info msg="  --iptables-lock-timeout=‘5s‘" subsys=daemon
level=info msg="  --iptables-random-fully=‘false‘" subsys=daemon
level=info msg="  --ipv4-node=‘auto‘" subsys=daemon
level=info msg="  --ipv4-pod-subnets=‘‘" subsys=daemon
level=info msg="  --ipv4-range=‘auto‘" subsys=daemon
level=info msg="  --ipv4-service-loopback-address=‘169.254.42.1‘" subsys=daemon
level=info msg="  --ipv4-service-range=‘auto‘" subsys=daemon
level=info msg="  --ipv6-cluster-alloc-cidr=‘f00d::/64‘" subsys=daemon
level=info msg="  --ipv6-mcast-device=‘‘" subsys=daemon
level=info msg="  --ipv6-node=‘auto‘" subsys=daemon
level=info msg="  --ipv6-pod-subnets=‘‘" subsys=daemon
level=info msg="  --ipv6-range=‘auto‘" subsys=daemon
level=info msg="  --ipv6-service-range=‘auto‘" subsys=daemon
level=info msg="  --ipvlan-master-device=‘undefined‘" subsys=daemon
level=info msg="  --join-cluster=‘false‘" subsys=daemon
level=info msg="  --k8s-api-server=‘‘" subsys=daemon
level=info msg="  --k8s-force-json-patch=‘false‘" subsys=daemon
level=info msg="  --k8s-heartbeat-timeout=‘30s‘" subsys=daemon
level=info msg="  --k8s-kubeconfig-path=‘‘" subsys=daemon
level=info msg="  --k8s-namespace=‘kube-system‘" subsys=daemon
level=info msg="  --k8s-require-ipv4-pod-cidr=‘false‘" subsys=daemon
level=info msg="  --k8s-require-ipv6-pod-cidr=‘false‘" subsys=daemon
level=info msg="  --k8s-service-cache-size=‘128‘" subsys=daemon
level=info msg="  --k8s-service-proxy-name=‘‘" subsys=daemon
level=info msg="  --k8s-sync-timeout=‘3m0s‘" subsys=daemon
level=info msg="  --k8s-watcher-endpoint-selector=‘metadata.name!=kube-scheduler,metadata.name!=kube-controller-manager,metadata.name!=etcd-operator,metadata.name!=gcp-controller-manager‘" subsys=daemon
level=info msg="  --k8s-watcher-queue-size=‘1024‘" subsys=daemon
level=info msg="  --keep-config=‘false‘" subsys=daemon
level=info msg="  --kube-proxy-replacement=‘strict‘" subsys=daemon
level=info msg="  --kube-proxy-replacement-healthz-bind-address=‘‘" subsys=daemon
level=info msg="  --kvstore=‘‘" subsys=daemon
level=info msg="  --kvstore-connectivity-timeout=‘2m0s‘" subsys=daemon
level=info msg="  --kvstore-lease-ttl=‘15m0s‘" subsys=daemon
level=info msg="  --kvstore-opt=‘map[]‘" subsys=daemon
level=info msg="  --kvstore-periodic-sync=‘5m0s‘" subsys=daemon
level=info msg="  --label-prefix-file=‘‘" subsys=daemon
level=info msg="  --labels=‘‘" subsys=daemon
level=info msg="  --lib-dir=‘/var/lib/cilium‘" subsys=daemon
level=info msg="  --log-driver=‘‘" subsys=daemon
level=info msg="  --log-opt=‘map[]‘" subsys=daemon
level=info msg="  --log-system-load=‘false‘" subsys=daemon
level=info msg="  --masquerade=‘true‘" subsys=daemon
level=info msg="  --max-controller-interval=‘0‘" subsys=daemon
level=info msg="  --metrics=‘‘" subsys=daemon
level=info msg="  --monitor-aggregation=‘medium‘" subsys=daemon
level=info msg="  --monitor-aggregation-flags=‘all‘" subsys=daemon
level=info msg="  --monitor-aggregation-interval=‘5s‘" subsys=daemon
level=info msg="  --monitor-queue-size=‘0‘" subsys=daemon
level=info msg="  --mtu=‘0‘" subsys=daemon
level=info msg="  --nat46-range=‘0:0:0:0:0:FFFF::/96‘" subsys=daemon
level=info msg="  --native-routing-cidr=‘‘" subsys=daemon
level=info msg="  --node-port-acceleration=‘disabled‘" subsys=daemon
level=info msg="  --node-port-algorithm=‘random‘" subsys=daemon
level=info msg="  --node-port-bind-protection=‘true‘" subsys=daemon
level=info msg="  --node-port-mode=‘snat‘" subsys=daemon
level=info msg="  --node-port-range=‘30000,32767‘" subsys=daemon
level=info msg="  --policy-audit-mode=‘false‘" subsys=daemon
level=info msg="  --policy-queue-size=‘100‘" subsys=daemon
level=info msg="  --policy-trigger-interval=‘1s‘" subsys=daemon
level=info msg="  --pprof=‘false‘" subsys=daemon
level=info msg="  --preallocate-bpf-maps=‘false‘" subsys=daemon
level=info msg="  --prefilter-device=‘undefined‘" subsys=daemon
level=info msg="  --prefilter-mode=‘native‘" subsys=daemon
level=info msg="  --prepend-iptables-chains=‘true‘" subsys=daemon
level=info msg="  --prometheus-serve-addr=‘‘" subsys=daemon
level=info msg="  --proxy-connect-timeout=‘1‘" subsys=daemon
level=info msg="  --proxy-prometheus-port=‘0‘" subsys=daemon
level=info msg="  --read-cni-conf=‘‘" subsys=daemon
level=info msg="  --restore=‘true‘" subsys=daemon
level=info msg="  --sidecar-istio-proxy-image=‘cilium/istio_proxy‘" subsys=daemon
level=info msg="  --single-cluster-route=‘false‘" subsys=daemon
level=info msg="  --skip-crd-creation=‘false‘" subsys=daemon
level=info msg="  --socket-path=‘/var/run/cilium/cilium.sock‘" subsys=daemon
level=info msg="  --sockops-enable=‘false‘" subsys=daemon
level=info msg="  --state-dir=‘/var/run/cilium‘" subsys=daemon
level=info msg="  --tofqdns-dns-reject-response-code=‘refused‘" subsys=daemon
level=info msg="  --tofqdns-enable-dns-compression=‘true‘" subsys=daemon
level=info msg="  --tofqdns-endpoint-max-ip-per-hostname=‘50‘" subsys=daemon
level=info msg="  --tofqdns-idle-connection-grace-period=‘0s‘" subsys=daemon
level=info msg="  --tofqdns-max-deferred-connection-deletes=‘10000‘" subsys=daemon
level=info msg="  --tofqdns-min-ttl=‘0‘" subsys=daemon
level=info msg="  --tofqdns-pre-cache=‘‘" subsys=daemon
level=info msg="  --tofqdns-proxy-port=‘0‘" subsys=daemon
level=info msg="  --tofqdns-proxy-response-max-delay=‘100ms‘" subsys=daemon
level=info msg="  --trace-payloadlen=‘128‘" subsys=daemon
level=info msg="  --tunnel=‘vxlan‘" subsys=daemon
level=info msg="  --version=‘false‘" subsys=daemon
level=info msg="  --write-cni-conf-when-ready=‘‘" subsys=daemon
level=info msg="     _ _ _" subsys=daemon
level=info msg=" ___|_| |_|_ _ _____" subsys=daemon
level=info msg="|  _| | | | | |     |" subsys=daemon
level=info msg="|___|_|_|_|___|_|_|_|" subsys=daemon
level=info msg="Cilium 1.9.9 5bcf83c 2021-07-19T16:45:00-07:00 go version go1.15.14 linux/amd64" subsys=daemon
level=info msg="cilium-envoy  version: 82a70d56bf324287ced3129300db609eceb21d10/1.17.3/Distribution/RELEASE/BoringSSL" subsys=daemon
level=info msg="clang (10.0.0) and kernel (5.11.1) versions: OK!" subsys=linux-datapath
level=info msg="linking environment: OK!" subsys=linux-datapath
level=info msg="Detected mounted BPF filesystem at /sys/fs/bpf" subsys=bpf
level=info msg="Mounted cgroupv2 filesystem at /run/cilium/cgroupv2" subsys=cgroups
level=info msg="Parsing base label prefixes from default label list" subsys=labels-filter
level=info msg="Parsing additional label prefixes from user inputs: []" subsys=labels-filter
level=info msg="Final label prefixes to be used for identity evaluation:" subsys=labels-filter
level=info msg=" - reserved:.*" subsys=labels-filter
level=info msg=" - :io.kubernetes.pod.namespace" subsys=labels-filter
level=info msg=" - :io.cilium.k8s.namespace.labels" subsys=labels-filter
level=info msg=" - :app.kubernetes.io" subsys=labels-filter
level=info msg=" - !:io.kubernetes" subsys=labels-filter
level=info msg=" - !:kubernetes.io" subsys=labels-filter
level=info msg=" - !:.*beta.kubernetes.io" subsys=labels-filter
level=info msg=" - !:k8s.io" subsys=labels-filter
level=info msg=" - !:pod-template-generation" subsys=labels-filter
level=info msg=" - !:pod-template-hash" subsys=labels-filter
level=info msg=" - !:controller-revision-hash" subsys=labels-filter
level=info msg=" - !:annotation.*" subsys=labels-filter
level=info msg=" - !:etcd_node" subsys=labels-filter
level=info msg="Auto-disabling \"enable-bpf-clock-probe\" feature since KERNEL_HZ cannot be determined" error="Cannot probe CONFIG_HZ" subsys=daemon
level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=10.221.0.0/16
level=info msg="Initializing daemon" subsys=daemon
level=info msg="Establishing connection to apiserver" host="https://apiserver.qiangyun.com:6443" subsys=k8s
level=info msg="Connected to apiserver" subsys=k8s
level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"enable-host-reachable-services\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
level=info msg="Inheriting MTU from external network interface" device=eth0 ipAddr=10.1.16.221 mtu=1500 subsys=mtu
level=info msg="Restored services from maps" failed=0 restored=11 subsys=service
level=info msg="Reading old endpoints..." subsys=daemon
level=info msg="No old endpoints found." subsys=daemon
level=info msg="Envoy: Starting xDS gRPC server listening on /var/run/cilium/xds.sock" subsys=envoy-manager
level=info msg="Waiting until all Cilium CRDs are available" subsys=k8s
level=info msg="All Cilium CRDs have been found and are available" subsys=k8s
level=info msg="Retrieved node information from kubernetes node" nodeName=prod-fe-k8s-wn1 subsys=k8s
level=info msg="Received own node information from API server" ipAddr.ipv4=10.1.16.221 ipAddr.ipv6="<nil>" k8sNodeIP=10.1.16.221 labels="map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/env:prod kubernetes.io/hostname:prod-fe-k8s-wn1 kubernetes.io/ingress:prod kubernetes.io/os:linux kubernetes.io/resource:prod-fe node-role.kubernetes.io/worker:worker topology.diskplugin.csi.alibabacloud.com/zone:cn-hangzhou-h]" nodeName=prod-fe-k8s-wn1 subsys=k8s v4Prefix=172.21.9.0/24 v6Prefix="<nil>"
level=info msg="Restored router IPs from node information" ipv4=172.21.9.225 ipv6="<nil>" subsys=k8s
level=info msg="k8s mode: Allowing localhost to reach local endpoints" subsys=daemon
level=info msg="Using auto-derived devices to attach Loadbalancer, Host Firewall or Bandwidth Manager program" devices="[eth0]" directRoutingDevice=eth0 subsys=daemon
level=info msg="Enabling k8s event listener" subsys=k8s-watcher
level=info msg="Removing stale endpoint interfaces" subsys=daemon
level=info msg="Skipping kvstore configuration" subsys=daemon
level=info msg="Restored router address from node_config" file=/var/run/cilium/state/globals/node_config.h ipv4=172.21.9.225 ipv6="<nil>" subsys=node
level=info msg="Initializing node addressing" subsys=daemon
level=info msg="Initializing kubernetes IPAM" subsys=ipam v4Prefix=172.21.9.0/24 v6Prefix="<nil>"
level=info msg="Restoring endpoints..." subsys=daemon
level=info msg="Endpoints restored" failed=0 restored=0 subsys=daemon
level=info msg="Addressing information:" subsys=daemon
level=info msg="  Cluster-Name: default" subsys=daemon
level=info msg="  Cluster-ID: 0" subsys=daemon
level=info msg="  Local node-name: prod-fe-k8s-wn1" subsys=daemon
level=info msg="  Node-IPv6: <nil>" subsys=daemon
level=info msg="  External-Node IPv4: 10.1.16.221" subsys=daemon
level=info msg="  Internal-Node IPv4: 172.21.9.225" subsys=daemon
level=info msg="  IPv4 allocation prefix: 172.21.9.0/24" subsys=daemon
level=info msg="  Loopback IPv4: 169.254.42.1" subsys=daemon
level=info msg="  Local IPv4 addresses:" subsys=daemon
level=info msg="  - 10.1.16.221" subsys=daemon
level=info msg="Creating or updating CiliumNode resource" node=prod-fe-k8s-wn1 subsys=nodediscovery
level=info msg="Waiting until all pre-existing resources related to policy have been received" subsys=k8s-watcher
level=info msg="Adding local node to cluster" node="{prod-fe-k8s-wn1 default [{InternalIP 10.1.16.221} {CiliumInternalIP 172.21.9.225}] 172.21.9.0/24 <nil> 172.21.9.26 <nil> 0 local 0 map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/env:prod kubernetes.io/hostname:prod-fe-k8s-wn1 kubernetes.io/ingress:prod kubernetes.io/os:linux kubernetes.io/resource:prod-fe node-role.kubernetes.io/worker:worker topology.diskplugin.csi.alibabacloud.com/zone:cn-hangzhou-h] 6}" subsys=nodediscovery
level=info msg="Annotating k8s node" subsys=daemon v4CiliumHostIP.IPv4=172.21.9.225 v4Prefix=172.21.9.0/24 v4healthIP.IPv4=172.21.9.26 v6CiliumHostIP.IPv6="<nil>" v6Prefix="<nil>" v6healthIP.IPv6="<nil>"
level=info msg="Initializing identity allocator" subsys=identity-cache
level=info msg="Cluster-ID is not specified, skipping ClusterMesh initialization" subsys=daemon
level=info msg="Setting up BPF datapath" bpfClockSource=ktime bpfInsnSet=v3 subsys=datapath-loader
level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=net.core.bpf_jit_enable sysParamValue=1
level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=net.ipv4.conf.all.rp_filter sysParamValue=0
level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=kernel.unprivileged_bpf_disabled sysParamValue=1
level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=kernel.timer_migration sysParamValue=0
level=info msg="regenerating all endpoints" reason="one or more identities created or deleted" subsys=endpoint-manager
level=info msg="All pre-existing resources related to policy have been received; continuing" subsys=k8s-watcher
level=info msg="Adding new proxy port rules for cilium-dns-egress:37581" proxy port name=cilium-dns-egress subsys=proxy
level=info msg="Serving cilium node monitor v1.2 API at unix:///var/run/cilium/monitor1_2.sock" subsys=monitor-agent
level=info msg="Validating configured node address ranges" subsys=daemon
level=info msg="Starting connection tracking garbage collector" subsys=daemon
level=info msg="Datapath signal listener running" subsys=signal
level=info msg="Starting IP identity watcher" subsys=ipcache
level=info msg="Initial scan of connection tracking completed" subsys=ct-gc
level=info msg="Regenerating restored endpoints" numRestored=0 subsys=daemon
level=info msg="Creating host endpoint" subsys=daemon
level=info msg="New endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2159 ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=info msg="Resolving identity labels (blocking)" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2159 identityLabels="k8s:node-role.kubernetes.io/worker=worker,k8s:topology.diskplugin.csi.alibabacloud.com/zone=cn-hangzhou-h,reserved:host" ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=info msg="Identity of endpoint changed" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=2159 identity=1 identityLabels="k8s:node-role.kubernetes.io/worker=worker,k8s:topology.diskplugin.csi.alibabacloud.com/zone=cn-hangzhou-h,reserved:host" ipv4= ipv6= k8sPodName=/ oldIdentity="no identity" subsys=endpoint
level=info msg="Launching Cilium health daemon" subsys=daemon
level=info msg="Finished regenerating restored endpoints" regenerated=0 subsys=daemon total=0
level=info msg="Launching Cilium health endpoint" subsys=daemon
level=info msg="Started healthz status API server" address="127.0.0.1:9876" subsys=daemon
level=info msg="Initializing Cilium API" subsys=daemon
level=info msg="Daemon initialization completed" bootstrapTime=8.140788349s subsys=daemon
level=info msg="Serving cilium API at unix:///var/run/cilium/cilium.sock" subsys=daemon
level=info msg="Configuring Hubble server" eventQueueSize=2048 maxFlows=4095 subsys=hubble
level=info msg="Starting local Hubble server" address="unix:///var/run/cilium/hubble.sock" subsys=hubble
level=info msg="Beginning to read perf buffer" startTime="2021-08-27 17:06:36.376614944 +0000 UTC m=+8.225871854" subsys=monitor-agent
level=info msg="New endpoint" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=3199 ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=info msg="Resolving identity labels (blocking)" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=3199 identityLabels="reserved:health" ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=info msg="Identity of endpoint changed" containerID= datapathPolicyRevision=0 desiredPolicyRevision=0 endpointID=3199 identity=4 identityLabels="reserved:health" ipv4= ipv6= k8sPodName=/ oldIdentity="no identity" subsys=endpoint
level=info msg="Compiled new BPF template" BPFCompilationTime=1.833919045s file-path=/var/run/cilium/state/templates/532a69347dd40c75334a195185011bc79bd07ca7/bpf_host.o subsys=datapath-loader
level=info msg="Rewrote endpoint BPF program" containerID= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=2159 identity=1 ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=info msg="Compiled new BPF template" BPFCompilationTime=1.88288814s file-path=/var/run/cilium/state/templates/fb6dc13c1055d6e188939f7cb8ae5c7e8ed3fe25/bpf_lxc.o subsys=datapath-loader
level=info msg="Rewrote endpoint BPF program" containerID= datapathPolicyRevision=0 desiredPolicyRevision=1 endpointID=3199 identity=4 ipv4= ipv6= k8sPodName=/ subsys=endpoint
level=info msg="Serving cilium health API at unix:///var/run/cilium/health.sock" subsys=health-server
level=info msg="Waiting for Hubble server TLS certificate and key files to be created" subsys=hubble
level=info msg="Conntrack garbage collector interval recalculated" deleteRatio=0.008514404296875 newInterval=7m30s subsys=map-ct
level=info msg="Conntrack garbage collector interval recalculated" deleteRatio=0.013427734375 newInterval=11m15s subsys=map-ct
level=info msg="Conntrack garbage collector interval recalculated" deleteRatio=0.0150604248046875 newInterval=16m53s subsys=map-ct
level=info msg="Conntrack garbage collector interval recalculated" deleteRatio=0.0257110595703125 newInterval=25m20s subsys=map-ct

Cilium路由模式（一）

码农公寓

Encapsulation 封装

封装优势

封装劣势

配置过程

相关文章