一，前言
阿里云Kubernetes集群网络目前有两种方案，一种是flannel方案，另外一种是基于Calico和弹性网卡ENI的Terway方案。Terway和Flannel类似，不同的地方在于，Terway支持Pod弹性网卡，以及Network Policy功能。
这个系列文章中我们以实际案例排查，来看一看在容器网络里面，抓包分析问题的技巧，在看这一章之前，建议先看声东同学的《集群网络详解》这一章节
集群网络的架构如下图所示，在问题实际发生时，抓eth0的报文会有什么问题？如何抓veth网卡呢？影响pod网络通讯的几个因素？（安全组，vpc路由表指向，系统内的转发参数等）
集群网络示意图1-1
二，示例
之前遇到一个liveness健康检查失败的问题，liveness是kubelet发起本机访问pod（如发送get请求）来确认的pod健康与否，对应健康检查的访问日志在pod内是可以看到的，因此可以尝试抓包排查，但是pod内部并不能简简单单的使用tcpdump抓包即可，docker/kubectl cp 一个tcpdump命令进去也不能直接抓报文，那么，pod内的网络报文怎么抓呢？直接在ecs上抓包，有的时候并不能很明朗的展示问题，这次我们就说一说pod内的网络报文怎么抓？直接抓-i any接口的报文， 是否存在不利于问题的分析干扰因素?
容器的网络隔离使用的是linux的network namespace ，因此我们需要切换到对应的ns里面抓包，下面开始演示
1.查看指定 pod 运行在哪个宿主机上
拿到宿主机的信息，然后登陆上去 cn-shenzhen.192.168.0.130
# kubectl get pods coredns-79989b94b6-d8kqn -o wide -n kube-system
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-79989b94b6-d8kqn 1/1 Running 0 20d 172.20.1.24 cn-shenzhen.192.168.0.130 <none> <none>2.获得容器的 pid
登录到对应的node上，找对应业务pod pid的方法（k8s里面引入了pause概念的容器忽略即可）
# docker ps |grep coredns-79989b94b6-d8kqn
1285db52efdd registry-vpc.cn-shenzhen.aliyuncs.com/acs/coredns "/coredns -conf /etc…" 2 weeks ago Up 2 weeks k8s_coredns_coredns-79989b94b6-d8kqn_kube-system_78659545-8c40-4e11-8998-4d5eb7e553e5_0
9f31f9fc7887 registry-vpc.cn-shenzhen.aliyuncs.com/acs/pause-amd64:3.0 "/pause" 2 weeks ago Up 2 weeks k8s_POD_coredns-79989b94b6-d8kqn_kube-system_78659545-8c40-4e11-8998-4d5eb7e553e5_0
# docker inspect -f {{.State.Pid}} 1285db52efdd
26063.进入该容器的 network namespace
#切换ns的命令 nsenter 依赖 util-linux，默认未安装的话，使用yum安装即可（centos）
# yum -y install util-linux.x86_64
进入到对应容器的network ns里面，并指向ip a查看ip
# nsenter --target 2606 -n
# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.20.1.24 netmask 255.255.255.255 broadcast 0.0.0.0
ether 42:75:08:d7:d8:49 txqueuelen 0 (Ethernet)
RX packets 43296250 bytes 7672294979 (7.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 42881762 bytes 10464280231 (9.7 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 17453316 bytes 1392774176 (1.2 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 17453316 bytes 1392774176 (1.2 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
对比一下第一步拿到的pod ip 是一样的
看下pod的监听
# netstat -antpl |grep -vi wait
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 172.20.1.24:50416 172.21.0.1:443 ESTABLISHED 2606/coredns
tcp6 0 0 :::9153 :::* LISTEN 2606/coredns
tcp6 0 0 :::8080 :::* LISTEN 2606/coredns
tcp6 0 0 :::8181 :::* LISTEN 2606/coredns
tcp6 0 0 :::53 :::* LISTEN 2606/coredns
tcp6 0 0 172.20.1.24:9153 172.20.2.69:33138 ESTABLISHED 2606/coredns
tcp6 0 0 172.20.1.24:9153 172.20.3.52:40684 ESTABLISHED 2606/coredns
tcp6 0 0 172.20.1.24:9153 172.20.3.53:59034 ESTABLISHED 2606/coredns
tcp6 0 0 172.20.1.24:9153 172.20.3.52:40598 ESTABLISHED 2606/coredns4.使用 tcpdump 抓包，指定 eth0 网卡
直接抓包尝试,演示抓的是coredns的包，所以看到的是一些解析相关的报文
# tcpdump -i eth0 port 53 -nnvv -xxx |more
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:34:21.420121 IP (tos 0x0, ttl 63, id 51339, offset 0, flags [DF], proto UDP (17), length 152)
172.20.2.69.59687 > 172.20.1.24.53: [udp sum ok] 40871+ AAAA? proj-arms-prometheus-1721440269810815-cn-shenzhen.cn-shenzhen.log.aliyunc
s.com.arms-prom.svc.cluster.local. (124)
0x0000: 4275 08d7 d849 2686 12b1 bde5 0800 4500
0x0010: 0098 c88b 4000 3f11 1744 ac14 0245 ac14
0x0020: 0118 e927 0035 0084 0382 9fa7 0100 0001
0x0030: 0000 0000 0000 3170 726f 6a2d 6172 6d73
0x0040: 2d70 726f 6d65 7468 6575 732d 3137 3231
0x0050: 3434 3032 3639 3831 3038 3135 2d63 6e2d
0x0060: 7368 656e 7a68 656e 0b63 6e2d 7368 656e
0x0070: 7a68 656e 036c 6f67 0861 6c69 7975 6e63
0x0080: 7303 636f 6d09 6172 6d73 2d70 726f 6d03
0x0090: 7376 6307 636c 7573 7465 7205 6c6f 6361
0x00a0: 6c00 001c 0001
5.退出network namespace
直接exit即可
# exit
logout
# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.130 netmask 255.255.255.0 broadcast 192.168.0.255
ether 00:16:3e:10:51:22 txqueuelen 1000 (Ethernet)
RX packets 3500312304 bytes 2290117520419 (2.0 TiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2159089229 bytes 2977416649618 (2.7 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
......命令扩展：
nsenter切换namespace
nsenter切换namespace，常用的是网络的命名空间，对应-t是pid，-n为切换网络命名空间
# nsenter -h
Usage:
nsenter [options] <program> [<argument>...]
Run a program with namespaces of other processes.
Options:
-t, --target <pid> target process to get namespaces from
-m, --mount[=<file>] enter mount namespace
-u, --uts[=<file>] enter UTS namespace (hostname etc)
-i, --ipc[=<file>] enter System V IPC namespace
-n, --net[=<file>] enter network namespace
-p, --pid[=<file>] enter pid namespace
-U, --user[=<file>] enter user namespace
-S, --setuid <uid> set uid in entered namespace
-G, --setgid <gid> set gid in entered namespace
--preserve-credentials do not touch uids or gids
-r, --root[=<dir>] set the root directory
-w, --wd[=<dir>] set the working directory
-F, --no-fork do not fork before exec'ing <program>
-Z, --follow-context set SELinux context according to --target PID
-h, --help display this help and exit
-V, --version output version information and exit
For more details see nsenter(1).
除了docker inspect找pid，还有别的方式查namespace么？lsns
lsns显示的pid列，就是前面docke拿到的pid，nsenter需要切换ns的pid
# lsns -h
Usage:
lsns [options] [<namespace>]
List system namespaces.
Options:
-l, --list use list format output
-n, --noheadings don't print headings
-o, --output <list> define which output columns to use
-p, --task <pid> print process namespaces
-r, --raw use the raw output format
-u, --notruncate don't truncate text in columns
-t, --type <name> namespace type (mnt, net, ipc, user, pid, uts)
-h, --help display this help and exit
-V, --version output version information and exit
Available columns (for --output):
NS namespace identifier (inode number)
TYPE kind of namespace
PATH path to the namespace
NPROCS number of processes in the namespace
PID lowest PID in the namespace
PPID PPID of the PID
COMMAND command line of the PID
UID UID of the PID
USER username of the PID
For more details see lsns(8).
demo：
# lsns |grep -vi "pause"
NS TYPE NPROCS PID USER COMMAND
4026531836 pid 173 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 22
4026531837 user 247 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 22
4026531838 uts 172 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 22
4026531839 ipc 165 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 22
4026531840 mnt 163 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 22
4026531856 mnt 1 28 root kdevtmpfs
4026531956 net 177 1 root /usr/lib/systemd/systemd --switched-root --system --deserialize 22
4026532183 mnt 1 2888 chrony /usr/sbin/chronyd
4026532188 mnt 1 7740 root /usr/bin/terwayd
4026532190 mnt 1 5172 nfsnobody /bin/node_exporter --path.procfs=/host/proc --path.sysfs=/host/sys --web.listen-address=0.0.0.0:9100 --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+)($|/) --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|cgroup|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|mqueue|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|sysfs|tracefs)$
4026532192 mnt 2 28585 root nginx: master process nginx -g daemon off
4026532193 uts 2 28585 root nginx: master process nginx -g daemon off
4026532194 pid 2 28585 root nginx: master process nginx -g daemon off
4026532195 mnt 4 2868 root sleep 3
4026532196 mnt 1 15067 root /opt/services/service-catalog controller-manager --port 8080 --leader-elect=false -v 10 --resync-interval 5m --broker-relist-interval 24h
4026532197 uts 1 15067 root /opt/services/service-catalog controller-manager --port 8080 --leader-elect=false -v 10 --resync-interval 5m --broker-relist-interval 24h
4026532198 pid 1 15067 root /opt/services/service-catalog controller-manager --port 8080 --leader-elect=false -v 10 --resync-interval 5m --broker-relist-interval 24h
4026532199 uts 4 2868 root sleep 3
4026532200 pid 4 2868 root sleep 3
4026532290 mnt 1 7451 nfsnobody /alicloud-monitor-controller agent --regionId=cn-zhangjiakou --clusterId=c50c9b260a208475987c2b502f5cc60b1 --logtostderr --v=4
4026532291 pid 1 7451 nfsnobody /alicloud-monitor-controller agent --regionId=cn-zhangjiakou --clusterId=c50c9b260a208475987c2b502f5cc60b1 --logtostderr --v=4
4026532426 mnt 1 5118 root /usr/local/bin/kube-proxy --proxy-mode=iptables --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf --cluster-cidr=172.20.0.0/16 --hostname-override=cn-zhangjiakou.i-8vbalkelzd05pwihto21
4026532427 pid 1 5118 root /usr/local/bin/kube-proxy --proxy-mode=iptables --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf --cluster-cidr=172.20.0.0/16 --hostname-override=cn-zhangjiakou.i-8vbalkelzd05pwihto21
4026532611 mnt 3 24418 root nginx: master process nginx -g daemon off
4026532612 uts 3 24418 root nginx: master process nginx -g daemon off
4026532613 pid 3 24418 root nginx: master process nginx -g daemon off
4026532614 mnt 2 28721 root /bin/sh -c /usr/local/bin/kubectl apply -f /etc/istio-operator/crds; sleep 4; /manager --watch-created-resources-events=false --zap-time-encoding=iso8601
4026532615 uts 2 28721 root /bin/sh -c /usr/local/bin/kubectl apply -f /etc/istio-operator/crds; sleep 4; /manager --watch-created-resources-events=false --zap-time-encoding=iso8601
4026532616 pid 2 28721 root /bin/sh -c /usr/local/bin/kubectl apply -f /etc/istio-operator/crds; sleep 4; /manager --watch-created-resources-events=false --zap-time-encoding=iso8601
4026532617 mnt 1 29704 1000 /bin/prometheus-config-reloader --log-format=logfmt --reload-url=http://127.0.0.1:9090/-/reload --config-file=/etc/prometheus/config/prometheus.yaml.gz --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
4026532618 uts 1 29704 1000 /bin/prometheus-config-reloader --log-format=logfmt --reload-url=http://127.0.0.1:9090/-/reload --config-file=/etc/prometheus/config/prometheus.yaml.gz --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
4026532619 pid 1 29704 1000 /bin/prometheus-config-reloader --log-format=logfmt --reload-url=http://127.0.0.1:9090/-/reload --config-file=/etc/prometheus/config/prometheus.yaml.gz --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
4026532620 mnt 1 29759 1000 /configmap-reload --webhook-url=http://127.0.0.1:9090/-/reload --volume-dir=/etc/prometheus/rules/prometheus-ack-prometheus-operator-prometheus-rulefiles-0
4026532621 uts 1 29759 1000 /configmap-reload --webhook-url=http://127.0.0.1:9090/-/reload --volume-dir=/etc/prometheus/rules/prometheus-ack-prometheus-operator-prometheus-rulefiles-0
4026532622 pid 1 29759 1000 /configmap-reload --webhook-url=http://127.0.0.1:9090/-/reload --volume-dir=/etc/prometheus/rules/prometheus-ack-prometheus-operator-prometheus-rulefiles-0
4026532623 mnt 1 29838 1000 /bin/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --config.file=/etc/prometheus/config_out/prometheus.env.yaml --storage.tsdb.path=/prometheus --storage.tsdb.retention=10d --web.enable-lifecycle --storage.tsdb.no-lockfile --web.external-url=http://ack-prometheus-operator-prometheus.monitoring:9090 --web.route-prefix=/
4026532624 uts 1 29838 1000 /bin/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --config.file=/etc/prometheus/config_out/prometheus.env.yaml --storage.tsdb.path=/prometheus --storage.tsdb.retention=10d --web.enable-lifecycle --storage.tsdb.no-lockfile --web.external-url=http://ack-prometheus-operator-prometheus.monitoring:9090 --web.route-prefix=/
4026532625 pid 1 29838 1000 /bin/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries --config.file=/etc/prometheus/config_out/prometheus.env.yaml --storage.tsdb.path=/prometheus --storage.tsdb.retention=10d --web.enable-lifecycle --storage.tsdb.no-lockfile --web.external-url=http://ack-prometheus-operator-prometheus.monitoring:9090 --web.route-prefix=/
4026532628 mnt 1 18129 root /usr/local/openjdk-11/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start
4026532629 uts 1 18129 root /usr/local/openjdk-11/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start
4026532630 pid 1 18129 root /usr/local/openjdk-11/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start
4026533118 mnt 2 5799 root /bin/sh /acs/entrypoint.sh
4026533968 mnt 1 8792 root calico-felix
4026534201 ipc 5 2868 root sleep 3
4026534204 net 5 2868 root sleep 3
4026534452 mnt 1 10617 root /coredns -conf /etc/coredns/Corefile
4026534453 uts 1 10617 root /coredns -conf /etc/coredns/Corefile
4026534454 pid 1 10617 root /coredns -conf /etc/coredns/Corefile
4026534460 mnt 1 10835 root /alicloud-disk-controller
4026534461 uts 1 10835 root /alicloud-disk-controller
4026534462 pid 1 10835 root /alicloud-disk-controller
4026534463 mnt 1 10841 1000 /bin/alertmanager --config.file=/etc/alertmanager/config/alertmanager.yaml --cluster.listen-address=[172.20.12.202]:6783 --storage.path=/alertmanager --data.retention=120h --web.listen-address=:9093 --web.external-url=http://ack-prometheus-operator-alertmanager.monitoring:9093 --web.route-prefix=/ --cluster.peer=alertmanager-ack-prometheus-operator-alertmanager-0.alertmanager-operated.monitoring.svc:6783
4026534464 uts 1 10841 1000 /bin/alertmanager --config.file=/etc/alertmanager/config/alertmanager.yaml --cluster.listen-address=[172.20.12.202]:6783 --storage.path=/alertmanager --data.retention=120h --web.listen-address=:9093 --web.external-url=http://ack-prometheus-operator-alertmanager.monitoring:9093 --web.route-prefix=/ --cluster.peer=alertmanager-ack-prometheus-operator-alertmanager-0.alertmanager-operated.monitoring.svc:6783
4026534465 pid 1 10841 1000 /bin/alertmanager --config.file=/etc/alertmanager/config/alertmanager.yaml --cluster.listen-address=[172.20.12.202]:6783 --storage.path=/alertmanager --data.retention=120h --web.listen-address=:9093 --web.external-url=http://ack-prometheus-operator-alertmanager.monitoring:9093 --web.route-prefix=/ --cluster.peer=alertmanager-ack-prometheus-operator-alertmanager-0.alertmanager-operated.monitoring.svc:6783
4026534468 mnt 1 12874 472 python -u /app/sidecar.py
4026534469 uts 1 12874 472 python -u /app/sidecar.py
4026534470 pid 1 12874 472 python -u /app/sidecar.py
...ip netns为什么看不到namespace？
# ip netns list
我们跟踪一下ip netns的执行过程，看到有目录找不到
# strace -F -ff -t -tt -s 4096 -o ip.out ip netns list
1599206106.304532 execve("/usr/sbin/ip", ["ip", "netns", "list"], [/* 22 vars */]) = 0
1599206106.305867 brk(NULL) = 0x2592000
1599206106.305962 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f953edeb000
1599206106.306041 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
1599206106.306130 open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
1599206106.306209 fstat(3, {st_mode=S_IFREG|0644, st_size=21765, ...}) = 0
1599206106.306277 mmap(NULL, 21765, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f953ede5000
1599206106.306337 close(3) = 0
1599206106.306398 open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
1599206106.306456 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\r\0\0\0\0\0\0@\0\0\0\0\0\0\0\30C\0\0\0\0\0\0\0\0\0\0@\0008\0\7\0@\0!\0 \0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
0\0\0\0\0\0\0\4\37\0\0\0\0\0\0\4\37\0\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0X-\0\0\0\0\0\0X- \0\0\0\0\0X- \0\0\0\0\0\20\3\0\0\0\0\0\0\270\3\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\210-\0\0\0\0\0\0\
210- \0\0\0\0\0\210- \0\0\0\0\0\20\2\0\0\0\0\0\0\20\2\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\310\1\0\0\0\0\0\0\310\1\0\0\0\0\0\0\310\1\0\0\0\0\0\0D\0\0\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0P\34
5td\4\0\0\0@\31\0\0\0\0\0\0@\31\0\0\0\0\0\0@\31\0\0\0\0\0\0\274\0\0\0\0\0\0\0\274\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0X-\0\0\0\0\0\0X- \0\0\0\0\0X- \0\0\0\0\0\250\2\0\0\0\0\0\0\250\2\0\0\0\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\24\0\0\0\3\0\0\0GNU\0005v\223\310\361\364\235\223\1\fN1R\234\7\31
5\322\275=\10\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\2\0\0\0\6\0\0\0 \0\0\0\0\0\0\0\33\0\0\0\32\0\0\0\2\0\0\0\7\0\0\0\230\2\21\0\200H\0\4\22\0\0@\203(\10\236\32\0\0\0\0\0\0\0\33\0\0\0\0\0\0\0\0\0\0\0\34\
0\0\0\0\0\0\0\35\0\0\0\0\0\0\0\36\0\0\0\0\0\0\0\37\0\0\0\0\0\0\0 \0\0\0\"\0\0\0#\0\0\0%\0\0\0&\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0'\0\0\0\0\0\0\0\0\0\0\0\353\26\251\30a\257\0\371\301S\
200\30\273\25sB\257\304M\17\221!\374\370\6\2\4\371\3733\373\17\371\31sB\372\31sB\225\263_\31\177\236\320\30a\242\222\6\5\350\7\371\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0=\1\0\0\22\0\0\0
\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\375\0\0\0\22\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\337\0\0\0 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0 \0\0\0", 832) = 832
1599206106.306534 fstat(3, {st_mode=S_IFREG|0755, st_size=19288, ...}) = 0
1599206106.306593 mmap(NULL, 2109712, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f953e9c7000
1599206106.306647 mprotect(0x7f953e9c9000, 2097152, PROT_NONE) = 0
1599206106.306706 mmap(0x7f953ebc9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f953ebc9000
1599206106.306771 close(3) = 0
1599206106.306827 open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
1599206106.306883 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340$\2\0\0\0\0\0@\0\0\0\0\0\0\0\370\301 \0\0\0\0\0\0\0\0\0@\0008\0\n\0@\0L\0K\0\6\0\0\0\5\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0
\0\0\0\0\0\0000\2\0\0\0\0\0\0000\2\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0\220\332\30\0\0\0\0\0\220\332\30\0\0\0\0\0\220\332\30\0\0\0\0\0\34\0\0\0\0\0\0\0\34\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0\1\0\0\0\
5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\250\35\34\0\0\0\0\0\250\35\34\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0 '\34\0\0\0\0\0 '<\0\0\0\0\0 '<\0\0\0\0\0\200Q\0\0\0\0\0\0\300\232\0\0\0\0\0
\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\200[\34\0\0\0\0\0\200[<\0\0\0\0\0\200[<\0\0\0\0\0\360\1\0\0\0\0\0\0\360\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0D\0\0
\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\7\0\0\0\4\0\0\0 '\34\0\0\0\0\0 '<\0\0\0\0\0 '<\0\0\0\0\0\20\0\0\0\0\0\0\0\240\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0P\345td\4\0\0\0\254\332\30\0\0\0\0\0\254\332\30\0\0\
0\0\0\254\332\30\0\0\0\0\0\214j\0\0\0\0\0\0\214j\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\
0 '\34\0\0\0\0\0 '<\0\0\0\0\0 '<\0\0\0\0\0\3408\0\0\0\0\0\0\3408\0\0\0\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\246D\350e\241\366r\264j\0222\367\302u\365;\6\4&\226\4\0\0\0\20\0\0\0\1\0\0\0GNU
\0\0\0\0\0\2\0\0\0\6\0\0\0 \0\0\0\0\0\0\0\363\3\0\0\7\0\0\0\0\1\0\0\16\0\0\0\0000\20D\240 \2\1\210\3\346\220\305E\214\0\300\0\10\0\5\200\0`\300\200\0\r\212\f\0\4\20\0\210D2\10.@\210T<, \0162H&\204\300\2
14\4\10\0\2\2\16\241\254\32\4f\300\0\3002\0\300\0P\1 \201\10\204\v ($\0\4 Z\0\20X\200\312DB(\0\6\200\20\30B\0 @\200\0\tP\0Q\212@\20\0\0\0\0\10\0\0\21\20", 832) = 832
1599206106.306950 fstat(3, {st_mode=S_IFREG|0755, st_size=2151672, ...}) = 0
1599206106.307013 mmap(NULL, 3981792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f953e5fa000
1599206106.307066 mprotect(0x7f953e7bc000, 2097152, PROT_NONE) = 0
1599206106.307120 mmap(0x7f953e9bc000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c2000) = 0x7f953e9bc000
1599206106.307205 mmap(0x7f953e9c2000, 16864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f953e9c2000
1599206106.307260 close(3) = 0
1599206106.307312 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f953ede4000
1599206106.307381 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f953ede2000
1599206106.307438 arch_prctl(ARCH_SET_FS, 0x7f953ede2740) = 0
1599206106.307591 mprotect(0x7f953e9bc000, 16384, PROT_READ) = 0
1599206106.307648 mprotect(0x7f953ebc9000, 4096, PROT_READ) = 0
1599206106.307714 mprotect(0x669000, 4096, PROT_READ) = 0
1599206106.307789 mprotect(0x7f953edec000, 4096, PROT_READ) = 0
1599206106.307863 munmap(0x7f953ede5000, 21765) = 0
1599206106.307978 socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, NETLINK_ROUTE) = 3
1599206106.308052 setsockopt(3, SOL_SOCKET, SO_SNDBUF, [32768], 4) = 0
1599206106.308108 setsockopt(3, SOL_SOCKET, SO_RCVBUF, [1048576], 4) = 0
1599206106.308162 setsockopt(3, SOL_NETLINK, 11, [1], 4) = -1 ENOPROTOOPT (Protocol not available)
1599206106.308221 bind(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
1599206106.308284 getsockname(3, {sa_family=AF_NETLINK, pid=3965, groups=00000000}, [12]) = 0
1599206106.308367 open("/proc/self/ns/net", O_RDONLY) = 4
1599206106.308453 sendto(3, "\34\0\0\0Z\0\1\0\0\0\0\0\0\0\0\0\0\0\0\0\10\0\3\0\4\0\0\0", 28, 0, NULL, 0) = 28
1599206106.308530 recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\34\0\0\0X\0\0\0\0\0\0\0}\17\0\0\0\0\0\0\10\0\1\0\377\377\377\377", 16384}], msg_controllen=0, ms
g_flags=0}, 0) = 28
1599206106.308604 close(4) = 0
1599206106.308658 socket(AF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, NETLINK_ROUTE) = 4
1599206106.308708 setsockopt(4, SOL_SOCKET, SO_SNDBUF, [32768], 4) = 0
1599206106.308754 setsockopt(4, SOL_SOCKET, SO_RCVBUF, [1048576], 4) = 0
1599206106.308800 setsockopt(4, SOL_NETLINK, 11, [1], 4) = -1 ENOPROTOOPT (Protocol not available)
1599206106.308847 bind(4, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
1599206106.308895 getsockname(4, {sa_family=AF_NETLINK, pid=-22296596, groups=00000000}, [12]) = 0
1599206106.308952 openat(AT_FDCWD, "/var/run/netns", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1599206106.309017 openat(AT_FDCWD, "/var/run/netns", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
1599206106.309075 exit_group(0) = ?
1599206106.309181 +++ exited with 0 +++
默认 Docker 创建的网络 namespace 不在ip netns读取的默认路径下（/var/run/netns） ，因此ip netns list直接看不到，需要 ln 软链接一下。链接完毕以后，我们就可以通过 ip netns 命令操作了。
# docker ps |grep f9bc06a148c9
f9bc06a148c9 15bd4d827695 "/alibabacloud-log-c…" 6 weeks ago Up 6 weeks k8s_alibaba-log-controller_alibaba-log-controller-74d85bbf76-tzth9_kube-system_828b171d-9915-11ea-831b-e672548dc4da_1
# docker inspect '--format={{ .State.Pid }}' f9bc06a148c9
12428
# nsenter -t 12428 -n
# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.20.12.223 netmask 255.255.255.255 broadcast 172.20.12.223
ether fa:ca:54:f6:59:69 txqueuelen 0 (Ethernet)
RX packets 426522 bytes 54197077 (51.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 435689 bytes 57149830 (54.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
软链给ip netns读取的目录
# mkdir /var/run/netns
# ln -s /proc/12428/ns/net /var/run/netns/12428
# ip netns list
12428 (id: 23)
# ip netns exec 12428 ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.20.12.223 netmask 255.255.255.255 broadcast 172.20.12.223
ether fa:ca:54:f6:59:69 txqueuelen 0 (Ethernet)
RX packets 426497 bytes 54193478 (51.6 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 435664 bytes 57146029 (54.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1000 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0那么问题来了：
默认lsns为什么可以找到进程的namespace，ip netns为什么不行？