九、docker的跨主机通讯之overlay（vxlan）

2023-10-03 10:32:40

在做这个实验前，需要用到consul，用于实现服务发现和DNS的功能，consul的安装相对简单，其详情功能介绍可查看官网：https://www.cnblogs.com/xiaohanlin/p/8016803.html，https://www.consul.io/docs/architecture

docker run -it -p 8500:8500 -h consul --name consule --restart=always progrium/consul -server --bootstrap　　#启动consul容器，并配置为server端

[root@docker-1-26 ~]# cat /etc/docker/daemon.json
{
  "hosts":["tcp://0.0.0.0:2376","unix:///var/run/docker.sock"],
  "cluster-store":"consul://10.0.0.26:8500",　　　　#向console进行注册
  "cluster-advertise":"10.0.0.26:2376"
}

[root@docker-2-27 ~]# cat /etc/docker/daemon.json
{
  "hosts":["tcp://0.0.0.0:2376","unix:///var/run/docker.sock"],
  "cluster-store":"consul://10.0.0.26:8500",
  "cluster-advertise":"10.0.0.27:2376"
}

环境搭建完成后，可登陆consul界面查看到共三个node，到这一步，测试要用的环境就准备好了

创建overlay类型的docker网络

[root@docker-1-26 ~]# docker network create -d overlay --subnet 20.0.0.0/24 --gateway 20.0.0.254 overlay_20201209
6dbac2cd67bc3cfcffb67f607a5719355570332ca45a8ab223710a1372ce6de2
[root@docker-1-26 ~]# docker network ls 
NETWORK ID          NAME                DRIVER              SCOPE
5ca0e6f8c027        bridge              bridge              local
2bb25566094c        host                host                local
640d652cd920        macvlan_20201206    macvlan             local
66221610ad40        none                null                local
6dbac2cd67bc        overlay_20201209    overlay             global　　#因为搭建了集群，创建的网络类型为overlay，scope为global，那集群内的其他节点都应该会有相应的网络

[root@docker-3-28 ~]# docker network ls 
NETWORK ID          NAME                DRIVER              SCOPE
0a4efb4f841d        bridge              bridge              local
2bb25566094c        host                host                local
a22940610925        macvlan_20201206    macvlan             local
66221610ad40        none                null                local
6dbac2cd67bc        overlay_20201209    overlay             global

现在三个节点分别创建容器：

[root@docker-2-27 ~]# docker run -it --network overlay_20201209 --name busybox_2 busybox:latest sh

[root@docker-3-28 ~]# docker run -it --network overlay_20201209 --name busybox_3 busybox:latest sh
/ # ip add 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
6: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue 
    link/ether 02:42:14:00:00:03 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.3/24 brd 20.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
9: eth1@if10: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1
       valid_lft forever preferred_lft forever
/ #

这个时候容器间跨容器通讯应该是正常的，实验过程中发现无法通讯，查看messages发现报错：

Dec 10 04:56:02 docker-3-28 dockerd: time="2020-12-10T04:56:02.630234941+08:00" level=error msg="2020/12/10 04:56:02 [ERR] memberlist: Push/Pull with docker-2-27 failed: dial tcp 10.0.0.27:7946: connect: no route to host\n"　　
#连接成员的7946端口有失败，检查端口状态正常
[root@docker-3-28 ~]# netstat -tnlup | grep 7946
tcp        0      0 10.0.0.28:7946          0.0.0.0:*               LISTEN      2153/dockerd        
udp        0      0 10.0.0.28:7946          0.0.0.0:*                           2153/dockerd

[root@docker-3-28 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2020-12-10 04:19:56 CST; 6min ago　　#防火墙状态开启的，关闭后容器可实现跨集群通讯

Dec 9 21:01:01 docker-1-26 systemd: Started Session 2 of user root.
Dec 9 21:01:01 docker-1-26 systemd: Starting Session 2 of user root.

/ # ping -c 4 20.0.0.3　　　　#实现了容器跨节点通讯
PING 20.0.0.3 (20.0.0.3): 56 data bytes
64 bytes from 20.0.0.3: seq=0 ttl=64 time=0.941 ms
64 bytes from 20.0.0.3: seq=1 ttl=64 time=1.355 ms
64 bytes from 20.0.0.3: seq=2 ttl=64 time=0.587 ms
64 bytes from 20.0.0.3: seq=3 ttl=64 time=0.644 ms

--- 20.0.0.3 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.587/0.881/1.355 ms
/ # ping -c 4 20.0.0.2
PING 20.0.0.2 (20.0.0.2): 56 data bytes
64 bytes from 20.0.0.2: seq=0 ttl=64 time=2.023 ms
64 bytes from 20.0.0.2: seq=1 ttl=64 time=0.674 ms
64 bytes from 20.0.0.2: seq=2 ttl=64 time=0.744 ms
64 bytes from 20.0.0.2: seq=3 ttl=64 time=0.616 ms

--- 20.0.0.2 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.616/1.014/2.023 ms

/ # ping www.baidu.com　　　　　　　　　　　　@容器内可以外网通讯
PING www.baidu.com (110.242.68.4): 56 data bytes
64 bytes from 110.242.68.4: seq=0 ttl=127 time=78.321 ms
64 bytes from 110.242.68.4: seq=1 ttl=127 time=84.424 ms
^C
--- www.baidu.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 78.321/81.372/84.424 ms

在overlay网络中，容器想对外提供服务的话，需要在创建容器的过程中配置端口映射

容器跨节点通讯时的网络原理学习，这里需要注意的是，容器间的隔离是通过namespace实现，但是通过ip netns只能查看到/var/run/netns下的内容，而docker网络中的net namespace是在/var/run/docker/netns，此处需要配置软链接实现ns查看。

[root@docker-1-26 netns]# ln -s /var/run/docker/netns/ /var/run/netns

[root@docker-1-26 ~]# ip netns
e1266b49bcf0 (id: 2)
2-d3f0d488f3 (id: 1)　　　　#node1与node2具有一个相同的net namespace
82a8fe4607bc (id: 0)
e9c0045a3c42
5b2dfc44066a
1bbd1db96893

[root@docker-2-27 ~]# ip netns
34db003b2f09 (id: 2)
1-d3f0d488f3 (id: 1)
8dbb8ea87eb6 (id: 0)
f6ed7fa0e3dd
6422b842d486

[root@docker-1-26 ~]# ip netns exec 2-d3f0d488f3 /bin/bash　　#查看网络命名空间
[root@docker-1-26 ~]# ifconfig

[root@docker-1-26 ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP
link/ether 26:01:b8:b0:dd:37 brd ff:ff:ff:ff:ff:ff
inet 20.0.0.254/24 brd 20.0.0.255 scope global br0
valid_lft forever preferred_lft forever
19: vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br0 state UNKNOWN
link/ether 26:01:b8:b0:dd:37 brd ff:ff:ff:ff:ff:ff link-netnsid 0　　#正式因为vxlan网卡属于netnsid0,netns id 0是桥接网络，所以能够在ens间构建vxlan隧道。
21: veth0@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master br0 state UP
link/ether d6:0e:6e:a1:17:dd brd ff:ff:ff:ff:ff:ff link-netnsid 1

容器跨节点通讯的原理图，其实vxlan隧道是构建于node节点 ens网卡间

最后是附上两个节点的IP信息

[root@docker-2-27 ~]# docker ps -l
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
415eb4c90bae        busybox:latest      "sh"                22 minutes ago      Up 22 minutes                           busybox_2
[root@docker-2-27 ~]# docker exec -it 415 sh
/ # ip add 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
11: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue 
    link/ether 02:42:14:00:00:01 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.1/24 brd 20.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
13: eth1@if14: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1
       valid_lft forever preferred_lft forever
/ # read escape sequence
[root@docker-2-27 ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:a8:31:57 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.27/24 brd 10.0.0.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fea8:3157/64 scope link 
       valid_lft forever preferred_lft forever
3: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:6b:a0:35:93 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global docker_gwbridge
       valid_lft forever preferred_lft forever
    inet6 fe80::42:6bff:fea0:3593/64 scope link 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:e8:91:f8:d1 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:e8ff:fe91:f8d1/64 scope link 
       valid_lft forever preferred_lft forever
14: veth8d7f1de@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP 
    link/ether be:9a:e7:28:16:60 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::bc9a:e7ff:fe28:1660/64 scope link 
       valid_lft forever preferred_lft forever
[root@docker-2-27 ~]#

10.0.0.26
[root@docker-1-26 ~]# docker ps 
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                                            NAMES
aa4941dd127a        busybox             "sh"                     15 minutes ago      Up 14 minutes                                                                                        busybox_1
4b8357352d40        progrium/consul     "/bin/start -server …"   23 minutes ago      Up 22 minutes       53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp   consule
[root@docker-1-26 ~]# docker exec -it aa4941dd127a sh
/ # ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
15: eth0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue 
    link/ether 02:42:14:00:00:02 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.2/24 brd 20.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
17: eth1@if18: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1
       valid_lft forever preferred_lft forever
/ # read escape sequence
[root@docker-1-26 ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:dc:55:5d brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.26/24 brd 10.0.0.255 scope global ens32
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fedc:555d/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:75:ef:c0:4a brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:75ff:feef:c04a/64 scope link 
       valid_lft forever preferred_lft forever
4: docker_gwbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:40:35:5d:89 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global docker_gwbridge
       valid_lft forever preferred_lft forever
    inet6 fe80::42:40ff:fe35:5d89/64 scope link 
       valid_lft forever preferred_lft forever
12: vethcfbfeaa@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP 
    link/ether 36:29:24:4b:b5:56 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::3429:24ff:fe4b:b556/64 scope link 
       valid_lft forever preferred_lft forever
18: veth5239bf9@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker_gwbridge state UP 
    link/ether 8a:08:e0:f8:ec:82 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::8808:e0ff:fef8:ec82/64 scope link 
       valid_lft forever preferred_lft forever
[root@docker-1-26 ~]#

View Code

码农公寓

相关文章