单主机多个bridge网络间通信

2023-10-13 22:31:58

实验如图所示：

实验目的：

将nginx的端口映射到宿主机，通过宿主机来访问到nginx，然后nginx的php设置将会寻找php容器所在，最终显示页面

创建bridge网卡

首先my_net和myapp_net是需要创建的两块bridge网卡

创建my_net网卡并指定网段为172.100.100.0/24

[root@localhost ~]# docker network create -d bridge --subnet 172.100.100.0/24 --gateway 172.100.100.1 my_net
c20026bdb8eaa6e8ad31f0af7f8ca8e1bc03a34611f72a0e4e7b759e909b613d

创建myapp_net网卡自动分配ip网段

[root@localhost ~]# docker network create -d bridge myapp_net
e9f6ca9fb740237679930c92ed35f5b59999305a4cf9c26bbac009a9573186ea

验证创建，图中所示的三块bridge网卡已经有了

[root@localhost ~]# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
e1aa5d692686        bridge              bridge              local
082e568090f3        host                host                local
c20026bdb8ea        my_net              bridge              local
e9f6ca9fb740        myapp_net           bridge              local
70d9c743f011        none                null                local

查看分配好的网卡及网段，br编号也可以和查看到的bridge网卡对应

my_net：172.100.100.1/24 myapp_net：172.18.0.1/24

[root@localhost ~]#  ip a
...
6: br-c20026bdb8ea: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:64:24:7e:29 brd ff:ff:ff:ff:ff:ff
    inet 172.100.100.1/24 brd 172.100.100.255 scope global br-c20026bdb8ea
       valid_lft forever preferred_lft forever
7: br-e9f6ca9fb740: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:fd:30:6a:d9 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-e9f6ca9fb740
       valid_lft forever preferred_lft forever
...

bbox2和bbox3通过my_net通信

容器名	ip	网卡接口	veth
bbox2	172.100.100.2	8@9	veth905e259
bbox3	172.100.100.3	10@11	veth2fef0c8

创建并运行容器bbbox2

[root@localhost ~]# docker  run -itd --name bbox2 --network my_net busybox
a78ab2796fa04c77c03ae8f42b37ea5a18cb0439ec0ea7d656a41a6c32f719fd
[root@localhost ~]# docker exec -it bbox2 /bin/sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
8: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:64:64:02 brd ff:ff:ff:ff:ff:ff
    inet 172.100.100.2/24 brd 172.100.100.255 scope global eth0
       valid_lft forever preferred_lft forever

查看绑定后的veth网卡编号

[root@localhost ~]# ip a
...
9: veth905e259@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c20026bdb8ea state UP group default 
    link/ether b2:41:c2:21:4b:28 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::b041:c2ff:fe21:4b28/64 scope link 
       valid_lft forever preferred_lft forever

创建并运行容器bbox3

[root@localhost ~]# docker run -itd --name bbox3 --network my_net busybox
55384a5e5eb3a8330ff44ed4f68bc1521d56dc474ebbfcc520628cb82679f621
[root@localhost ~]# docker exec -it bbox3 /bin/sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
10: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:64:64:03 brd ff:ff:ff:ff:ff:ff
    inet 172.100.100.3/24 brd 172.100.100.255 scope global eth0
       valid_lft forever preferred_lft forever

查看绑定后的veth网卡编号

[root@localhost ~]# ip a
...
11: veth2fef0c8@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c20026bdb8ea state UP group default 
    link/ether 12:54:25:ff:5f:e2 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::1054:25ff:feff:5fe2/64 scope link 
       valid_lft forever preferred_lft forever

查看两个容器的bridge网卡桥接情况，两个veth网卡已经桥接在了br-86b6623a9b9f网卡上

[root@localhost ~]# brctl show
bridge name    bridge id        STP enabled    interfaces
br-c20026bdb8ea        8000.024264247e29    no        veth2fef0c8
                            veth905e259
br-e9f6ca9fb740        8000.0242fd306ad9    no        
docker0        8000.024231921c1e    no        
virbr0        8000.525400fa47fa    yes        virbr0-nic

验证连通性

[root@localhost ~]# docker exec -it bbox2 /bin/sh
/ # ping 172.100.100.3 
PING 172.100.100.3 (172.100.100.3): 56 data bytes
64 bytes from 172.100.100.3: seq=0 ttl=64 time=0.102 ms
64 bytes from 172.100.100.3: seq=1 ttl=64 time=0.118 ms
^C
--- 172.100.100.3 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.102/0.110/0.118 ms

nginx容器连接三块bridge网卡

容器名	ip	网卡接口	veth
nginx/docker0	172.17.0.3	30@31	vethca1d1aa
nginx/my_net	172.100.100.4	36@37	veth581fcff
nginx/myapp_net	172.18.0.2	38@39	vethe6f1a56

构建nginx镜像

如图中架构所示，nginx容器，需要连接三块网卡docker0、my_net、myapp_net，先使用docker0将nginx搭建好

用dockerfile构建nginx镜像，先拖入nginx安装包

[root@localhost ~]# vim Dockerfile
FROM centos
MAINTAINER FeiYi
RUN yum -y install net-tools iproute pcre-devel openssl-devel gcc gcc-c++ make zlib-devel elinks
ADD nginx-1.11.1.tar.gz /usr/src
ENV NGINX_DIR /usr/src/nginx-1.11.1
WORKDIR $NGINX_DIR
RUN ./configure --prefix=/usr/local/nginx --user=nginx --group=nginx && make && make install
WORKDIR /
RUN useradd nginx
RUN ln -s /usr/local/nginx/sbin/nginx /usr/sbin/nginx
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

构建nginx镜像

[root@localhost ~]# docker build -t nginx /root
Successfully built 658d57927dd4
Successfully tagged nginx:latest
[root@localhost ~]# docker images
REPOSITORY    TAG          IMAGE ID            CREATED             SIZE
nginx         latest       658d57927dd4        16 seconds ago      423MB
busybox       latest       83aa35aa1c79        3 weeks ago         1.22MB
httpd         latest       c5a012f9cf45        4 weeks ago         165MB
centos        latest       5e35e350aded        4 months ago        203MB

启动运行nginx容器

这里要做端口映射，因为主机的80端口没有使用，所以就直接使用主机的80端口，方便访问

[root@localhost ~]# docker run -itd -p 80:80 --name nginx nginx
18f479e8dff2aba7875e4b7c060cc0543a794a5607523adb8f15096b436d1ad4

查看nginx容器获取到的docker0网卡的ip

[root@localhost ~]# docker exec -it nginx /bin/bash
[root@18f479e8dff2 /]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
30: eth0@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

查看nginx容器与物理机的网卡绑定

[root@localhost ~]# ip a
...
31: vethca1d1aa@if30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default 
    link/ether 2e:99:37:55:ad:7d brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::2c99:37ff:fe55:ad7d/64 scope link 
       valid_lft forever preferred_lft forever

nginx容器连接网卡my_net和myapp_net

nginx容器绑定my_net网卡和myapp_net网卡

[root@localhost ~]# docker network connect my_net nginx
[root@localhost ~]# docker network connect myapp_net nginx

绑定完之后，nginx中应该有三个网段的ip地址，进入容器查看

[root@localhost ~]# docker exec -it nginx /bin/bash
[root@18f479e8dff2 /]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
30: eth0@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
36: eth1@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:64:64:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.100.100.4/24 brd 172.100.100.255 scope global eth1
       valid_lft forever preferred_lft forever
38: eth2@if39: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth2
       valid_lft forever preferred_lft forever

在查看与物理机绑定的veth的对应

[root@localhost ~]# ip a
...
37: vetha9d59ba@if36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c20026bdb8ea state UP group default 
    link/ether 42:00:4e:37:fc:1e brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::4000:4eff:fe37:fc1e/64 scope link 
       valid_lft forever preferred_lft forever
39: veth581fcff@if38: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-e9f6ca9fb740 state UP group default 
    link/ether 52:bf:2c:79:4b:9e brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::50bf:2cff:fe79:4b9e/64 scope link 
       valid_lft forever preferred_lft forever

bbox1监控nginx的流量

bbox1使用joined网络和nginx公用网络配置

[root@localhost ~]# docker run -itd --name bbox1 --network container:nginx busybox
86a43eaa60902dde2a1252de7f8433758be4aa871fc0ed4779ff16c491c23878
[root@localhost ~]# docker exec -it bbox1 /bin/sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
30: eth0@if31: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
36: eth1@if37: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:64:64:04 brd ff:ff:ff:ff:ff:ff
    inet 172.100.100.4/24 brd 172.100.100.255 scope global eth1
       valid_lft forever preferred_lft forever
38: eth2@if39: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth2
       valid_lft forever preferred_lft forever

构建PHP镜像

将包拖入创建的php目录中

php-5.3.28.tar.gz

编写Dockerfile

[root@localhost ~]# mkdir php
[root@localhost ~]# cd php/
[root@localhost php]# vim Dockerfile
FROM centos
MAINTAINER FeiYi
ENV PHP_INSTALL_DIR /usr/src/php-5.3.28
ENV PHP_DIR /usr/local/php
ENV PHP_SH /etc/init.d/php-fpm
ADD php-5.3.28.tar.gz /usr/src
RUN yum -y install net-tools coreutils chkconfig iproute sed \
pcre-devel ncurses-devel openssl-devel zlib-devel autoconf \
libjpeg-devel libxml2-devel libpng-devel gd gcc gcc-c++ make perl perl-devel
WORKDIR /
WORKDIR $PHP_INSTALL_DIR
RUN ./configure --prefix=$PHP_DIR --with-gd --with-zlib \
--with-config-file-path=$PHP_DIR \
--enable-fpm --enable-mbstring --with-jpeg-dir=/usr/lib && make && make install
WORKDIR /
RUN cp $PHP_INSTALL_DIR/php.ini-development $PHP_DIR/php.ini
RUN sed -i '/default_charset/c \default_charset = "utf-8"' $PHP_DIR/php.ini
RUN sed -i '/short_open_tag/c \short_open_tag = On' $PHP_DIR/php.ini
RUN cp $PHP_INSTALL_DIR/sapi/fpm/init.d.php-fpm $PHP_SH
RUN chmod +x $PHP_SH && chkconfig --add php-fpm
RUN cp $PHP_DIR/etc/php-fpm.conf.default $PHP_DIR/etc/php-fpm.conf
RUN sed -i '/;pid = run/c \pid = run/php-fpm.pid' $PHP_DIR/etc/php-fpm.conf && \
sed -i '/user =/c \user = nginx' $PHP_DIR/etc/php-fpm.conf && \
sed -i '/group =/c \group = nginx' $PHP_DIR/etc/php-fpm.conf && \
sed -i '/pm.max_children/c \pm.max_children = 50' $PHP_DIR/etc/php-fpm.conf && \
sed -i '/pm.start_servers/c \pm.start_servers = 20' $PHP_DIR/etc/php-fpm.conf && \
sed -i '/pm.min_spare_servers/c \pm.min_spare_servers = 5' $PHP_DIR/etc/php-fpm.conf && \
sed -i '/pm.max_spare_servers/c \pm.max_spare_servers = 35' $PHP_DIR/etc/php-fpm.conf
EXPOSE 80 3306 9000
CMD ["bin/bash"]

文件中并没有去启动服务以及整合nginx和php之间的关系

开始构建

[root@localhost ~]# docker build -t php /root/php
Successfully built 7a683c08335f
Successfully tagged php:latest

使用myapp_net启动并运行容器

容器名	ip	网卡接口	veth
php/myapp_net	172.18.0.3	56@57	vethb0a948a

[root@localhost ~]# docker run -itd --name php --network myapp_net php
9cbfe998c72d80cbdc5bec74b4ff32dcb24bbda32615a759156b47f724a1ea9a

进入容器查看ip和桥接网卡

[root@localhost ~]# docker exec -it php /bin/bash
[root@9cbfe998c72d /]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
56: eth0@if57: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.3/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever

查看物理机与php的桥接网卡

[root@localhost ~]# ip a
...
57: vethb0a948a@if56: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-e9f6ca9fb740 state UP group default 
    link/ether 5e:d5:3f:ca:15:1e brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::5cd5:3fff:feca:151e/64 scope link 
       valid_lft forever preferred_lft forever

验证与nginx的连通性

[root@localhost ~]# docker exec -it php /bin/bash
[root@2466ba91f03e /]# ping 172.18.0.2
PING 172.18.0.2 (172.18.0.2) 56(84) bytes of data.
64 bytes from 172.18.0.2: icmp_seq=1 ttl=64 time=0.072 ms
^C
--- 172.18.0.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.072/0.072/0.072/0.000 ms

整合nginx和php

因为之前启动nginx容器并没有进行端口映射，先将nginx容器删除

nginx容器中

[root@localhost ~]# docker exec -it nginx /bin/bash
[root@18f479e8dff2 /]# vi /usr/local/nginx/conf/nginx.conf
# 找到以下内容，按照注释修改
        location / {
            root   html;
            index  index.html index.htm index.php; # 添加index.php
        }
        location ~ \.php$ {   # 可以解开注释，也可以自行添加
            root           /www/;  # 该目录是php容器中的目录
            fastcgi_pass   172.18.0.3:9000;            //php容器地址
            fastcgi_index  index.php;
            fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
            include        fastcgi.conf;
         }
[root@18f479e8dff2 /]# nginx -s reload

php容器中

[root@localhost ~]# docker exec -it php /bin/bash
[root@9cbfe998c72d /]# useradd nginx
[root@9cbfe998c72d /]# mkdir /www
[root@9cbfe998c72d /]# vi /www/index.php
<?
phpinfo();
?>
[root@9cbfe998c72d /]# vi /usr/local/php/etc/php-fpm.conf
listen = 172.18.0.3:9000  # php容器ip
[root@9cbfe998c72d /]# /etc/init.d/php-fpm start

验证

使用物理机直接访问本机ip

[root@localhost ~]# elinks 192.168.1.11/index.php

也可以通过windows宿主机访问

码农公寓