ovs实践1:基础概念和gre隧道实践操作

2024-03-23 14:39:34

1.
测试环境
两个虚机:nginx-1和nginx-3
centos7 系统内核3.10.0-514.el7.x86_64
ovs_version: "2.0.0"

2.
注意事项
关闭selinux
关闭NetworkManager(必须关闭,不然出莫名其妙的网络故障)
关闭firewalld

3.
安装openvswitch(可编译安装,这里使用yum安装,简单测试)

yum install openvswitch openvswitch-devel openvswitch-test openvswitch-debuginfor
systemctl enable  openvswitch
systemctl start  openvswitch

4.
创建桥前的网络情况

创建桥前

[root@nginx-1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
      valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
      valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:80:20:b2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.32.139/24 brd 192.168.32.255 scope global dynamic ens33
      valid_lft 1721sec preferred_lft 1721sec
    inet6 fe80::6d0c:96f7:cbae:c576/64 scope link
      valid_lft forever preferred_lft forever
[root@nginx-1 ~]#
[root@nginx-1 ~]# ip r
default via 192.168.32.2 dev ens33  proto static  metric 100
192.168.32.0/24 dev ens33  proto kernel  scope link  src 192.168.32.132  metric 100
[root@nginx-1 ~]#

5.
ovs基础知识,参考文档:https://opengers.github.io/openstack/openstack-base-use-openvswitch/
port类型说明

normal类型
操作系统中已有的网卡(物理网卡或者虚拟机中的虚拟网卡)连接到ovs桥上,
ovs会生成一个port处理这块网卡的进出数据包.这个端口类型为normal
挂载到OVS上的网卡设备不支持分配IP地址

internal
internal端口是ovs内部创建的虚拟网卡接口
每创建一个Port,OVS会自动创建一个同名接口(Interface)挂载到新创建的Port上
interface可用来配置ip

patch
多个ovs网桥可用patch port连接,类似于veth pair.
从一个Patch Port收到的数据包会被转发到另一个Patch Port
patch port总是成对出现,分别连接在两个网桥上.使用Patch连接的两个网桥跟一个网桥没什么区别

tunnel
OVS中支持添加隧道(Tunnel)端口,常见隧道技术有两种gre或vxlan.
网络之上构建一层虚拟网络,上层应用只与虚拟网络相关.

Interface
nterface是连接到Port的网络接口设备,是OVS与外部交换数据包的组件,在通常情况下,
Port和Interface是一对一的关系,只有在配置Port为 bond模式后,Port和Interface是一对多的关系.

6.
实际操作

虚机1创建桥

[root@nginx-1 ~]# ovs-vsctl add-br br0
[root@nginx-1 ~]#

[root@nginx-1 ~]# ovs-vsctl show
67ccf09b-d0d9-4ccb-9f2f-1ac7918e5c46
    Bridge "br0"
        Port "br0"
            Interface "br0"
                type: internal
    ovs_version: "2.0.0"
[root@nginx-1 ~]#

虚机2创建桥

[root@nginx-3 ~]# ovs-vsctl add-br br0
[root@nginx-3 ~]#

[root@nginx-3 ~]# ovs-vsctl show
25cdb171-59ee-4eb4-a8af-cd93dc460926
    Bridge "br0"
        Port "br0"
            Interface "br0"
                type: internal
    ovs_version: "2.0.0"

桥默认有一个br0端口和br0接口,type类型为internel
internal端口是ovs内部创建的虚拟网卡接口
每创建一个桥自动创建个类型为internal的和桥一样名字的port和interface.

7.
测试思路

虚机1和虚机2的br0各定义同网段ip,测试互通.
虚机1配置br0为192.168.100.2/24
虚机2配置br0为192.168.100.3/24

临时配置命令,见下:

[root@nginx-1 ~]# ifconfig br0 192.168.100.2/24 up

[root@nginx-3 ~]# ifconfig br0 192.168.100.3/24 up

在没有建立隧道前,网络是不通的.
虚机1

[root@nginx-1 ~]# ip a |grep br0
5: br0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    inet 192.168.100.2/24 brd 192.168.100.255 scope global br0
[root@nginx-1 ~]# ip r
default via 192.168.32.2 dev ens33
169.254.0.0/16 dev ens33  scope link  metric 1002
192.168.32.0/24 dev ens33  proto kernel  scope link  src 192.168.32.132
192.168.100.0/24 dev br0  proto kernel  scope link  src 192.168.100.2
[root@nginx-1 ~]# ping 192.168.100.2
PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.
64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from 192.168.100.2: icmp_seq=2 ttl=64 time=0.167 ms
^C
--- 192.168.100.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1042ms
rtt min/avg/max/mdev = 0.045/0.106/0.167/0.061 ms
[root@nginx-1 ~]# ping 192.168.100.3
PING 192.168.100.3 (192.168.100.3) 56(84) bytes of data.
From 192.168.100.2 icmp_seq=1 Destination Host Unreachable
From 192.168.100.2 icmp_seq=2 Destination Host Unreachable
From 192.168.100.2 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.100.3 ping statistics ---
6 packets transmitted, 0 received, +3 errors, 100% packet loss, time 5138ms
pipe 4
[root@nginx-1 ~]#

虚机2

[root@nginx-3 ~]# ip a|grep br0
5: br0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    inet 192.168.100.3/24 brd 192.168.100.255 scope global br0
[root@nginx-3 ~]# ip r
default via 192.168.32.2 dev ens33
169.254.0.0/16 dev ens33  scope link  metric 1002
169.254.0.0/16 dev br0  scope link  metric 1005
192.168.32.0/24 dev ens33  proto kernel  scope link  src 192.168.32.134
192.168.100.0/24 dev br0  proto kernel  scope link  src 192.168.100.3
[root@nginx-3 ~]# ping 192.168.100.3
PING 192.168.100.3 (192.168.100.3) 56(84) bytes of data.
64 bytes from 192.168.100.3: icmp_seq=1 ttl=64 time=0.066 ms
64 bytes from 192.168.100.3: icmp_seq=2 ttl=64 time=0.095 ms
^C
--- 192.168.100.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1042ms
rtt min/avg/max/mdev = 0.066/0.080/0.095/0.017 ms
[root@nginx-3 ~]# ping 192.168.100.2
PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.
From 192.168.100.3 icmp_seq=1 Destination Host Unreachable
From 192.168.100.3 icmp_seq=2 Destination Host Unreachable
From 192.168.100.3 icmp_seq=3 Destination Host Unreachable
^C
--- 192.168.100.2 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4092ms
pipe 4
[root@nginx-3 ~]#

8.
上面只是临时配置br0的地址,重启就消失.
可写入网卡配置文件,永久生效.参考见下:

虚机1

[root@nginx-1 ~]# cd /etc/sysconfig/network-scripts/
[root@nginx-1 network-scripts]# pwd
/etc/sysconfig/network-scripts
[root@nginx-1 network-scripts]# cat ifcfg-br0
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br0
DEVICE=br0
ONBOOT=yes
IPADDR=192.168.100.2
NETMASK=255.255.255.0
[root@nginx-1 network-scripts]#

虚机2

[root@nginx-3 ~]# cd /etc/sysconfig/network-scripts/
[root@nginx-3 network-scripts]# cat ifcfg-br0
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=none
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=br0
DEVICE=br0
ONBOOT=yes
IPADDR=192.168.100.3
NETMASK=255.255.255.0
[root@nginx-3 network-scripts]#

9.
创建gre隧道

虚机1
网桥br0添加gre1端口和gre1接口,设置端口类型为gre,配置远端Ip为192.168.32.140,远端ip为另一方的ip.

[root@nginx-1 ~]# ovs-vsctl add-port br0 gre1 -- set interface gre1 type=gre option:remote_ip=192.168.32.140

虚机2
网桥br0添加gre1端口和gre1接口,设置端口类型为gre,配置远端Ip为192.168.32.139,远端ip为另一方的ip.

[root@nginx-3 ~]# ovs-vsctl add-port br0 gre1 -- set interface gre1 type=gre option:remote_ip=192.168.32.139

ovs信息

虚机1

[root@nginx-1 ~]# ovs-vsctl show
67ccf09b-d0d9-4ccb-9f2f-1ac7918e5c46
    Bridge "br0"
        Port "gre1"
            Interface "gre1"
                type: gre
                options: {remote_ip="192.168.32.140"}
        Port "br0"
            Interface "br0"
                type: internal
    ovs_version: "2.0.0"
[root@nginx-1 ~]#

虚机2

[root@nginx-3 ~]# ovs-vsctl show
25cdb171-59ee-4eb4-a8af-cd93dc460926
    Bridge "br0"
        Port "gre1"
            Interface "gre1"
                type: gre
                options: {remote_ip="192.168.32.139"}
        Port "br0"
            Interface "br0"
                type: internal
    ovs_version: "2.0.0"
[root@nginx-3 ~]#

隧道建立后,网络实现互通.

[root@nginx-1 ~]# ip a |grep br0
5: br0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    inet 192.168.100.2/24 brd 192.168.100.255 scope global br0
[root@nginx-1 ~]# ip r
default via 192.168.32.2 dev ens33
169.254.0.0/16 dev ens33  scope link  metric 1002
169.254.0.0/16 dev br0  scope link  metric 1005
192.168.32.0/24 dev ens33  proto kernel  scope link  src 192.168.32.139
192.168.100.0/24 dev br0  proto kernel  scope link  src 192.168.100.2
[root@nginx-1 ~]# ping 192.168.100.3
PING 192.168.100.3 (192.168.100.3) 56(84) bytes of data.
64 bytes from 192.168.100.3: icmp_seq=1 ttl=64 time=3.61 ms
64 bytes from 192.168.100.3: icmp_seq=2 ttl=64 time=1.61 ms
64 bytes from 192.168.100.3: icmp_seq=3 ttl=64 time=0.936 ms
^C
--- 192.168.100.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.936/2.056/3.613/1.135 ms
[root@nginx-1 ~]#
[root@nginx-3 ~]# ip a |grep br0
5: br0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    inet 192.168.100.3/24 brd 192.168.100.255 scope global br0
[root@nginx-3 ~]# ip r
default via 192.168.32.2 dev ens33
169.254.0.0/16 dev ens33  scope link  metric 1002
169.254.0.0/16 dev br0  scope link  metric 1005
192.168.32.0/24 dev ens33  proto kernel  scope link  src 192.168.32.140
192.168.100.0/24 dev br0  proto kernel  scope link  src 192.168.100.3
[root@nginx-3 ~]# ping 192.168.100.2
PING 192.168.100.2 (192.168.100.2) 56(84) bytes of data.
64 bytes from 192.168.100.2: icmp_seq=1 ttl=64 time=3.59 ms
64 bytes from 192.168.100.2: icmp_seq=2 ttl=64 time=3.38 ms
64 bytes from 192.168.100.2: icmp_seq=3 ttl=64 time=0.469 ms
^C
--- 192.168.100.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.469/2.483/3.596/1.427 ms
[root@nginx-3 ~]#

10.
创建gre通道后系统重启,br0无法获取ip.

暂时没有找到彻底解决这个问题的方法.怀疑是gre本身的问题导致.
临时测试,可以通过把执行命令写入rc.local,解决掉这个问题,命令见下:

[root@nginx-3 ~]#
[root@nginx-1 network-scripts]# cat /etc/rc.d/rc.local |grep -v ^#
touch /var/lock/subsys/local
ifdown br0 &&ifup br0                                  ##加这条命令
[root@nginx-1 network-scripts]#

注意必须保证rc.local具有x权限,可执行命令chmod +x /etc/rc.d/rc.local

虚机2做一样的操作.

11.
互相ping,互相抓包.
必须ping才能抓到包.注意了.
如果没有tcpdump命令,安装即可,参考命令:

yum -y install tcpdump

虚机1br0端口抓包icmp

[root@nginx-1 ~]# tcpdump -i br0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:38:48.871368 IP 192.168.100.3 > ovs1: ICMP echo request, id 2235, seq 10, length 64
14:38:48.871442 IP ovs1 > 192.168.100.3: ICMP echo reply, id 2235, seq 10, length 64
14:38:49.873007 IP 192.168.100.3 > ovs1: ICMP echo request, id 2235, seq 11, length 64
14:38:49.873059 IP ovs1 > 192.168.100.3: ICMP echo reply, id 2235, seq 11, length 64
14:38:50.874202 IP 192.168.100.3 > ovs1: ICMP echo request, id 2235, seq 12, length 64
14:38:50.874286 IP ovs1 > 192.168.100.3: ICMP echo reply, id 2235, seq 12, length 64
14:38:51.876210 IP 192.168.100.3 > ovs1: ICMP echo request, id 2235, seq 13, length 64
14:38:51.876272 IP ovs1 > 192.168.100.3: ICMP echo reply, id 2235, seq 13, length 64
14:38:52.878047 IP 192.168.100.3 > ovs1: ICMP echo request, id 2235, seq 14, length 64
14:38:52.878112 IP ovs1 > 192.168.100.3: ICMP echo reply, id 2235, seq 14, length 64
14:38:53.880465 IP 192.168.100.3 > ovs1: ICMP echo request, id 2235, seq 15, length 64
14:38:53.880496 IP ovs1 > 192.168.100.3: ICMP echo reply, id 2235, seq 15, length 64
14:38:54.881610 IP 192.168.100.3 > ovs1: ICMP echo request, id 2235, seq 16, length 64
14:38:54.881681 IP ovs1 > 192.168.100.3: ICMP echo reply, id 2235, seq 16, length 64
^C
14 packets captured
14 packets received by filter
0 packets dropped by kernel

虚机2br0端口抓包icmp

[root@nginx-3 ~]# tcpdump -i br0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:40:13.642693 IP 192.168.100.2 > ovs2: ICMP echo request, id 2253, seq 1, length 64
14:40:13.642725 IP ovs2 > 192.168.100.2: ICMP echo reply, id 2253, seq 1, length 64
14:40:14.642769 IP 192.168.100.2 > ovs2: ICMP echo request, id 2253, seq 2, length 64
14:40:14.642810 IP ovs2 > 192.168.100.2: ICMP echo reply, id 2253, seq 2, length 64
14:40:15.643391 IP 192.168.100.2 > ovs2: ICMP echo request, id 2253, seq 3, length 64
14:40:15.643427 IP ovs2 > 192.168.100.2: ICMP echo reply, id 2253, seq 3, length 64
14:40:16.645941 IP 192.168.100.2 > ovs2: ICMP echo request, id 2253, seq 4, length 64
14:40:16.645992 IP ovs2 > 192.168.100.2: ICMP echo reply, id 2253, seq 4, length 64
14:40:17.647337 IP 192.168.100.2 > ovs2: ICMP echo request, id 2253, seq 5, length 64
14:40:17.647410 IP ovs2 > 192.168.100.2: ICMP echo reply, id 2253, seq 5, length 64
14:40:18.650543 IP 192.168.100.2 > ovs2: ICMP echo request, id 2253, seq 6, length 64
14:40:18.650606 IP ovs2 > 192.168.100.2: ICMP echo reply, id 2253, seq 6, length 64
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel

虚机1ens33端口抓gre包

[root@nginx-1 ~]#  tcpdump -i ens33 'proto gre' -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes
14:46:39.388735 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo request, id 2271, seq 22, length 64
14:46:39.388786 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo reply, id 2271, seq 22, length 64
14:46:40.389953 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo request, id 2271, seq 23, length 64
14:46:40.390074 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo reply, id 2271, seq 23, length 64
14:46:41.390733 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo request, id 2271, seq 24, length 64
14:46:41.390852 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo reply, id 2271, seq 24, length 64
14:46:42.393785 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo request, id 2271, seq 25, length 64
14:46:42.393947 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo reply, id 2271, seq 25, length 64
14:46:43.396383 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo request, id 2271, seq 26, length 64
14:46:43.396545 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo reply, id 2271, seq 26, length 64
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel

虚机2ens33端口抓gre包

[root@nginx-3 ~]#  tcpdump -i ens33 'proto gre' -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes
14:47:57.785331 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo request, id 2298, seq 6, length 64
14:47:57.785451 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo reply, id 2298, seq 6, length 64
14:47:58.789091 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo request, id 2298, seq 7, length 64
14:47:58.789169 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo reply, id 2298, seq 7, length 64
14:47:59.787219 IP 192.168.32.139 > 192.168.32.140: GREv0, length 46: ARP, Request who-has 192.168.100.3 tell 192.168.100.2, length 28
14:47:59.787618 IP 192.168.32.140 > 192.168.32.139: GREv0, length 46: ARP, Reply 192.168.100.3 is-at 12:8f:8c:8f:a8:4e, length 28
14:47:59.789472 IP 192.168.32.139 > 192.168.32.140: GREv0, length 102: IP 192.168.100.2 > 192.168.100.3: ICMP echo request, id 2298, seq 8, length 64
14:47:59.789573 IP 192.168.32.140 > 192.168.32.139: GREv0, length 102: IP 192.168.100.3 > 192.168.100.2: ICMP echo reply, id 2298, seq 8, length 64
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel

由以上抓包可见,配置实现了gre隧道的连通.

上一篇：openstack ovs-gre 网速慢解决方案


下一篇：openstack网络基础：网络叠加模式VLAN、VxLAN、GRE