交叉编译tcpreplay
前言
最近项目上出现个问题，其中一个报文发出去后，在互联网中被丢包。后续的重传也被丢包。因为该连接只有该数据包被丢弃，前面和后面的数据包都正常传送。怀疑是该报文被网络中某个设备拦截并丢弃。
为了验证猜想，准备对被丢弃的报文进行重放测试。设备是arm64的openwrt定制系统，需要交叉编译tcpreplay。
在编译过程中颇费了一些周章，所以记录下踩过的这些雷，希望能够帮助到有同样困扰的人。
1. 预备工作
1.1 libpcap库
编译tcpreplay使用libpcap库,需要提前下载并编译,在此不再赘述.
2. 下载tcpreplay源码
去github下载tcpreplay的最新源码。
root@localhost:~## git clone https://github.com/appneta/tcpreplay
3. 配置编译环境
3.1 autoreconf生成配置脚本
root@localhost:~/tcpreplay# autoreconf -vif
3.2 configure生成Makefile
大多数人在这一步常常会遇到各种问题，一部分是编译所需的依赖不满足，另一部分是configure的参数未使用正确，导致执行失败。
3.2.1 第一次配置失败
root@localhost:~/tcpreplay# ./configure --host=aarch64-openwrt-linux --with-libpcap=/root/libpcap/ --prefix=/root/tcpreplay
checking whether to enable maintainer-specific portions of Makefiles... yes
...//省略
checking for libpcap... configure: error: "Unable to find matching library for header file in /root/libpcap/"
提示在指定的路径下找不到匹配的libpcap库。
查看/root/libpcap/下确实有相关头文件和库文件。后分析得知，configure默认查找动态库文件，而我们提供的是静态libpcap.a文件，所以提示找不到。
root@appex:~/libpcap# ls lib/
libpcap.a pkgconfig
为了让,我增加了让configure去查找静态libpcap.a --enable-static参数,结果还是提示同样错误。
难道不支持使用静态libpcap库？于是查看configure -h，发现需要使用 --enable-static-link 来使用静态libpcap库。
root@localhost:~/tcpreplay# ./configure -h
...//省略
--enable-static-link Use static libraries ( .a or .A.tbd ) - default no
--enable-dynamic-link Use shared libraries ( .so .dylib or .tbd ) -
...//省略
3.2.2 第二次配置成功
root@localhost:~/tcpreplay# ./configure --host=aarch64-openwrt-linux --with-libpcap=/root/libpcap/ --prefix=/root/tcpreplay --enable-static_link
checking whether to enable maintainer-specific portions of Makefiles... yes
...//省略
##########################################################################
TCPREPLAY Suite Configuration Results (4.3.3)
##########################################################################
libpcap: /root/libpcap/ (>= 0.9.6)
PF_RING libpcap no
libdnet: no
autogen: /usr/bin/autogen (5.18.12)
Use libopts tearoff: yes
64bit counter support: yes
tcpdump binary path: /usr/sbin/tcpdump
fragroute support: no
tcpbridge support: yes
tcpliveplay support: yes
Supported Packet Injection Methods (*):
Linux TX_RING: no
Linux PF_PACKET: yes
BSD BPF: no
libdnet: no
pcap_inject: yes
pcap_sendpacket: yes **
pcap_netmap no
Linux/BSD netmap: no
Tuntap device support: yes
* In order of preference; see configure --help to override
** Required for tcpbridge
4. 编译tcpreplay
到这一步一般都是make && make install了。大多数情况都能正确编译，少部分情况需要手动做一些调整。
4.1 编译错误1
root@localhost:~/tcpreplay# make
Making all in scripts
make[1]: Entering directory '/root/tcpreplay/scripts'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/tcpreplay/scripts'
Making all in lib
...//省略
make all-am
make[4]: Entering directory '/root/tcpreplay/src/common'
CC cidr.o
In file included from ../../src/defines.h:57:0,
from cidr.c:22:
/root/libpcap//include/pcap.h:43:10: fatal error: pcap/pcap.h: No such file or directory
#include <pcap/pcap.h>
^~~~~~~~~~~~~
compilation terminated.
Makefile:456: recipe for target 'cidr.o' failed
make[4]: *** [cidr.o] Error 1
make[4]: Leaving directory '/root/tcpreplay/src/common'
Makefile:385: recipe for target 'all' failed
make[3]: *** [all] Error 2
make[3]: Leaving directory '/root/tcpreplay/src/common'
Makefile:1194: recipe for target 'all-recursive' failed
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory '/root/tcpreplay/src'
Makefile:574: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/root/tcpreplay/src'
Makefile:450: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
错误提示无法找到 pcap/pcap.h 文件。
明明库文件目录下存在对应文件啊。
root@localhost:~/libpcap# ls include/pcap/pcap.h
include/pcap/pcap.h
后分析得知，gcc的 <> 查找路径里 不包含 /root/libpcap/include,所以差找不到。
#include "..." search starts here:
#include <...> search starts here:
/home/work_arm_src/openwrt/sdk_upload/aarch64-openwrt-linux-gcc/bin/../lib/gcc/aarch64-openwrt-linux-musl/7.3.0/../../../../aarch64-openwrt-linux-musl/sys-include
/home/work_arm_src/openwrt/sdk_upload/aarch64-openwrt-linux-gcc/bin/../lib/gcc/aarch64-openwrt-linux-musl/7.3.0/../../../../aarch64-openwrt-linux-musl/include
/home/work_arm_src/openwrt/sdk_upload/aarch64-openwrt-linux-gcc/bin/../lib/gcc/aarch64-openwrt-linux-musl/7.3.0/include
/home/work_arm_src/openwrt/sdk_upload/aarch64-openwrt-linux-gcc/usr/include
我在这里直接将include目录拷贝到交叉编译器的usr目录下了。
其实有个更优雅的办法，就是编译libpcap库的时候指定 PKG_CONFIG_PATH为编译器的usr/lib/pkgconfig, 这样编译出来的 头文件和库文件自动在编译器的查找路径内。后续就不会遇到这个问题了。
4.2 编译错误2
root@localhost:~/tcpreplay# make
Making all in scripts
make[1]: Entering directory '/root/tcpreplay/scripts'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/tcpreplay/scripts'
Making all in lib
...//省略
CCLD tcpreplay
../libopts/.libs/libopts.a(libopts_la-libopts.o): In function `ao_realloc':
/root/tcpreplay/libopts/autoopts.c:78: undefined reference to `rpl_realloc'
./common/libcommon.a(utils.o): In function `_our_safe_realloc':
/root/tcpreplay/src/common/utils.c:76: undefined reference to `rpl_realloc'
collect2: error: ld returned 1 exit status
Makefile:691: recipe for target 'tcpreplay' failed
make[3]: *** [tcpreplay] Error 1
make[3]: Leaving directory '/root/tcpreplay/src'
Makefile:1194: recipe for target 'all-recursive' failed
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory '/root/tcpreplay/src'
Makefile:574: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/root/tcpreplay/src'
Makefile:450: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
提示 undefined reference to `rpl_realloc’ 。这是肿么回事，查看configure，发现有ac_cv_func_realloc_0_nonnull 参数，如果为no的话，则会使用rpl_realloc。
if test $ac_cv_func_realloc_0_nonnull = yes; then :
$as_echo "#define HAVE_REALLOC 1" >>confdefs.h
else
$as_echo "#define HAVE_REALLOC 0" >>confdefs.h
case " $LIBOBJS " in
*" realloc.$ac_objext "* ) ;;
*) LIBOBJS="$LIBOBJS realloc.$ac_objext"
;;
esac
$as_echo "#define realloc rpl_realloc" >>confdefs.h
fi
于是重新修改configure命令如下
root@localhost:~/tcpreplay# ac_cv_func_realloc_0_nonnull=yes ./configure --host=aarch64-openwrt-linux --with-libpcap=/root/libpcap/ --prefix=/root/tcpreplay --enable-static_link
重新生成Makefile后，再make编译就直接成功了。
root@localhost:~/tcpreplay# make
Making all in scripts
make[1]: Entering directory '/root/tcpreplay/scripts'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/tcpreplay/scripts'
Making all in lib
make[1]: Entering directory '/root/tcpreplay/lib'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/root/tcpreplay/lib'
Making all in libopts
make[1]: Entering directory '/root/tcpreplay/libopts'
make all-am
make[2]: Entering directory '/root/tcpreplay/libopts'
...//省略
make[3]: Leaving directory '/root/tcpreplay/src'
make[2]: Leaving directory '/root/tcpreplay/src'
make[1]: Leaving directory '/root/tcpreplay/src'
make[1]: Entering directory '/root/tcpreplay'
make[1]: Nothing to be done for 'all-am'.
make[1]: Leaving directory '/root/tcpreplay'
后记
交叉编译中的configure命令是门学问，需要多钻研多学习，欢迎大家一起交流成长。