Hadoop3.x进阶：源码编译全记录

2022-10-19 16:00:28

一、概述


    为了远程调试集群，我本地编译了一下源码，记录下Hadoop3.1.1源码编译记录和遇到的问题，只写关键步骤，其他自行百度。

软件版本：

    Centos7.2 可联网

    Hadoop3.1.1

    Maven3.5.2

    Jdk1.8

    Protobuf2.5.0 

    cmake3.14.5



1.官网下载源码包，里面包含各个版本hadoop安装包和源码包：

        https://archive.apache.org/dist/hadoop/common/

    我下载的是：

https://archive.apache.org/dist/hadoop/common/hadoop-3.1.1/hadoop-3.1.1-src.tar.gz


2.上传到/opt解压，我这里在Centos7.2桌面版里面装了linux版本的Idea2018.3.6，我直接导入到了他的workspace工作空间下，以备后面源码调试使用。

cd /data/ideaIU201836/Aworkspacetar -zxvf  hadoop-3.1.1-src.tar.gz


3.我的机器是可以联网的，最好在联网环境编译，会依赖各种jar，不然很麻烦，先安装jdk、maven，编译之前还有一些依赖库需要安装：

yum -y install kernel-devel  yum -y install gcc*  yum -y install cmake  yum -y install glibc-headers  yum -y install gcc-c++  yum -y install zip-devel  yum -y install openssl-devel  yum -y install svn  yum -y install git  yum -y install ncurses-devel  yum -y install lzo-devel  yum -y install autoconf  yum -y install libtool  yum -y install automake


4.还需要安装ProtocolBuffer2.5.0，MapReduce和HDFS用protocol buffer来压缩和交换数据，资源在我百度网盘：

    链接：https://pan.baidu.com/s/1ljoP-tLCPkVbtssEdiGTFQ

     提取码：q5fe


    安装命令，大体就这几步：

tar -zxf protobuf-2.5.0.tar.gzcd protobuf-2.5.0./configure --prefix=/data/protobuf-2.5.0make && make install//在/etc/profile中添加bin路径到path中export PATH=/data/protobuf-2.5.0/bin:$PATH//使配置生效source /etc/profile //安装完成查看版本 protoc --version


5.下面就是编译了，这里跳过测试，编译命令有的不一样，你自己百度下：

mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true 仅编译正常源码部分，对于native部分不进行编译，最终结果打包
mvn package -Pdist,native,docs -DskipTests -Dtar 编译正常部分源码、native依赖库以及帮助文档，最终结果打包
mvn -X package -Pdist,native,docs -DskipTests -Dtar -Dmaven.skip.test=true -Dmaven.javadoc.skip=true 我们使用这种去编译


6.只要是能联网基本没啥问题，编译成功：




报错记录：


1.编译 Apache Hadoop Common 报错

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile (cmake-compile) on project hadoop-common: CMake failed with error code 1 -> [Help 1]org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile (cmake-compile) on project hadoop-common: CMake failed with error code 1    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)

上面还有一点报错信息，应该是要升级CMake版本：

[WARNING] CMake Error at CMakeLists.txt:23 (cmake_minimum_required):[WARNING]   CMake 3.1 or higher is required.  You are running version 2.8.12.2[WARNING] [WARNING] [WARNING] -- Configuring incomplete, errors occurred![WARNING] See also "/data/ideaIU201836/Aworkspace/hadoop-3.1.1-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeOutput.log".


我这里升级成了cmake-3.14.5问题解决，大体步骤如下：

先卸载原有的cmakeyum remove cmake下载cd /optwget https://github.com/Kitware/CMake/releases/download/v3.14.5/cmake-3.14.5-Linux-x86_64.tar.gzc解压：tar zxvf cmake-3.14.5-Linux-x86_64.tar.gz配置环境变量：export CMAKE_HOME=/opt/cmake-3.14.5-Linux-x86_64export PATH=$PATH:$CMAKE_HOME/bin//生效source /etc/profile//查看版本是否安装成功cmake -version

2.报错信息：Apache Hadoop Client Packaging Invariants for Test报错

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed.    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51)    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)    at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)    at java.lang.reflect.Method.invoke (Method.java:498)    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)Caused by: org.apache.maven.plugin.MojoExecutionException: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed.    at org.apache.maven.plugins.enforcer.EnforceMojo.execute (EnforceMojo.java:243)    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:134)    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51)    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)    at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)    at java.lang.reflect.Method.invoke (Method.java:498)    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)[ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR]   mvn <goals> -rf :hadoop-client-check-test-invariants

解决方法：

修改文件/data/ideaIU201836/Aworkspace/hadoop-3.1.1-src/hadoop-client-modules/pom.xml，注释掉报错工程那一行即可：


  <modules>    <!-- Left as an empty artifact w/dep for compat -->    <module>hadoop-client</module>    <!-- Should be used at compile scope for access to IA.Public classes -->    <module>hadoop-client-api</module>    <!-- Should be used at runtime scope for remaining classes necessary for hadoop-client-api to function -->    <module>hadoop-client-runtime</module>    <!-- Should be used at test scope for those that need access to mini cluster that works with above api and runtime -->    <module>hadoop-client-minicluster</module>    <!-- Checks invariants above -->    <module>hadoop-client-check-invariants</module>    //修改 这里注释掉这一行   <!-- <module>hadoop-client-check-test-invariants</module>-->    <!-- Attempt to use the created libraries -->    <module>hadoop-client-integration-tests</module>


   至此Hadoop3.1.1编译结束，我看网上很多源码调试的，但是都写的不太清楚，后面我会讲解各个组件如何进行远程调试集群。


上一篇：130道 Java多线程面试题汇总


下一篇：Hadoop3.x集成Spark