Hadoop3.x进阶:源码编译全记录

一、概述


    为了远程调试集群,我本地编译了一下源码,记录下Hadoop3.1.1源码编译记录和遇到的问题,只写关键步骤,其他自行百度。

软件版本:

    Centos7.2 可联网

    Hadoop3.1.1

    Maven3.5.2

    Jdk1.8

    Protobuf2.5.0 

    cmake3.14.5



1.官网下载源码包,里面包含各个版本hadoop安装包和源码包:

        https://archive.apache.org/dist/hadoop/common/

    我下载的是:

https://archive.apache.org/dist/hadoop/common/hadoop-3.1.1/hadoop-3.1.1-src.tar.gz


2.上传到/opt解压,我这里在Centos7.2桌面版里面装了linux版本的Idea2018.3.6,我直接导入到了他的workspace工作空间下,以备后面源码调试使用。

cd /data/ideaIU201836/Aworkspacetar -zxvf  hadoop-3.1.1-src.tar.gz


3.我的机器是可以联网的,最好在联网环境编译,会依赖各种jar,不然很麻烦,先安装jdk、maven,编译之前还有一些依赖库需要安装:

yum -y install kernel-devel  yum -y install gcc*  yum -y install cmake  yum -y install glibc-headers  yum -y install gcc-c++  yum -y install zip-devel  yum -y install openssl-devel  yum -y install svn  yum -y install git  yum -y install ncurses-devel  yum -y install lzo-devel  yum -y install autoconf  yum -y install libtool  yum -y install automake


4.还需要安装ProtocolBuffer2.5.0,MapReduce和HDFS用protocol buffer来压缩和交换数据,资源在我百度网盘:

    链接:https://pan.baidu.com/s/1ljoP-tLCPkVbtssEdiGTFQ

     提取码:q5fe


    安装命令,大体就这几步:

tar -zxf protobuf-2.5.0.tar.gzcd protobuf-2.5.0./configure --prefix=/data/protobuf-2.5.0make && make install//在/etc/profile中添加bin路径到path中export PATH=/data/protobuf-2.5.0/bin:$PATH//使配置生效source /etc/profile //安装完成查看版本 protoc --version


5.下面就是编译了,这里跳过测试,编译命令有的不一样,你自己百度下:

mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true 仅编译正常源码部分,对于native部分不进行编译,最终结果打包
mvn package -Pdist,native,docs -DskipTests -Dtar 编译正常部分源码、native依赖库以及帮助文档,最终结果打包
mvn -X package -Pdist,native,docs -DskipTests -Dtar -Dmaven.skip.test=true -Dmaven.javadoc.skip=true 我们使用这种去编译


6.只要是能联网基本没啥问题,编译成功:






报错记录


1.编译 Apache Hadoop Common 报错

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile (cmake-compile) on project hadoop-common: CMake failed with error code 1 -> [Help 1]org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile (cmake-compile) on project hadoop-common: CMake failed with error code 1    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)

上面还有一点报错信息,应该是要升级CMake版本:

[WARNING] CMake Error at CMakeLists.txt:23 (cmake_minimum_required):[WARNING]   CMake 3.1 or higher is required.  You are running version 2.8.12.2[WARNING] [WARNING] [WARNING] -- Configuring incomplete, errors occurred![WARNING] See also "/data/ideaIU201836/Aworkspace/hadoop-3.1.1-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeOutput.log".


我这里升级成了cmake-3.14.5问题解决,大体步骤如下:

先卸载原有的cmakeyum remove cmake下载cd /optwget https://github.com/Kitware/CMake/releases/download/v3.14.5/cmake-3.14.5-Linux-x86_64.tar.gzc解压:tar zxvf cmake-3.14.5-Linux-x86_64.tar.gz配置环境变量:export CMAKE_HOME=/opt/cmake-3.14.5-Linux-x86_64export PATH=$PATH:$CMAKE_HOME/bin//生效source /etc/profile//查看版本是否安装成功cmake -version

2.报错信息:Apache Hadoop Client Packaging Invariants for Test报错

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce (enforce-banned-dependencies) on project hadoop-client-check-test-invariants: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed.    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51)    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)    at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)    at java.lang.reflect.Method.invoke (Method.java:498)    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)Caused by: org.apache.maven.plugin.MojoExecutionException: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed.    at org.apache.maven.plugins.enforcer.EnforceMojo.execute (EnforceMojo.java:243)    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:134)    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208)    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154)    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146)    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51)    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309)    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194)    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107)    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:955)    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:290)    at org.apache.maven.cli.MavenCli.main (MavenCli.java:194)    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)    at java.lang.reflect.Method.invoke (Method.java:498)    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)[ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR]   mvn <goals> -rf :hadoop-client-check-test-invariants

解决方法:

修改文件/data/ideaIU201836/Aworkspace/hadoop-3.1.1-src/hadoop-client-modules/pom.xml,注释掉报错工程那一行即可:


 <modules>    <!-- Left as an empty artifact w/dep for compat -->    <module>hadoop-client</module>    <!-- Should be used at compile scope for access to IA.Public classes -->    <module>hadoop-client-api</module>    <!-- Should be used at runtime scope for remaining classes necessary for hadoop-client-api to function -->    <module>hadoop-client-runtime</module>    <!-- Should be used at test scope for those that need access to mini cluster that works with above api and runtime -->    <module>hadoop-client-minicluster</module>    <!-- Checks invariants above -->    <module>hadoop-client-check-invariants</module>    //修改 这里注释掉这一行   <!-- <module>hadoop-client-check-test-invariants</module>-->    <!-- Attempt to use the created libraries -->    <module>hadoop-client-integration-tests</module>


   至此Hadoop3.1.1编译结束,我看网上很多源码调试的,但是都写的不太清楚,后面我会讲解各个组件如何进行远程调试集群。


上一篇:130道 Java多线程面试题汇总


下一篇:Hadoop3.x集成Spark