编译安装hadoop2.x

2023-02-16 07:48:06

1、Requirements:

* Unix System

* JDK 1.7+

* Maven 3.0 or later

* Findbugs 1.3.9 (if running findbugs)

* ProtocolBuffer 2.5.0

* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac

* Zlib devel (if compiling native code)

* openssl devel ( if compiling native hadoop-pipes )

* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

安装以上先决条件：

yum groupinstall “Development tools”

yum -y install cmake zlib openssl gcc zlib-devel openssl-devel make

tar -zxvf protobuf-2.5.0.tar.gz

cd protobuf-2.5.0

./configure

make && make install

protoc --version

tar -zxvf findbugs-3.0.0.tar.gz -C /usr/local/findbugs

mv /usr/local/findbugs-3.0.0 /usr/local/findbugs

设置环境变量：

export FINDBUGS_HOME = /usr/local/findbugs

export PATH=$PATH:$FINDBUGS_HOME/bin

生效环境变量：source /etc/profile

验证：findbugs -version

安装jdk，略。

2、安装mave：

1）解压缩maven

tar -zxvf apache-maven-3.3.3-bin.tar.gz

mv apache-maven-3.3.3 /opt/service/maven-3.3.3

chown -R hadoop:hadoop /opt/service/maven-3.3.3/

2）配置maven环境变量：

export MAVEN_HOME=/opt/service/maven-3.3.3

export PATH=$MAVEN_HOME/bin:$PATH

3）本机编译环境如下：

#java -version

java version "1.7.0_67"

Java(TM) SE Runtime Environment (build 1.7.0_67-b01)

Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

# scala -version

Scala code runner version 2.10.6 -- Copyright 2002-2013, LAMP/EPFL

# mvn -version

Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T19:57:37+08:00)

Maven home: /opt/service/maven-3.3.3

Java version: 1.7.0_67, vendor: Oracle Corporation

Java home: /opt/service/jdk1.7.0_67/jre

Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "3.10.0-514.el7.x86_64", arch: "amd64", family: "unix"

4）修改maven仓库默认位置：

vim conf/settings.xml

   --添加如下内容：
   <localRepository>/home/hadoop/.m2/repository/</localRepository>


5）修改maven镜像位置为阿里云的maven镜像仓库

<mirrors>

<mirror>
          <id>alimaven</id>
          <mirrorOf>central</mirrorOf>
          <name>aliyun maven</name>
          <url>http://maven.aliyun.com/nexus/content/repositories/central/</url>
      </mirror>

      <mirror>
          <id>repo1</id>
          <mirrorOf>central</mirrorOf>
          <name>Human Readable Name for this Mirror</name>
          <url>http://repo1.maven.org/maven2/</url>
      </mirror>

      <mirror>
          <id>repo2</id>
          <mirrorOf>central</mirrorOf>
          <name>Human Readable Name for this Mirror</name>
          <url>http://repo2.maven.org/maven2/</url>
      </mirror>

</mirrors>

更改完成之后可以在命令行，执行命令验证一下：mvn help:system

命令执行完成后我们可以在目录/home/hadoop/.m2/repository/下看到很多文件，这时表示maven已经配置成功了。

$ ls /home/hadoop/.m2/repository/

backport-util-concurrent classworlds com commons-cli commons-lang jdom jtidy junit org plexus xmlpull xpp3

3、编译hadoop

export MAVEN_OPTS="-Xms256m -Xmx512m"

mvn package -DskipTests -Pdist,native -Drequire.snappy

4、编译成功后的tar包存储在

./hadoop-dist/target/hadoop-2.6.0.tar.gz

5、配置hadoop的native支持
     用步骤6压缩包中lib/native目录下文件替换目标节点lib/native文件，然后通过以下指令检查hadoop库支持：

$bin/ hadoop checknative

Native library checking:

hadoop: true /opt/cdh5/hadoop-2.6.0-cdh5.10.0/lib/native/libhadoop.so

zlib:    true /lib64/libz.so.1

snappy: true /lib64/libsnappy.so.1

lz4:     true revision:99

bzip2:   true /lib64/libbz2.so.1

openssl: true /lib64/libcrypto.so

至此，hadoop完全分布式安装成功。

码农公寓

相关文章