ubuntu14.04下编译安装ambari-2.4.2.0

2022-10-02 11:10:47

ubuntu14.04下编译安装ambari-2.4.2.0

编译前的准备工作

准备工作有：

下面是具体配置

系统参数

1、fs.inotify.max_user_watches = 8192 调整到524288

echo fs.inotify.max_user_watches=524288 >> /etc/sysctl.conf && sysctl -p

2、将每个进程可以打开的文件数目加大到65535，缺省为1024

ulimit -SHn 65535

sudo tee /etc/security/limits.d/90-nproc.conf <<-'EOF'

* hard nproc 64000

* soft nproc 64000

root hard nproc 65535

root soft nproc 65535

EOF

ulimit -a

系统依赖（编译环境）

sudo apt-get install build-essential cmake automake autoconf bison libboost-all-dev libtool check libmcrypt-dev libeditline-dev libssl-dev

离线安装包

1. httpserver

sh starthttpd.sh

#/usr/bin

cd /media/gordon/DataDisk/软件包

/usr/bin/python -m SimpleHTTPServer 8000 >/dev/null 2>&1 &

exit 0

2. 布置本地软件源

http://192.168.56.1:8000/ambari-HDP/

-rwxr-x--- 1 gitlab root 23667150 3月 31 16:26 grafana-2.6.0.linux-x64.tar.gz*

-rwxr-x--- 1 gitlab root 210185315 3月 31 17:15 hadoop-2.7.1.2.3.4.0-3347.tar.gz*

-rwxr-x--- 1 gitlab root 101551599 3月 31 16:26 hbase-1.1.2.2.3.4.0-3347.tar.gz*

-rwxr-x--- 1 gitlab root 251655846 3月 31 16:26 phoenix-4.4.0.2.3.4.0-3347.tar.gz*

java环境

vim /etc/profile

#set for java

export JAVA_HOME=/opt/jdk1.8.0_111

export PATH=$PATH:$JAVA_HOME/bin

export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib

export _JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true"

maven环境

vim /etc/profile

#set maven environment

M2_HOME=/opt/apache-maven-3.3.9/

export MAVEN_OPTS="-Xms256m -Xmx512m"

export PATH=$M2_HOME/bin:$PATH

maven有2个位置可以选择配置M2_HOME/conf/setting.xml和~/.m2/setting.xml

前者是全局性的，后者是针对用户的。

查询资料是说用户配置优先级高于全局配置

设置maven-repo本地源

vim /opt/apache-maven-3.3.9/conf/settings.xml

  <localRepository>/opt/maven-repo</localRepository>

gord@gord:/opt$ mvn -v

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)

Maven home: /opt/apache-maven-3.3.9

Java version: 1.8.0_111, vendor: Oracle Corporation

Java home: /opt/jdk1.8.0_111/jre

Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "3.13.0-24-generic", arch: "amd64", family: "unix"

Nodejs环境

今天是2017.04.01 https://nodejs.org/en/官网推荐v6.10.1版本 #实验证明这个版本安装ambari存在很大问题，切换到0.10.44版本的nodejs

可在淘宝下载各个版本：https://npm.taobao.org/mirrors/node

tar xfp node-v0.10.44-linux-x64.tar.gz -C /opt

vim /etc/profile

#set for nodejs

export NODE_HOME=/opt/node-v0.10.44-linux-x64

#export NODE_HOME=/opt/node-v4.4.4-linux-x64

#export NODE_HOME=/opt/node-v6.10.1-linux-x64

export PATH=$PATH:$NODE_HOME/bin

source /etc/profile

配置淘宝源

kylin@Ubuntu:~$ npm config set registry https://registry.npm.taobao.org

kylin@Ubuntu:~$ npm info underscor  （如果上面配置正确这个命令会有字符串response）

gord@gord:~$ npm --registry https://registry.npm.taobao.org info underscore

修改源地址为官方源

npm config set registry https://registry.npmjs.org/

安装指定版本的brunch

npm root -g

cd ...

npm install -g brunch@1.7.20  #务必要装这个版本

brunch -V

1.7.20

如果存在问题，执行卸载brunch

npm remove -g brunc

git环境

root@gord:~# git --version

git version 1.9.1

ant环境

root@gord:~# apt-get install ant

root@gord:~# ant -version

Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0

Apache Ant(TM) version 1.9.3 compiled on April 8 2014

python环境

apt install python-pip

准备工作总结：

profile配置总结

#set for java

export JAVA_HOME=/opt/jdk1.8.0_111

export PATH=$PATH:$JAVA_HOME/bin

export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib

export _JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true"

#set for maven

M2_HOME=/opt/apache-maven-3.3.9/

export MAVEN_OPTS="-Xms256m -Xmx512m"

export PATH=$M2_HOME/bin:$PATH

#set for nodejs

export NODE_HOME=/opt/node-v0.10.44-linux-x64

#export NODE_HOME=/opt/node-v4.4.4-linux-x64

#export NODE_HOME=/opt/node-v6.10.1-linux-x64

export PATH=$PATH:$NODE_HOME/bin

至此编译ambari所依赖的环境配置完毕，并可作为通用的编译环境推广使用。

推广到其它编译环境中，注意nodejs的版本可能需要更换！

准备工作至关重要，关系到编译的成败和时间的消耗！

编译安装ambari

主要步骤：

参考

https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.4.2

版本控制

wget http://www.apache.org/dist/ambari/ambari-2.4.2/apache-ambari-2.4.2-src.tar.gz (use the suggested mirror from above)

tar xfvz apache-ambari-2.4.2-src.tar.gz

cd apache-ambari-2.4.2-src

mvn versions:set -DnewVersion=2.4.2.0.0

root@gord:/opt/apache-ambari-2.4.2-src# cd ambari-metrics

/opt/apache-ambari-2.4.2-src/ambari-metrics /opt/apache-ambari-2.4.2-src

mvn versions:set -DnewVersion=2.4.2.0.0

root@gord:/opt/apache-ambari-2.4.2-src/ambari-metrics# cd ..

/opt/apache-ambari-2.4.2-src

编译

-B非交互

mvn -B clean install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6"

-X -e debug

mvn -X -e clean install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6"

如果存在手工下载软件包到指定位置的情况，上述命令去掉clean参数，防止被clean

编译成功显示

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO]

[INFO] Ambari Main ........................................ SUCCESS [  2.741 s]

[INFO] Apache Ambari Project POM .......................... SUCCESS [  0.155 s]

[INFO] Ambari Web ......................................... SUCCESS [01:00 min]

[INFO] Ambari Views ....................................... SUCCESS [  1.475 s]

[INFO] Ambari Admin View .................................. SUCCESS [ 19.390 s]

[INFO] ambari-metrics ..................................... SUCCESS [  0.770 s]

[INFO] Ambari Metrics Common .............................. SUCCESS [  0.732 s]

[INFO] Ambari Metrics Hadoop Sink ......................... SUCCESS [  4.208 s]

[INFO] Ambari Metrics Flume Sink .......................... SUCCESS [  2.511 s]

[INFO] Ambari Metrics Kafka Sink .......................... SUCCESS [  1.676 s]

[INFO] Ambari Metrics Storm Sink .......................... SUCCESS [  4.837 s]

[INFO] Ambari Metrics Storm Sink (Legacy) ................. SUCCESS [  3.420 s]

[INFO] Ambari Metrics Collector ........................... SUCCESS [ 23.296 s]

[INFO] Ambari Metrics Monitor ............................. SUCCESS [  1.944 s]

[INFO] Ambari Metrics Grafana ............................. SUCCESS [  2.544 s]

[INFO] Ambari Metrics Assembly ............................ SUCCESS [02:26 min]

[INFO] Ambari Server ...................................... SUCCESS [04:24 min]

[INFO] Ambari Functional Tests ............................ SUCCESS [  1.385 s]

[INFO] Ambari Agent ....................................... SUCCESS [ 31.964 s]

[INFO] Ambari Client ...................................... SUCCESS [  0.084 s]

[INFO] Ambari Python Client ............................... SUCCESS [  1.168 s]

[INFO] Ambari Groovy Client ............................... SUCCESS [  8.157 s]

[INFO] Ambari Shell ....................................... SUCCESS [  0.105 s]

[INFO] Ambari Python Shell ................................ SUCCESS [  1.422 s]

[INFO] Ambari Groovy Shell ................................ SUCCESS [  3.714 s]

[INFO] ambari-logsearch ................................... SUCCESS [  0.041 s]

[INFO] Ambari Logsearch Appender .......................... SUCCESS [ 31.627 s]

[INFO] Ambari Logsearch Solr Client ....................... SUCCESS [01:38 min]

[INFO] Ambari Logsearch Portal ............................ SUCCESS [06:10 min]

[INFO] Ambari Logsearch Log Feeder ........................ SUCCESS [06:03 min]

[INFO] Ambari Logsearch Assembly .......................... SUCCESS [  0.323 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 24:14 min

[INFO] Finished at: 2017-04-05T18:44:18+08:00

[INFO] Final Memory: 274M/850M

[INFO] ------------------------------------------------------------------------

整理deb包

root@gord:/opt/apache-ambari-2.4.2-src# find ./ -name "*.deb" |awk '{print "cp",$0,"/media/sf_DataDisk/软件包/ambari-repo/Compiled/"}'|sh

./target/ambari_2.4.2.0.0_all.deb

./ambari-metrics/target/ambari-metrics_2.4.2.0.0_all.deb

./ambari-metrics/ambari-metrics-assembly/target/ambari-metrics-assembly_2.4.2.0.0_all.deb

./ambari-agent/target/ambari-agent_2.4.2.0-0.deb

./ambari-shell/target/ambari-shell_2.4.2.0.0_all.deb

./ambari-shell/ambari-python-shell/target/ambari-python-shell_2.4.2.0-0.deb

./ambari-project/target/ambari-project_2.4.2.0.0_all.deb

./ambari-server/target/ambari-server_2.4.2.0-0-dist.deb

./ambari-client/target/ambari-client_2.4.2.0.0_all.deb

./ambari-client/python-client/target/python-client_2.4.2.0-0.deb

./ambari-client/groovy-client/target/groovy-client_2.4.2.0.0_all.deb

安装

服务端安装

##gdebi ambari-server*   #This should also pull in postgres packages as well.

dpkg -i ambari-server_2.4.2.0-0-dist.deb

apt-get -f install

设置与启动

ambari-server setup #默认即可

ambari-server start

如果遇到报错：/usr/sbin/ambari-server: line 33: buildNumber: unbound variable

打开ambari-server脚本，把${buildNumber}这行换成 HASH="${VERSION}"

相关的命令有以下：

ambari-server status

ambari-server stop

客户端安装

dpkg -i ambari-agent*

apt-get -f install

编辑ambari-agent配置文件，设置之前安装过Ambari-server 的主机名：

vi /etc/ambari-agent/conf/ambari-agent.ini

[server]

hostname=<your.ambari.server.hostname>

url_port=8440

secured_url_port=8441

ambari-agent start

如果遇到报错：/usr/sbin/ambari-agent: line 23: buildNumber: unbound variable

打开ambari-agent脚本，把${buildNumber}这行换成 HASH="${VERSION}"

相关的命令有以下：

ambari-agent status # 检查agent服务运行状态

ambari-agent stop   # 停止agent服务

访问

http://<ambari-server-host>:8080.

Log in with username admin and password admin

问题解决

1、Caused by: org.vafer.jdeb.PackagingException: “/opt/apache-ambari-2.4.2-src/ambari-metrics/ambari-metrics-grafana/src/main/package/deb/control” is not a valid ‘control’ directory)

/opt/apache-ambari-2.4.2-src/ambari-metrics/ambari-metrics-grafana# vim pom.xml

      <plugin>

         <groupId>org.vafer</groupId>

         <artifactId>jdeb</artifactId>

         <version>1.0.1</version>

         <executions>

           <execution>

             <!--Stub execution on direct plugin call - workaround for ambari deb build process-->

             <id>stub-execution</id>

             <phase>none</phase>

             <goals>

               <goal>jdeb</goal>

             </goals>

           </execution>

         </executions>

         <configuration>

           <skip>true</skip>

           <attach>false</attach>

           <submodules>false</submodules>

           <controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>

         </configuration>

       </plugin>

2、如果中间断掉，建议去掉clean，继续编译。

3、安装Nodejs 6.x.x版本不能编译，建议卸载，安装0.10.44。

[INFO] Ambari Web ......................................... FAILURE [ 13.325 s]

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (compile-brunch) on project ambari-web: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1]

版本检查：

root@gord:/opt/node-v6.10.1-linux-x64/lib/node_modules# brunch --version

2.10.9

root@gord:/opt/node-v6.10.1-linux-x64/lib/node_modules# npm --version

3.10.10

root@gord:/opt/node-v6.10.1-linux-x64/lib/node_modules# node --version

v6.10.1

改成安装node 0.10.44，brunch1.7.20后正常

4、编译过程中出现版本不一致问题，请参考http://blog.csdn.net/royma_1990/article/details/51749981。

5、下载慢问题处理

修改如下配置文件：

cat ambari-metrics/pom.xml

   <!--TODO change to HDP URL-->

    <hbase.tar>http://192.168.56.1:8000/ambari-HDP/hbase-1.1.2.2.3.4.0-3347.tar.gz</hbase.tar>

    <hbase.folder>hbase-1.1.2.2.3.4.0-3347</hbase.folder>

    <hadoop.tar>http://192.168.56.1:8000/ambari-HDP/hadoop-2.7.1.2.3.4.0-3347.tar.gz</hadoop.tar>

    <hadoop.folder>hadoop-2.7.1.2.3.4.0-3347</hadoop.folder>

    <hbase.winpkg.zip>https://msibuilds.blob.core.windows.net/hdp/2.x/2.2.4.2/2/hbase-0.98.4.2.2.4.2-0002-hadoop2.winpkg.zip</hbase.winpkg.zip>

    <hbase.winpkg.folder>hbase-0.98.4.2.2.4.2-0002-hadoop2</hbase.winpkg.folder>

    <hadoop.winpkg.zip>https://msibuilds.blob.core.windows.net/hdp/2.x/2.2.4.2/2/hadoop-2.6.0.2.2.4.2-0002.winpkg.zip</hadoop.winpkg.zip>

    <hadoop.winpkg.folder>hadoop-2.6.0.2.2.4.2-0002</hadoop.winpkg.folder>

    <grafana.folder>grafana-2.6.0</grafana.folder>

    <grafana.tar>http://192.168.56.1:8000/ambari-HDP/grafana-2.6.0.linux-x64.tar.gz</grafana.tar>

    <phoenix.tar>http://192.168.56.1:8000/ambari-HDP/phoenix-4.4.0.2.3.4.0-3347.tar.gz</phoenix.tar>

    <phoenix.folder>phoenix-4.4.0.2.3.4.0-3347</phoenix.folder>

6、警告：（可忽略）

[INFO] npm WARN deprecated graceful-fs@2.0.3: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.

root@gord:/opt/apache-ambari-2.4.2-src/ambari-web# npm ls graceful-fs

Ambari@2.4.0 /opt/apache-ambari-2.4.2-src/ambari-web

├─┬ karma@0.11.14

│ └── graceful-fs@2.0.3

├─┬ karma-coverage@0.2.7

│ └─┬ dateformat@1.0.12

│   └─┬ meow@3.7.0

│     └─┬ read-pkg-up@1.0.1

│       └─┬ read-pkg@1.1.0

│         ├─┬ load-json-file@1.1.0

│         │ └── graceful-fs@4.1.11

│         └─┬ path-type@1.1.0

│           └── graceful-fs@4.1.11

├─┬ mocha-phantomjs@3.1.6

│ └─┬ mocha@1.13.0

│   └─┬ glob@3.2.3

│     └── graceful-fs@2.0.3

└─┬ phantomjs@1.9.20

  └─┬ fs-extra@0.26.7

    └── graceful-fs@4.1.11

7、找不到org.apache.storm:storm-core:jar:1.1.0-SNAPSHOT

[INFO] Ambari Metrics Storm Sink .......................... FAILURE [ 4.111 s]

[ERROR] Failed to execute goal on project ambari-metrics-storm-sink: Could not resolve dependencies for project org.apache.ambari:ambari-metrics-storm-sink:jar:2.4.2.0.0: Could not find artifact org.apache.storm:storm-core:jar:1.1.0-SNAPSHOT in apache-hadoop (http://repo.hortonworks.com/content/repositories/releases/org/apache/) -> [Help 1]

vim ambari-metrics/ambari-metrics-storm-sink/pom.xml

  <properties>

    <!--storm.version>1.1.0-SNAPSHOT</storm.version-->

    <storm.version>1.1.0</storm.version>

  </properties>

8、ambari-funtest

[ERROR] Failed to create debian package /home/kylin/apache-ambari-2.4.2-src/ambari-funtest/target/ambari-funtest_2.4.2.0.0_all.deb

org.vafer.jdeb.PackagingException: “/home/kylin/apache-ambari-2.4.2-src/ambari-funtest/src/main/package/deb/control” is not a valid ‘control’ directory)

解决办法：在上面添加一个元素

/opt/apache-ambari-2.4.2-src/ambari-funtest$ vi pom.xml

    <plugins>

      ...

      <plugin>

         <groupId>org.vafer</groupId>

         <artifactId>jdeb</artifactId>

         <version>1.0.1</version>

         <executions>

           <execution>

             <!--Stub execution on direct plugin call - workaround for ambari deb build process-->

             <id>stub-execution</id>

             <phase>none</phase>

             <goals>

               <goal>jdeb</goal>

             </goals>

           </execution>

         </executions>

         <configuration>

           <skip>true</skip>

           <attach>false</attach>

           <submodules>false</submodules>

           <controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>

         </configuration>

       </plugin>

    </plugins>

    <resources>

      <resource>

10、[ERROR] Failed to execute goal org.codehaus.mojo:xml-maven-plugin:1.0:transform (default) on project ambari-server: Failed to transform input file /opt/apache-ambari-2.4.2-src/ambari-server/target/findbugs/findbugsXml.html: java.net.UnknownHostException: www.w3.org: Unknown host www.w3.org -> [Help 1]

解决办法：重新执行

mvn -B install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true

11、findbugsXml.html

[ERROR] Failed to execute goal org.codehaus.mojo:xml-maven-plugin:1.0:transform (default) on project ambari-server: Failed to transform input file /opt/apache-ambari-2.4.2-src/ambari-server/target/findbugs/findbugsXml.html: The entity name must immediately follow the ‘&’ in the entity reference. -> [Help 1]

解决办法1： 重新执行

mvn -B install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true

解决办法2: 如果网络情况不好，这一步就很难过去。那只好修改pom.xml文件，将findbugs内容注释掉：

/opt/apache-ambari-2.4.2-src/ambari-server$ vim pom.xml

      <plugin>

        <groupId>org.codehaus.mojo</groupId>

        <artifactId>findbugs-maven-plugin</artifactId>

        <version>3.0.3</version>

        <configuration>

          <failOnError>false</failOnError>

          <threshold>Low</threshold>

          <findbugsXmlOutputDirectory>${project.basedir}/target/findbugs</findbugsXmlOutputDirectory>

        </configuration>

        <executions>

          <!--

          <execution>

            <phase>verify</phase>

            <goals>

              <goal>check</goal>

            </goals>

          </execution>

          -->

        </executions>

      </plugin>

      <plugin>

        <groupId>org.codehaus.mojo</groupId>

        <artifactId>xml-maven-plugin</artifactId>

        <version>1.0</version>

        <executions>

           <!--

           <execution>

            <phase>verify</phase>

            <goals>

              <goal>transform</goal>

            </goals>

          </execution>

          -->

        </executions>

12、[ERROR] Failed to execute goal org.vafer:jdeb:1.0.1:jdeb (default-cli) on project ambari-logsearch: Failed to create debian package /opt/apache-ambari-2.4.2-src/ambari-logsearch/target/ambari-logsearch_2.4.2.0.0_all.deb: "/opt/apache-ambari-2.4.2-src/ambari-logsearch/src/main/package/deb/control" is not a valid 'control' directory) -> [Help 1]

解决办法：在上面添加一个元素

/opt/apache-ambari-2.4.2-src/ambari-logsearch$ vim pom.xml

       <plugin>

         <groupId>org.vafer</groupId>

         <artifactId>jdeb</artifactId>

         <version>1.0.1</version>

         <executions>

           <execution>

             <!--Stub execution on direct plugin call - workaround for ambari deb build process-->

             <id>stub-execution</id>

             <phase>none</phase>

             <goals>

               <goal>jdeb</goal>

             </goals>

           </execution>

         </executions>

         <configuration>

           <skip>true</skip>

           <attach>false</attach>

           <submodules>false</submodules>

           <controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>

         </configuration>

       </plugin>

    </plugins>

  </build>

参考资源：

https://github.com/apache/ambari

https://cwiki.apache.org/confluence/display/AMBARI/Installation+Guide+for+Ambari+2.4.2

http://blog.csdn.net/chengyuqiang/article/details/53788351

http://blog.csdn.net/royma_1990/article/details/51749981

@firecontainer

April 6, 2017 4:53 PM

上一篇：CentOS 6.4中升级编译安装GCC 4.8.1 + GDB 7.6.1 + Eclipse 以及Kdump配置


下一篇：Apache+Tomcat构建Tomcat负载均衡集群