大数据平台搭建（Ambari +HDP）

2024-01-21 11:33:41

安装包

ambari-2.7.3.0-centos7.tar.gz
HDP-3.1.0.0-centos7-rpm.tar.gz
HDP-UTILS-1.1.0.22-centos7.tar.gz
HDP-GPL-3.1.0.0-centos7-gpl.tar.gz

Ambari

Ambari 跟 Hadoop 等开源软件一样，也是 Apache Software Foundation 中的一个项目，并且是*项目。就 Ambari 的作用来说，就是创建、管理、监视 Hadoop 的集群，但是这里的 Hadoop 指的是 Hadoop 整个生态圈（例如 Hive，Hbase，Sqoop，Zookeeper 等），而并不仅是特指 Hadoop。用一句话来说，Ambari 就是为了让 Hadoop 以及相关的大数据软件更容易使用的一个管理工具。
Ambari 自身也是一个分布式架构的软件，主要由两部分组成：Ambari Server 和 Ambari Agent。简单来说，用户通过 Ambari Server 通知 Ambari Agent 安装对应的软件；Agent 会定时地发送各个机器每个软件模块的状态给 Ambari Server，最终这些状态信息会呈现在 Ambari 的 GUI，方便用户了解到集群的各种状态，并进行相应的维护。

HDP

HDP 是 hortonworks 的软件栈，里面包含了 Hadoop 生态系统的所有软件项目，比如 HBase、Zookeeper、Hive、Pig 等

HDP-UTILS

HDP-UTILS 是工具类库

准备工作

集群节点规划

主机名	IP	功能	内存	磁盘
yum.hdp	192.168.229.30	Ambari/HDP packages	1G	50G
node-01.hdp	192.168.229.31	Ambari Server/Agent	3G	50G
node-02.hdp	192.168.229.32	Ambari Agent	2G	50G
node-03.hdp	192.168.229.33	Ambari Agent	2G	50G

克隆 4 台虚拟机

修改主机名

# vi /etc/hostname

修改 IP 地址

# vi /etc/sysconfig/network-scripts/ifcfg-ens33
# service network restart
# ping www.baidu.com

hosts 文件添加主机名和IP地址映射

windows 系统

hosts 文件路径：C:\Windows\System32\drivers\etc

192.168.229.30 yum.hdp
192.168.229.31 node-01.hdp
192.168.229.32 node-02.hdp
192.168.229.33 ndoe-03.hpd

centos 系统

[root@node-01 ~]# vi /etc/hosts
192.168.229.30 yum.hdp
192.168.229.31 node-01.hdp
192.168.229.32 node-02.hdp
192.168.229.33 node-03.hdp

配置免密

[root@yum ~]# ssh-keygen -t rsa
[root@yum ~]# ssh-copy-id yum.hdp
[root@yum ~]# ssh-copy-id node-01.hdp
[root@yum ~]# ssh-copy-id node-02.hdp
[root@yum ~]# ssh-copy-id node-03.hdp

[root@node-01 ~]# ssh-keygen -t rsa
[root@node-01 ~]# ssh-copy-id node-01.hdp
[root@node-01 ~]# ssh-copy-id node-02.hdp
[root@node-01 ~]# ssh-copy-id node-03.hdp

[root@node-02 ~]# ssh-keygen -t rsa
[root@node-02 ~]# ssh-copy-id node-02.hdp
[root@node-02 ~]# ssh-copy-id node-03.hdp

安装 mysql

在 node-01.hdp 主机节点上安装 mysql，安装教程

在 yum.hdp 上安装 Ambari

安装 yum 相关工具

[root@yum ~]# yum install yum-utils -y
[root@yum ~]# yum repolist
[root@yum ~]# yum install createrepo -y

上传安装包到 yum.hdp

创建 hdp 目录，将安装包上传到该目录下

[root@yum ~]# mkdir hdp

安装 Apache httpd

httpd 为 Apache HTTP 服务器程序，直接执行程序可启动服务器的服务

[root@yum ~]# yum install httpd -y

安装完成后，会生成 /var/www/html 目录（相当于 Tomcat 的 webapps 目录），进入到 /var/www/html 目录下，创建 ambari 和 hdp 目录，用来存放安装文件

[root@yum ~]# cd /var/www/html/
[root@yum html]# mkdir ambari hdp hdp-utils hdp-gpl

[root@yum ~]# cd /root/hdp/
[root@yum hdp]# tar -zxvf ambari-2.7.3.0-centos7.tar.gz -C /var/www/html/ambari/
[root@yum hdp]# tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp-utils
[root@yum hdp]# tar -zxvf HDP-3.1.0.0-centos7-rpm.tar.gz -C /var/www/html/hdp/
[root@yum hdp]# tar -zxvf HDP-GPL-3.1.0.0-centos7-gpl.tar.gz -C /var/www/html/hdp-gpl/

启动 httpd 服务

# 启动 httpd 服务
[root@yum ~]# systemctl start httpd
 Active: active (running) since 三 2021-05-26 15:24:34 CST; 16s ago #表示服务正在运行
     Docs: man:httpd(8)
           man:apachectl(8)
 Main PID: 1247 (httpd)
# 查看 httpd 服务状态
[root@yum ~]# systemctl status httpd
# 设置 httpd 服务开机启动
[root@yum ~]# systemctl enable httpd

默认端口80，浏览器输入：http://192.168.229.30

配置本地 Repo 数据源

创建 ambari.repo 配置文件

[root@yum yum]# cd /etc/yum.repos.d
[root@yum yum.repos.d]# vim ambari.repo

#VERSION_NUMBER=2.7.3.0-139
[ambari-2.7.3.0]
#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.3.0
#baseurl=http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0
baseurl=http://yum.hdp/ambari/ambari/centos7/2.7.3.0-139/
gpgcheck=1
#gpgkey=http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
gpgkey=http://yum.hdp/ambari/ambari/centos7/2.7.3.0-139/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

创建 hdp.repo 配置文件

[root@yum yum]# cd /etc/yum.repos.d
[root@yum yum.repos.d]# vim hdp.repo

#VERSION_NUMBER=3.1.0.0-78
[HDP-3.1.0.0]
name=HDP Version - HDP-3.1.0.0
#baseurl=http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.0.0
baseurl=http://yum.hdp/hdp/HDP/centos7
gpgcheck=1
#gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
gpgkey=http://yum.hdp/hdp/HDP/centos7/3.1.0.0-78/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
#baseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7
baseurl=http://yum.hdp/hdp-utils/HDP-UTILS/centos7
gpgcheck=1
#gpgkey=http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
gpgkey=http://yum.hdp/hdp-utils/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1

[HDP-GPL-3.1.0.0]
name=HDP-GPL Version - HDP-GPL-3.1.0.0
baseurl=http://yum.hdp/hdp-gpl/HDP-GPL/centos7
gpgcheck=1
gpgkey=http://yum.hdp/hdp-gpl/HDP-GPL/centos7/3.1.0.0-78/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1


- 分发 ambari.repo 和 hdp.repo 配置文件

```bash
[root@yum yum]# cd /etc/yum.repos.d
[root@yum yum]# scp ambari.repo node-01.hdp:$PWD
[root@yum yum]# scp ambari.repo node-02.hdp:$PWD
[root@yum yum]# scp ambari.repo node-03.hdp:$PWD
[root@yum yum]# scp hdp.repo node-01.hdp:$PWD
[root@yum yum]# scp hdp.repo node-02.hdp:$PWD
[root@yum yum]# scp hdp.repo node-03.hdp:$PWD

生成本地源

使用 createrepo 命令，创建 yum 本地源（软件仓库），即为存放本地特定位置的众多rpm包建立索引，描述各包所需依赖信息，并形成元数据

[root@yum yum]# createrepo /var/www/html/hdp/HDP/centos7/
Spawning worker 0 with 101 pkgs
Spawning worker 1 with 100 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

[root@yum yum]# createrepo /var/www/html/hdp-utils/HDP-UTILS/centos7/
Spawning worker 0 with 8 pkgs
Spawning worker 1 with 8 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

[root@yum yum.repos.d]# createrepo /var/www/html/hdp-gpl/HDP-GPL/centos7/
Spawning worker 0 with 2 pkgs
Spawning worker 1 with 2 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

在 node-01.hdp 节点上 mysql 上创建相应的用户和数据库

创建 ambari 数据库及数据库的用户名和密码

[root@node-01 mysql]# mysql -uroot -p
Enter password: #输入数据库密码

mysql> create database ambari character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE USER ‘ambari‘@‘%‘IDENTIFIED BY ‘Ambari123‘;
Query OK, 0 rows affected (0.00 sec)

mysql>  GRANT ALL PRIVILEGES ON ambari.* TO ‘ambari‘@‘%‘;
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

创建hive数据库及hive库的用户名和密码

[root@node-01 mysql]# mysql -uroot -p
Enter password: #输入数据库密码

mysql> create database hive character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE USER ‘hive‘@‘%‘IDENTIFIED BY ‘Hive123‘;
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON hive.* TO ‘hive‘@‘%‘;
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

创建oozie数据库及oozie库的用户名和密码

[root@node-01 mysql]# mysql -uroot -p
Enter password: #输入数据库密码

mysql> create database oozie character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE USER ‘oozie‘@‘%‘IDENTIFIED BY ‘Oozie123‘;
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON oozie.* TO ‘oozie‘@‘%‘;
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

创建ranger数据库及ranger库的用户名和密码（不用创建）

[root@node-01 mysql]# mysql -uroot -p
Enter password: #输入数据库密码

mysql> create database ranger character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> CREATE USER ‘rangeradmin‘@‘%‘IDENTIFIED BY ‘Ranger123‘;
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON ranger.* TO ‘rangeradmin‘@‘%‘;
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

查看所有创建的数据库

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| ambari             |
| hive               |
| mysql              |
| oozie              |
| performance_schema |
| ranger             |
| sys                |
+--------------------+
8 rows in set (0.00 sec)

下载 mysql-connection-java

[root@node-01 mysql]# yum install mysql-connector-java

#查看是否下载 msyql 驱动 jar包
[root@node-01 mysql]# ls /usr/share/java | grep mysql
mysql-connector-java.jar

在 node-01.hdp 节点上安装 Ambari-Server

[root@node-01 ~]# yum install ambari-server
[root@node-01 mysql]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is ‘enabled‘
SELinux mode is ‘enforcing‘
Temporarily disabling SELinux
WARNING: SELinux is set to ‘permissive‘ mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /root/apps/jdk1.8.0_141
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost): node-01.hdp
Port (3306): 3306
Database name (ambari): ambari
Username (root): ambari
Enter Database Password (root):
Re-enter password:
Configuring ambari database...
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.3.0.139.jar
....
Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...
Adjusting ambari-server permissions and ownership...
Ambari Server ‘setup‘ completed successfully.

使用 ambari 登录数据库

[root@node-01 mysql]# mysql -uambari -p
Enter password: #密码为Ambari123
mysql> use ambari;
Database changed
#执行 sql 脚本创建ambari数据库表
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
#查看数据库表是否创建成功
mysql> show tables;

启动Ambari-Server 服务

[root@node-01 mysql]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start........................................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server ‘start‘ completed successfully.

停止 Ambari-Server 服务： # ambari-server stop

安装Ambari-Agent

[root@node-01 ~]# yum -y install ambari-agent
[root@node-02 ~]# yum -y install ambari-agent
[root@node-03 ~]# yum -y install ambari-agent

访问Ambari web页面

默认端口8080，Username：admin；Password：admin；http://192.168.229.31:8080

开始安装集群

启用安装向导创建集群

配置集群名称

选择版本

YUM 本地源如下：

HDP-3.1：http://yum.hdp/hdp/HDP/centos7

HDP-3.1-GPL:http://yum.hdp/hdp-gpl/HDP-GPL/centos7

HDP-UTILS-1.1.0.22:http://yum.hdp/hdp-utils/HDP-UTILS/centos7

配置集群节点、密钥

主机确认

选择大数据组件

节点分配

分配从属与客户端

定制服务

#重新执行命令
[root@node-01 ~]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
Using python  /usr/bin/python
Setup ambari-server
Copying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources/mysql-connector-java.jar
If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.
JDBC driver was successfully initialized.
Ambari Server ‘setup‘ completed successfully.