大数据产品管理平台Apache Ambari研究

2022-01-23 06:51:21

1、环境准备

1.1、配置时间同步

centos7开始使用chrony进行始终同步，安装chrony并配置始终同步，设置为开机启动

yum -y install chrony  #系统默认已经安装，如未安装，请执行以下命令安装

1.2、配置主机名、映射

  • 设置主机名：
[root@cdh1 ~]hostnamectl set-hostname hdp1
[root@cdh2 ~]hostnamectl set-hostname hdp3
[root@cdh3 ~]hostnamectl set-hostname hdp5
  • 设置主机映射：
vi /etc/hosts，添加以下内容【每台都需要】
10.10.101.1 hdp1
10.10.101.3 hdp3
10.10.101.5 hdp5

1.3、取消大叶内存

注意：【所有节点都需要】

sysctl -w vm.swappiness=0 echo "vm.swappiness=0" >>/etc/sysctl.conf echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled

1.4、数据库驱动配置

注意：【所有节点都需要】

第一步：重命名Mysql驱动包，把版本号去掉

mv mysql-connector-java-5.1.44-bin.jar mysql-connector-java.jar

第二步：将jar包移到java共享目录下

mv mysql-connector-java.jar /usr/share/java/

第三步：将Mysql驱动包分发到另外两台服务器

scp mysql-connector-java.jar admin@10.10.101.3:/home/admin/
scp mysql-connector-java.jar admin@10.10.101.5:/home/admin/
sudo mkdir /usr/share/java/
sudo cp /home/admin/mysql-connector-java.jar /usr/share/java/

1.5、安装JDK【/etc/profile】

注意：要设置java环境变量，每个机器都要配置

从10.10.101.4上scp目录：/usr/local/jdk1.8.0_112，可以自行下载解压

  • cd /usr/local/
  • 设置软连接：sudo ln -s jdk1.8.0_112 jdk
  • 设置环境变量：
    1. sudo vi /etc/profile
      • export JAVA_HOME=/usr/local/jdk
      • export PATH=$PATH:$JAVA_HOME/bin
    1. source /etc/profile

1.6、免密

切换root：sudo -i -uroot

查看密钥：cat ~/.ssh/id_rsa

生成密钥：ssh-keygen

ssh-copy-id hdp1

 

ssh-copy-id -i

1.7、关闭防火墙

查看防火墙状态：firewall-cmd -state

停止firewall ：

systemctl stop firewalld.service

禁止firewall开机启动：

systemctl disable firewalld.service

1.8、关闭selinux

  • 获取当前selinux状态：getenforce
    1. Enforcing为开启
    2. Disabled为关闭
  • sudo vim /etc/sysconfig/selinux
  • 替换：SELINUX=disabled
  • 重启：reboot

1.9、设置unlimit参数

注意：【所有节点】

官方建议大于等于10000

cd /etc/security/limits.d/
sudo vi hadoop.conf
  • 输入以下内容
*        soft    noproc 65535
*        hard    noproc 65535
*        soft    nofile 65535
*        hard    nofile 65535
  • 退出当前会话，重新登录生效

 

2、制作本地yum源

使用【10.10.101.251】,我的实验环境服务器，需替换。

2.1、安装包准备

2.2、yum文件准备

下载方式：

本地方式：

在各自的安装包中存在。比如：ambari中：ambari/centos7/2.6.2.2-1/ambari.repo

2.3、安装http

yum install httpd
systemctl start httpd
systemctl enable httpd

2.4、制作源

mkdir /var/www/html/ambari/HDP -P
mkdir /var/www/html/ambari/HDP-UTILS
cd

2.5、编辑yum文件

cd /etc/yum.repos.d
sudo vi ambari.repo
#VERSION_NUMBER=2.6.2.2-1
[ambari-2.6.2.2]
name=ambari Version - ambari-2.6.2.2
baseurl=http://10.10.101.251/hdp/ambari/centos7/2.6.2.2-1/
gpgcheck=1
gpgkey=http://10.10.101.251/hdp/ambari/centos7/2.6.2.2-1/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
sudo vi hdp.repo
#VERSION_NUMBER=2.6.5.0-292
[HDP-2.6.5.0]
name=HDP Version - HDP-2.6.5.0
baseurl=http://10.10.101.251/hdp/HDP/centos7/2.6.5.0-292/
gpgcheck=1
gpgkey=http://10.10.101.251/hdp/HDP/centos7/2.6.5.0-292/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://10.10.101.251/hdp/HDP-UTILS/centos7/1.1.0.22/
gpgcheck=1
gpgkey=http://10.10.101.251/hdp/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

 

sudo vi hdp.gpl.repo
#VERSION_NUMBER=2.6.5.0-292
[HDP-GPL-2.6.5.0]
name=HDP-GPL Version - HDP-GPL-2.6.5.0
baseurl=http://10.10.101.251/hdp/HDP-GPL/centos7/2.6.5.0-292/
gpgcheck=1
gpgkey=http://10.10.101.251/hdp/HDP-GPL/centos7/2.6.5.0-292/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

3、数据库准备

此处不再赘述，实验环境使用存在的数据库：ambari_dianxin

  • 地址：10.10.101.10
  • User：ambari_dianxin
  • Pwd：ambari_dianxin
  • 进入数据库：mysql -uambari_dianxin -pambari_dianxin

4、安装ambari-server

  • 下载：
sudo yum install -y ambari-server
  • 设置：
sudo ambari-server setup
[admin@hdp1 yum.repos.d]$ sudo ambari-server reset
Using python  /usr/bin/python
Resetting ambari-server
**** WARNING **** You are about to reset and clear the Ambari Server database. This will remove all cluster host and configuration information from the database. You will be required to re-configure the Ambari server and re-run the cluster wizard.
Are you SURE you want to perform the reset [yes/no] (no)? yes
ERROR: Exiting with exit code 1.
REASON: Ambari doesn't support resetting exernal DB automatically. To reset Ambari Server schema you must first drop and then create it using DDL scripts from "/var/lib/ambari-server/resources/"
[admin@hdp1 yum.repos.d]$ sudo ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Ambari-server daemon is configured to run under user 'ambari'. Change this setting [y/n] (n)?
Adjusting ambari-server permissions and ownership...
Checking firewall status...
WARNING: iptables is running. Confirm the necessary Ambari ports are accessible. Refer to the Ambari documentation for more details on ports.
OK to continue [y/n] (y)?
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/jdk
Validating JDK on Ambari Server...done.
Checking GPL software agreement...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (3):
Hostname (10.10.101.10):
Port (3309):
Database name (ambari_dianxin):
Username (ambari_dianxin):
Enter Database Password (ambari_dianxin):
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
............
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.


  • 执行sql：由于sql文件与数据库不在一个节点，所以需要转发过去。
scp /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql admin@10.10.101.10:/home/admin/
  • 执行ambari的建表语句：
source /home/admin/Ambari-DDL-MySQL-CREATE.sql
  • 启动服务：
ambari-server start
  • 成功提示：Ambari Server 'start' completed successfully
  • 在界面验证：http://10.10.1.1:8080 账户：admin，密码：admin
  • 失败查看日志：/var/log/ambari-server/ambari-server.log

5、搭建集群

登录Ambari界面：http://10.10.1.1:8080

大数据产品管理平台Apache Ambari研究

大数据产品管理平台Apache Ambari研究

大数据产品管理平台Apache Ambari研究

大数据产品管理平台Apache Ambari研究

输入地址：（IP是我的实验环境服务器，需替换）

http://10.10.101.251/hdp/HDP/centos7/2.6.5.0-292/

http://10.10.101.251/hdp/HDP-UTILS/centos7/1.1.0.22/

大数据产品管理平台Apache Ambari研究

查看密钥：

cat ~/.ssh/id_rsa

然后输入到文本框

大数据产品管理平台Apache Ambari研究

如果这一步出现如下错误，通过以下方式解决，

错误：

ERROR 2018-05-30 00:12:25,280 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)

ERROR 2018-05-30 00:12:25,280 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.

解决方式：

1.修改/etc/python/cert-verification.cfg配置文件：

# vim /etc/python/cert-verification.cfg

[https]

verify=disable

2、编辑 /etc/ambari-agent/conf/ambari-agent.ini 配置文件，在 [security] 节部分，确保设置如下两个值，其它值保持不变：

    [root@ambari ~]# vi /etc/ambari-agent/conf/ambari-agent.ini

    [security]

    ssl_verify_cert=0

    force_https_protocol=PROTOCOL_TLSv1_2

保存退出，重启 ambari-agent：

    [root@ambari ~]# ambari-agent restart

若还是不能注册ambari-agent,使用下面介绍的方式。

如果上面方式还不能注册，说明jdk版本不对，使用默认oracle啊jdk，（2.6.2.2版本ambari需要使用java version "1.8.0_112"）


如果一个节点重启后agent无法连接的话解决方法如下

ambari管理大数据集群，节点失去心跳，操作方法：
1、systemctl stop ambari-agent

2、在失去心跳节点打开配置
vi /etc/ambari-agent/conf/ambari-agent.ini
在[security] 下添加
force_https_protocol=PROTOCOL_TLSv1_2

3、关闭状态
vi /etc/python/cert-verification.cfg
如下：
[https]
verify=disable

4、systemctl stop ambari-agent


5.1、安装HDP

选择服务，全部为默认配置【除开密码配置以及数据库配置】

大数据产品管理平台Apache Ambari研究

大数据产品管理平台Apache Ambari研究

大数据产品管理平台Apache Ambari研究

大数据产品管理平台Apache Ambari研究

如果测试的时候报错，则设置mysql驱动：

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

大数据产品管理平台Apache Ambari研究

5.2、设置oozie

大数据产品管理平台Apache Ambari研究

大数据产品管理平台Apache Ambari研究

大数据产品管理平台Apache Ambari研究

大数据产品管理平台Apache Ambari研究

大数据产品管理平台Apache Ambari研究

大数据产品管理平台Apache Ambari研究

大数据产品管理平台Apache Ambari研究

等待安装即可。

6、卸载Ambari

停服务

sudo ambari-agent stop
sudo ambari-server stop

卸载所有组件：

sudo yum remove -y hadoop_2* hdp-select* ranger_2* zookeeper* bigtop*atlas-metadata* ambari* spark* slide* strom* hive* oozie_2*

如有遗漏自行添加

查看ambari是否被卸载

rpm -qa|grep ambari
rpm -e ambari-server-2.6.2.2-1.x86_64

删除目录：

sudo rm -rf /var/lib/ambari*
sudo rm -rf /usr/lib/python2.6/site-packages/ambari_*
sudo rm -rf /usr/lib/python2.6/site-packages/resource_management
sudo rm -rf /usr/lib/ambari-*
 
sudo rm -rf /etc/ambari-*
sudo rm -rf /etc/hadoop
sudo rm -rf /etc/hbase
sudo rm -rf /etc/hive
sudo rm -rf /etc/hive2
sudo rm -rf /etc/oozie
sudo rm -rf /etc/sqoop
sudo rm -rf /etc/zookeeper
sudo rm -rf /etc/flume
sudo rm -rf /etc/storm
sudo rm -rf /etc/tez_hive2
sudo rm -rf /etc/spark2
sudo rm -rf /etc/phoenix
sudo rm -rf /etc/pig
sudo rm -rf /etc/hive-hcatalog
sudo rm -rf /etc/tez
sudo rm -rf /etc/falcon
sudo rm -rf /etc/knox
sudo rm -rf /etc/hive-webhcat
sudo rm -rf /etc/kafka
sudo rm -rf /etc/slider
sudo rm -rf /etc/storm-slider-client
sudo rm -rf /etc/spark
 
sudo rm -rf /var/run/spark
sudo rm -rf /var/run/hadoop
sudo rm -rf /var/run/hbase
sudo rm -rf /var/run/zookeeper
sudo rm -rf /var/run/flume
sudo rm -rf /var/run/storm
sudo rm -rf /var/run/webhcat
sudo rm -rf /var/run/hadoop-yarn
sudo rm -rf /var/run/hadoop-mapreduce
sudo rm -rf /var/run/kafka
sudo rm -rf /var/run/hive        
sudo rm -rf /var/run/oozie      
sudo rm -rf /var/run/sqoop     
sudo rm -rf /var/run/hive-hcatalog
sudo rm -rf /var/run/falcon     
sudo rm -rf /var/run/hadoop-hdfs  
sudo rm -rf /var/run/ambari-metrics-collector
sudo rm -rf /var/run/ambari-metrics-monitor        
sudo rm -rf /var/log/hadoop-hdfs   
sudo rm -rf /var/log/hive-hcatalog
sudo rm -rf /var/log/ambari-metrics-monitor
sudo rm -rf /var/log/hadoop
sudo rm -rf /var/log/hbase
sudo rm -rf /var/log/flume
sudo rm -rf /var/log/sqoop
sudo rm -rf /var/log/ambari-server
sudo rm -rf /var/log/ambari-agent
sudo rm -rf /var/log/storm
sudo rm -rf /var/log/hadoop-yarn
sudo rm -rf /var/log/hadoop-mapreduce
sudo rm -rf /var/log/knox
sudo rm -rf /var/lib/slider
 
sudo rm -rf /var/lib/pgsql/
sudo rm -rf /usr/lib/flume
sudo rm -rf /usr/lib/storm
sudo rm -rf /var/lib/hive
sudo rm -rf /var/lib/oozie
sudo rm -rf /var/lib/flume
sudo rm -rf /var/lib/hadoop-yarn
sudo rm -rf /var/lib/hadoop-mapreduce
sudo rm -rf /var/lib/hadoop-hdfs
sudo rm -rf /var/lib/zookeeper
sudo rm -rf /var/lib/knox
sudo rm -rf /var/log/hive
sudo rm -rf /var/log/oozie
sudo rm -rf /var/log/zookeeper
sudo rm -rf /var/log/falcon
sudo rm -rf /var/log/webhcat
sudo rm -rf /var/log/spark
sudo rm -rf /var/tmp/oozie
sudo rm -rf /tmp/ambari-qa
sudo rm -rf /tmp/hive
sudo rm -rf /var/hadoop
sudo rm -rf /hadoop/falcon
sudo rm -rf /tmp/hadoop
sudo rm -rf /tmp/hadoop-hdfs
sudo rm -rf /usr/hdp
sudo rm -rf /usr/hadoop
sudo rm -rf /opt/hadoop
sudo rm -rf /tmp/hadoop
sudo rm -rf /var/hadoop
sudo rm -rf /hadoop
 
sudo rm -rf /usr/bin/worker-lanucher
sudo rm -rf /usr/bin/zookeeper-client
sudo rm -rf /usr/bin/zookeeper-server
sudo rm -rf /usr/bin/zookeeper-server-cleanup
sudo rm -rf /usr/bin/yarn
sudo rm -rf /usr/bin/storm
sudo rm -rf /usr/bin/storm-slider
sudo rm -rf /usr/bin/worker-lanucher
sudo rm -rf /usr/bin/storm
sudo rm -rf /usr/bin/storm-slider
sudo rm -rf /usr/bin/sqoop
sudo rm -rf /usr/bin/sqoop-codegen
sudo rm -rf /usr/bin/sqoop-create-hive-table
sudo rm -rf /usr/bin/sqoop-eval
sudo rm -rf /usr/bin/sqoop-export
sudo rm -rf /usr/bin/sqoop-help
sudo rm -rf /usr/bin/sqoop-import
sudo rm -rf /usr/bin/sqoop-import-all-tables
sudo rm -rf /usr/bin/sqoop-job
sudo rm -rf /usr/bin/sqoop-list-databases
sudo rm -rf /usr/bin/sqoop-list-tables
sudo rm -rf /usr/bin/sqoop-merge
sudo rm -rf /usr/bin/sqoop-metastore
sudo rm -rf /usr/bin/sqoop-version
sudo rm -rf /usr/bin/slider
sudo rm -rf /usr/bin/ranger-admin-start
sudo rm -rf /usr/bin/ranger-admin-stop
sudo rm -rf /usr/bin/ranger-kms
sudo rm -rf /usr/bin/ranger-usersync-start
sudo rm -rf /usr/bin/ranger-usersync-stop
sudo rm -rf /usr/bin/pig
sudo rm -rf /usr/bin/phoenix-psql
sudo rm -rf /usr/bin/phoenix-queryserver
sudo rm -rf /usr/bin/phoenix-sqlline
sudo rm -rf /usr/bin/phoenix-sqlline-thin
sudo rm -rf /usr/bin/oozie
sudo rm -rf /usr/bin/oozied.sh
sudo rm -rf /usr/bin/mapred
sudo rm -rf /usr/bin/mahout
sudo rm -rf /usr/bin/kafka
sudo rm -rf /usr/bin/hive
sudo rm -rf /usr/bin/hiveserver2
sudo rm -rf /usr/bin/hbase
sudo rm -rf /usr/bin/hcat
sudo rm -rf /usr/bin/hdfs
sudo rm -rf /usr/bin/hadoop
sudo rm -rf /usr/bin/flume-ng
sudo rm -rf /usr/bin/falcon
sudo rm -rf /usr/bin/beeline
sudo rm -rf /usr/bin/atlas-start
sudo rm -rf /usr/bin/atlas-stop
sudo rm -rf /usr/bin/accumulo

如有遗漏自行添加


如果安装过程中在某个组件报错，那么直接删除对应组件的所有东西，重新安装即可，比如：symlink target /usr/hdp/current/oozie-client for oozie already exists and it is not a symlink.

删除安装包：

sudo yum remove oozie_2* -y

找到oozie的rpm包

rpm -qa|grep oozie

删除包

sudo rpm -e oozie_2_6_5_0_292-4.2.0.2.6.5.0-292.noarch

如果报错：/var/tmp/rpm-tmp.YhTbCT: line 1: pushd: /usr/hdp/2.6.5.0-292/oozie: No such file or directory

先创建目录：

sudo mkdir –p /usr/hdp/2.6.5.0-292/oozie

再删除：

sudo rpm -e oozie_2_6_5_0_292-4.2.0.2.6.5.0-292.noarch


删除目录：

sudo rm -rf /usr/hdp/current/oozie-client/
sudo rm -rf /usr/hdp/2.6.5.0-292/oozie
sudo rm -rf /etc/oozie/
sudo rm -rf /var/lib/oozie/
sudo rm -rf /var/log/oozie/
sudo userdel oozie
sudo rm -rf /home/oozie


THE END




上一篇：预防SQL注入攻击之我见


下一篇：php foreach用法和实例