Cloudera-manager(CDH6.3.0)大数据平台搭建一指禅(impala,kudu,hdfs,hive,kafka,yarn,spark,hbase,hue)

Cloudera-manager(CDH6.3.0)大数据平台搭建一指禅指南(impala,kudu,hdfs,hive,kafka,yarn,spark,hbase,hue)

CHD6,大量hadoop生态的重大更新升级,果断把现有系统升级到CHD6上。

准备:
192.168.88.31           master--》机器名修改为:manager 还是重复的master把
192.168.88.32           utility
192.168.88.33           gateway
192.168.88.34           worker1
每台虚拟机8g内存(不想给自己找麻烦,就搞8G内存,否则会卡主各种莫名其妙的问题的),建议最低4台测试。3台虚拟机也是可以安装环境的。
[root@manager ~]# vi /etc/sysconfig/network
HOSTNAME=master

# [root@manager ~]# vi /etc/hosts
127.0.0.1   localhost manager localhost4 localhost4.localdomain4                 (需要删除)
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6   (需要删除)

//永久性的修改主机名称,重启后能保持修改后的。
[root@manager ~]# hostnamectl set-hostname master
[root@manager ~]# cat /etc/hostname

[root@manager ~]# reboot

[root@manager ~]# hostname
manager

设置静态IP、DNS、网关

[hadoop@utility ~]$ sudo vi /etc/sysconfig/network-scripts/ifcfg-ens33
#BOOTPROTO=dhcp
BOOTPROTO=static #设置网卡获得ip地址的方式,可能的选项为static,dhcp或bootp,分别对应静态指定的 ip地址,通过dhcp协议获得的ip地址,通过bootp协议获得的ip地址
IPADDR=192.168.88.31 #如果设置网卡获得 ip地址的方式为静态指定,此字段就指定了网卡对应的ip地址
NETMASK=255.255.255.0 #网卡对应的网络掩码
NETWORK=192.168.88.0 #网卡对应的网络地址
GATEWAY=192.168.88.2  #有时候叫做default route address
DNS1=192.168.88.2

[hadoop@utility ~]$ sudo service network restart
[hadoop@utility ~]$ sudo ifconfig -a
[hadoop@utility ~]$ sudo nslookup www.baidu.com

[hadoop@utility ~]$ sudo nmcli con mod ens33 ipv4.dns "192.168.88.2"
[hadoop@utility ~]$ sudo nmcli con up ens33
[hadoop@utility ~]$ cat /etc/resolv.conf

给每台机器的hadoop用户添加sudo免密码的功能

[root@localhost ~]# vi /etc/sudoers
添加如下的声明:
hadoop ALL=(ALL) NOPASSWD: ALL

[hadoop@manager ~]$ sudo yum install -y vim wget ntp net-tools

给每台机器都增加hosts文件的配置,修改完成后,相互之间可以ping通

[hadoop@manager ~]$ sudo vi /etc/hosts
192.168.88.31 master
192.168.88.32 utility
192.168.88.33 gateway
192.168.88.34 worker1
在这里有个坑需要注意的,以前初始的hosts文件中是有
127.0.0.1   localhost manager localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
的,这个是需要删除的,否则后面安装报错。

[hadoop@manager ~]$ ping master
[hadoop@manager ~]$ ping www.baidu.com

每台主机关闭防火墙(折行写的命令最后不能有空格,否则报错)

[hadoop@manager ~]$
sudo systemctl stop firewalld \
&& sudo systemctl disable firewalld \
&& sudo systemctl status firewalld

每台机器关闭selinux

[hadoop@manager ~]$ sudo setenforce 0
[hadoop@manager ~]$ sudo vim /etc/selinux/config
SELINUX=permissive

[hadoop@manager ~]$ getenforce
Permissive

每台机器配置NTP服务

[hadoop@manager ~]$ sudo vim /etc/ntp.conf
#将时钟服务器更改为如下4个
server 0.cn.pool.ntp.org
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org
server 3.cn.pool.ntp.org

[hadoop@manager ~]$
sudo systemctl start ntpd \
&& sudo systemctl enable ntpd \
&& sudo ntpdate -u 0.cn.pool.ntp.org \
&& sudo hwclock --systohc \
&& sudo date

在tools主机上安装(生产环境,减轻master的负担,不过在我本机开发,都是用的master,所以这里我用master进行安装)

上传下载的文件到

[hadoop@manager cloudera]$ pwd
/home/hadoop/Downloads/cloudera
cloudera/
├── cloudera-manager
│   ├── allkeys.asc
│   ├── cloudera-manager-agent-6.3.0-1281944.el7.x86_64.rpm
│   ├── cloudera-manager-daemons-6.3.0-1281944.el7.x86_64.rpm
│   ├── cloudera-manager-server-6.3.0-1281944.el7.x86_64.rpm
│   ├── cloudera-manager-server-db-2-6.3.0-1281944.el7.x86_64.rpm
│   ├── enterprise-debuginfo-6.3.0-1281944.el7.x86_64.rpm
│   └── oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
├── mysql
│   └── mysql-5.7.27-1.el7.x86_64.rpm-bundle.tar
├── mysql-jdbc
│   └── mysql-connector-java-5.1.47.tar.gz
└── parcel
    ├── CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel
    └── manifest.json

安装httpd服务,以及为本地创建yum安装的库做准备

[hadoop@manager cloudera]$
sudo yum -y install httpd createrepo \
&& sudo systemctl start httpd \
&& sudo systemctl enable httpd \
&& sudo cd /home/hadoop/Downloads/cloudera/cloudera-manager/ \
&& sudo createrepo . \
&& sudo mv /home/hadoop/Downloads/cloudera/cloudera-manager/ /var/www/html/

Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.njupt.edu.cn
 * extras: mirrors.njupt.edu.cn
 * updates: mirrors.njupt.edu.cn
Package createrepo-0.9.9-28.el7.noarch already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package httpd.x86_64 0:2.4.6-90.el7.centos will be installed
--> Processing Dependency: httpd-tools = 2.4.6-90.el7.centos for package: httpd-2.4.6-90.el7.centos.x86_64
--> Running transaction check
---> Package httpd-tools.x86_64 0:2.4.6-90.el7.centos will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
 Package                                             Arch                                           Version                                                     Repository                                    Size
===================================================================================================================================================================================================================
Installing:
 httpd                                               x86_64                                         2.4.6-90.el7.centos                                         base                                         2.7 M
Installing for dependencies:
 httpd-tools                                         x86_64                                         2.4.6-90.el7.centos                                         base                                          91 k

Transaction Summary
===================================================================================================================================================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 2.8 M
Installed size: 9.5 M
Downloading packages:
(1/2): httpd-2.4.6-90.el7.centos.x86_64.rpm                                                                                                                                                 | 2.7 MB  00:00:01     
httpd-tools-2.4.6-90.el7.cento FAILED                                          
http://mirrors.cqu.edu.cn/CentOS/7.7.1908/os/x86_64/Packages/httpd-tools-2.4.6-90.el7.centos.x86_64.rpm: [Errno 14] curl#6 - "Could not resolve host: mirrors.cqu.edu.cn; Unknown error"B/s | 2.2 MB  00:00:00 ETA 
Trying other mirror.
(2/2): httpd-tools-2.4.6-90.el7.centos.x86_64.rpm                                                                                                                                           |  91 kB  00:00:00     
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                              271 kB/s | 2.8 MB  00:00:10     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : httpd-tools-2.4.6-90.el7.centos.x86_64                                                                                                                                                          1/2 
  Installing : httpd-2.4.6-90.el7.centos.x86_64                                                                                                                                                                2/2 
  Verifying  : httpd-tools-2.4.6-90.el7.centos.x86_64                                                                                                                                                          1/2 
  Verifying  : httpd-2.4.6-90.el7.centos.x86_64                                                                                                                                                                2/2 

Installed:
  httpd.x86_64 0:2.4.6-90.el7.centos                                                                                                                                                                               

Dependency Installed:
  httpd-tools.x86_64 0:2.4.6-90.el7.centos                                                                                                                                                                         

Complete!
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.
Spawning worker 0 with 2 pkgs
Spawning worker 1 with 2 pkgs
Spawning worker 2 with 1 pkgs
Spawning worker 3 with 1 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
[hadoop@manager ~]$

[root@manager cloudera-manager]# pwd
/var/www/html/cloudera-manager
[root@manager cloudera-manager]# createrepo .
Spawning worker 0 with 2 pkgs
Spawning worker 1 with 2 pkgs
Spawning worker 2 with 1 pkgs
Spawning worker 3 with 1 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
[root@manager cloudera-manager]# ls -al
total 1378008
drwxr-xr-x. 3 hadoop hadoop       4096 Oct 31 00:17 .
drwxr-xr-x. 3 root   root           30 Oct 30 23:03 ..
-rw-r--r--. 1 hadoop hadoop      14041 Oct 30 20:25 allkeys.asc
-rw-r--r--. 1 hadoop hadoop   10479136 Oct 30 20:25 cloudera-manager-agent-6.3.0-1281944.el7.x86_64.rpm
-rw-r--r--. 1 hadoop hadoop 1201341068 Oct 30 20:28 cloudera-manager-daemons-6.3.0-1281944.el7.x86_64.rpm
-rw-r--r--. 1 hadoop hadoop      11464 Oct 30 20:25 cloudera-manager-server-6.3.0-1281944.el7.x86_64.rpm
-rw-r--r--. 1 hadoop hadoop      10996 Oct 30 20:25 cloudera-manager-server-db-2-6.3.0-1281944.el7.x86_64.rpm
-rw-r--r--. 1 hadoop hadoop   14209884 Oct 30 20:25 enterprise-debuginfo-6.3.0-1281944.el7.x86_64.rpm
-rw-r--r--. 1 hadoop hadoop  184988341 Oct 30 20:26 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
drwxr-xr-x. 2 root   root         4096 Oct 31 00:17 repodata                                                -----》这个目录要检查一下,否则将来的本地yum是没法通过的
[root@manager cloudera-manager]#
完成后,我们使用firefox打开--》http://192.168.88.31/cloudera-manager/     ,之后我们要能看到一个网页的文件列表
Index of /cloudera-manager
[ICO]    Name    Last modified    Size    Description
[PARENTDIR]    Parent Directory          -      
[TXT]    allkeys.asc     2019-10-30 20:25     14K     
[ ]    cloudera-manager-age..>    2019-10-30 20:25     10M     
[ ]    cloudera-manager-dae..>    2019-10-30 20:28     1.1G     
[ ]    cloudera-manager-ser..>    2019-10-30 20:25     11K     
[ ]    cloudera-manager-ser..>    2019-10-30 20:25     11K     
[ ]    enterprise-debuginfo..>    2019-10-30 20:25     14M     
[ ]    oracle-j2sdk1.8-1.8...>    2019-10-30 20:26     176M     
[DIR]    repodata/                                                      ----》这个是需要检查的,如果没有,将来是会报错的,必须检查

安装JDK

[hadoop@manager cloudera-manager]$ java -version
openjdk version "1.8.0_222-ea"
OpenJDK Runtime Environment (build 1.8.0_222-ea-b03)
OpenJDK 64-Bit Server VM (build 25.222-b03, mixed mode)

[hadoop@manager cloudera-manager]$ rpm -qa | grep java
java-1.7.0-openjdk-headless-1.7.0.221-2.6.18.1.el7.x86_64
javamail-1.4.6-8.el7.noarch
tzdata-java-2019b-1.el7.noarch
java-1.8.0-openjdk-1.8.0.222.b03-1.el7.x86_64
java-1.8.0-openjdk-headless-1.8.0.222.b03-1.el7.x86_64
python-javapackages-3.4.1-11.el7.noarch
javapackages-tools-3.4.1-11.el7.noarch
java-1.7.0-openjdk-1.7.0.221-2.6.18.1.el7.x86_64
javassist-3.16.1-10.el7.noarch

原来安装的统统都删除掉,但是noarch文件可以不用删除

[hadoop@manager cloudera-manager]$ 
sudo rpm -e --nodeps java-1.7.0-openjdk-headless-1.7.0.221-2.6.18.1.el7.x86_64 \
&& sudo rpm -e --nodeps java-1.8.0-openjdk-1.8.0.222.b03-1.el7.x86_64 \
&& sudo rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.222.b03-1.el7.x86_64 \
&& sudo rpm -e --nodeps java-1.7.0-openjdk-1.7.0.221-2.6.18.1.el7.x86_64


[hadoop@manager cloudera-manager]$ sudo rpm -ivh /var/www/html/cloudera-manager/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
warning: /var/www/html/cloudera-manager/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:oracle-j2sdk1.8-1.8.0+update181-1################################# [100%]
[hadoop@manager jdk1.8.0_181-cloudera]$ pwd
/usr/java/jdk1.8.0_181-cloudera

[hadoop@manager jdk1.8.0_181-cloudera]$ sudo vi /etc/profile
# set java environment
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:${PATH}

[hadoop@manager jdk1.8.0_181-cloudera]$ source /etc/profile
[hadoop@manager jdk1.8.0_181-cloudera]$ java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

安装mysql

[hadoop@manager ~]$ sudo yum -y remove mariadb-libs
Loaded plugins: fastestmirror, langpacks
Resolving Dependencies
--> Running transaction check
---> Package mariadb-libs.x86_64 1:5.5.64-1.el7 will be erased
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: perl-DBD-MySQL-4.023-6.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: 1:net-snmp-agent-libs-5.7.2-43.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: 2:postfix-2.10.1-7.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: 1:net-snmp-5.7.2-43.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: 1:qt-mysql-4.8.7-3.el7_6.x86_64
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: perl-DBD-MySQL-4.023-6.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: 1:net-snmp-agent-libs-5.7.2-43.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: 2:postfix-2.10.1-7.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: 1:qt-mysql-4.8.7-3.el7_6.x86_64
--> Processing Dependency: mariadb-libs(x86-64) = 1:5.5.64-1.el7 for package: 1:mariadb-server-5.5.64-1.el7.x86_64
--> Processing Dependency: mariadb-libs(x86-64) = 1:5.5.64-1.el7 for package: 1:mariadb-5.5.64-1.el7.x86_64
--> Processing Dependency: mysql-libs for package: 1:net-snmp-5.7.2-43.el7.x86_64
--> Running transaction check
---> Package mariadb.x86_64 1:5.5.64-1.el7 will be erased
---> Package mariadb-server.x86_64 1:5.5.64-1.el7 will be erased
--> Processing Dependency: mariadb-server for package: akonadi-mysql-1.9.2-4.el7.x86_64
---> Package net-snmp.x86_64 1:5.7.2-43.el7 will be erased
---> Package net-snmp-agent-libs.x86_64 1:5.7.2-43.el7 will be erased
--> Processing Dependency: libnetsnmpagent.so.31()(64bit) for package: keepalived-1.3.5-16.el7.x86_64
--> Processing Dependency: libnetsnmpmibs.so.31()(64bit) for package: keepalived-1.3.5-16.el7.x86_64
---> Package perl-DBD-MySQL.x86_64 0:4.023-6.el7 will be erased
---> Package postfix.x86_64 2:2.10.1-7.el7 will be erased
---> Package qt-mysql.x86_64 1:4.8.7-3.el7_6 will be erased
--> Running transaction check
---> Package akonadi-mysql.x86_64 0:1.9.2-4.el7 will be erased
---> Package keepalived.x86_64 0:1.3.5-16.el7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
 Package                                                  Arch                                        Version                                                 Repository                                      Size
===================================================================================================================================================================================================================
Removing:
 mariadb-libs                                             x86_64                                      1:5.5.64-1.el7                                          @anaconda                                      4.4 M
Removing for dependencies:
 akonadi-mysql                                            x86_64                                      1.9.2-4.el7                                             @anaconda                                      9.6 k
 keepalived                                               x86_64                                      1.3.5-16.el7                                            @anaconda                                      1.0 M
 mariadb                                                  x86_64                                      1:5.5.64-1.el7                                          @anaconda                                       49 M
 mariadb-server                                           x86_64                                      1:5.5.64-1.el7                                          @anaconda                                       58 M
 net-snmp                                                 x86_64                                      1:5.7.2-43.el7                                          @anaconda                                      882 k
 net-snmp-agent-libs                                      x86_64                                      1:5.7.2-43.el7                                          @anaconda                                      2.1 M
 perl-DBD-MySQL                                           x86_64                                      4.023-6.el7                                             @anaconda                                      323 k
 postfix                                                  x86_64                                      2:2.10.1-7.el7                                          @anaconda                                       12 M
 qt-mysql                                                 x86_64                                      1:4.8.7-3.el7_6                                         @anaconda                                       74 k

Transaction Summary
===================================================================================================================================================================================================================
Remove  1 Package (+9 Dependent packages)

Installed size: 128 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
icedtea-web-1.7.1-1.el7.x86_64 has missing requires of java-1.8.0-openjdk
  Erasing    : akonadi-mysql-1.9.2-4.el7.x86_64                                                                                                                                                               1/10 
  Erasing    : 1:net-snmp-5.7.2-43.el7.x86_64                                                                                                                                                                 2/10 
  Erasing    : 1:mariadb-server-5.5.64-1.el7.x86_64                                                                                                                                                           3/10 
  Erasing    : 1:mariadb-5.5.64-1.el7.x86_64                                                                                                                                                                  4/10 
  Erasing    : perl-DBD-MySQL-4.023-6.el7.x86_64                                                                                                                                                              5/10 
  Erasing    : 1:qt-mysql-4.8.7-3.el7_6.x86_64                                                                                                                                                                6/10 
  Erasing    : 2:postfix-2.10.1-7.el7.x86_64                                                                                                                                                                  7/10 
  Erasing    : keepalived-1.3.5-16.el7.x86_64                                                                                                                                                                 8/10 
  Erasing    : 1:net-snmp-agent-libs-5.7.2-43.el7.x86_64                                                                                                                                                      9/10 
  Erasing    : 1:mariadb-libs-5.5.64-1.el7.x86_64                                                                                                                                                            10/10 
  Verifying  : 1:mariadb-libs-5.5.64-1.el7.x86_64                                                                                                                                                             1/10 
  Verifying  : 1:mariadb-5.5.64-1.el7.x86_64                                                                                                                                                                  2/10 
  Verifying  : 1:qt-mysql-4.8.7-3.el7_6.x86_64                                                                                                                                                                3/10 
  Verifying  : keepalived-1.3.5-16.el7.x86_64                                                                                                                                                                 4/10 
  Verifying  : akonadi-mysql-1.9.2-4.el7.x86_64                                                                                                                                                               5/10 
  Verifying  : 2:postfix-2.10.1-7.el7.x86_64                                                                                                                                                                  6/10 
  Verifying  : perl-DBD-MySQL-4.023-6.el7.x86_64                                                                                                                                                              7/10 
  Verifying  : 1:net-snmp-agent-libs-5.7.2-43.el7.x86_64                                                                                                                                                      8/10 
  Verifying  : 1:mariadb-server-5.5.64-1.el7.x86_64                                                                                                                                                           9/10 
  Verifying  : 1:net-snmp-5.7.2-43.el7.x86_64                                                                                                                                                                10/10 

Removed:
  mariadb-libs.x86_64 1:5.5.64-1.el7                                                                                                                                                                               

Dependency Removed:
  akonadi-mysql.x86_64 0:1.9.2-4.el7              keepalived.x86_64 0:1.3.5-16.el7          mariadb.x86_64 1:5.5.64-1.el7       mariadb-server.x86_64 1:5.5.64-1.el7       net-snmp.x86_64 1:5.7.2-43.el7      
  net-snmp-agent-libs.x86_64 1:5.7.2-43.el7       perl-DBD-MySQL.x86_64 0:4.023-6.el7       postfix.x86_64 2:2.10.1-7.el7       qt-mysql.x86_64 1:4.8.7-3.el7_6           

Complete!

[hadoop@manager ~]$ cd /home/hadoop/Downloads/cloudera/mysql/;tar -xvf mysql-5.7.27-1.el7.x86_64.rpm-bundle.tar
mysql-community-libs-5.7.27-1.el7.x86_64.rpm
mysql-community-embedded-devel-5.7.27-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm
mysql-community-devel-5.7.27-1.el7.x86_64.rpm
mysql-community-embedded-compat-5.7.27-1.el7.x86_64.rpm
mysql-community-common-5.7.27-1.el7.x86_64.rpm
mysql-community-client-5.7.27-1.el7.x86_64.rpm
mysql-community-server-5.7.27-1.el7.x86_64.rpm
mysql-community-test-5.7.27-1.el7.x86_64.rpm
mysql-community-embedded-5.7.27-1.el7.x86_64.rpm

[hadoop@manager mysql]$ pwd
/home/hadoop/Downloads/cloudera/mysql
[hadoop@manager ~]$
sudo yum install -y libaio \
&& sudo rpm -ivh mysql-community-common-5.7.27-1.el7.x86_64.rpm \
&& sudo rpm -ivh mysql-community-libs-5.7.27-1.el7.x86_64.rpm \
&& sudo rpm -ivh mysql-community-client-5.7.27-1.el7.x86_64.rpm \
&& sudo rpm -ivh mysql-community-server-5.7.27-1.el7.x86_64.rpm \
&& sudo rpm -ivh mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm

[root@manager ~]# echo character-set-server=utf8 >> /etc/my.cnf

这里的数据库密码【123456Aa.】,比较重要,后面都会用到这个密码的
数据库名称也是比较重要的,后面也会要求填写这些数据库名称的(也就是在安装CDH之前,就要想好要安装哪些hadoop组件,一个组件,一个mysql库)

[hadoop@manager mysql]$
(
cat <<EOF
set password for root@localhost = password('123456Aa.');
grant all privileges on *.* to 'root'@'%' identified by '123456Aa.';
flush privileges;
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456Aa.';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456Aa.';
SHOW DATABASES;
EOF
) >> /home/hadoop/Downloads/cloudera/c.sql

[root@manager ~]# systemctl enable mysqld && systemctl start mysqld && grep password /var/log/mysqld.log | sed 's/.*\(............\)$/\1/'
dqfwl;rFO4>o               --》首次登录的密码

mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY '123xxxxxx';
Query OK, 0 rows affected (0.01 sec)

mysql> exit;

mysql> source /home/hadoop/Downloads/cloudera/c.sql
Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 1 row affected (0.00 sec)

Query OK, 1 row affected (0.01 sec)

Query OK, 1 row affected (0.00 sec)

Query OK, 1 row affected (0.00 sec)

Query OK, 1 row affected (0.00 sec)

Query OK, 1 row affected (0.00 sec)

Query OK, 1 row affected (0.00 sec)

Query OK, 1 row affected (0.00 sec)

Query OK, 1 row affected (0.00 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

Query OK, 0 rows affected, 1 warning (0.00 sec)

+--------------------+
| Database           |
+--------------------+
| information_schema |
| amon               |
| hue                |
| metastore          |
| mysql              |
| nav                |
| navms              |
| oozie              |
| performance_schema |
| rman               |
| scm                |
| sentry             |
| sys                |
+--------------------+
13 rows in set (0.00 sec)

mysql>

准备jdbc的驱动程序

[hadoop@manager mysql-jdbc]$
sudo mkdir -p /usr/share/java/ \
&& cd /home/hadoop/Downloads/cloudera/mysql-jdbc/;tar -zxvf mysql-connector-java-5.1.47.tar.gz \
&& sudo cp /home/hadoop/Downloads/cloudera/mysql-jdbc/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47-bin.jar /usr/share/java/mysql-connector-java.jar

所有机器更换国内的阿里yum镜像(这里有坑,不是master机器做就可以了,是所有机器都要做)

[root@manager ~]# mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bk
[root@manager ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

[root@manager ~]#
(
cat <<EOF
[cloudera-manager]
name=Cloudera Manager 6.3.0
baseurl=http://192.168.88.31/cloudera-manager/
gpgcheck=0
enabled=1
EOF
) >> /etc/yum.repos.d/cloudera-manager.repo \
&& yum clean all \
&& yum makecache

[root@manager ~]# vi /etc/yum.repos.d/cloudera-manager.repo
[root@manager ~]# cat /etc/yum.repos.d/cloudera-manager.repo
安装Cloudera Manager
[hadoop@manager ~]$ sudo yum install -y cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package cloudera-manager-agent.x86_64 0:6.3.0-1281944.el7 will be installed
--> Processing Dependency: python-psycopg2 for package: cloudera-manager-agent-6.3.0-1281944.el7.x86_64
--> Processing Dependency: openssl-devel for package: cloudera-manager-agent-6.3.0-1281944.el7.x86_64
--> Processing Dependency: mod_ssl for package: cloudera-manager-agent-6.3.0-1281944.el7.x86_64
--> Processing Dependency: MySQL-python for package: cloudera-manager-agent-6.3.0-1281944.el7.x86_64
--> Processing Dependency: /lib/lsb/init-functions for package: cloudera-manager-agent-6.3.0-1281944.el7.x86_64
--> Processing Dependency: libpq.so.5()(64bit) for package: cloudera-manager-agent-6.3.0-1281944.el7.x86_64
---> Package cloudera-manager-daemons.x86_64 0:6.3.0-1281944.el7 will be installed
---> Package cloudera-manager-server.x86_64 0:6.3.0-1281944.el7 will be installed
--> Running transaction check
---> Package MySQL-python.x86_64 0:1.2.5-1.el7 will be installed
---> Package mod_ssl.x86_64 1:2.4.6-90.el7.centos will be installed
---> Package openssl-devel.x86_64 1:1.0.2k-19.el7 will be installed
--> Processing Dependency: zlib-devel(x86-64) for package: 1:openssl-devel-1.0.2k-19.el7.x86_64
--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-19.el7.x86_64
---> Package postgresql-libs.x86_64 0:9.2.24-1.el7_5 will be installed
---> Package python-psycopg2.x86_64 0:2.5.1-4.el7 will be installed
---> Package redhat-lsb-core.x86_64 0:4.1-27.el7.centos.1 will be installed
--> Processing Dependency: redhat-lsb-submod-security(x86-64) = 4.1-27.el7.centos.1 for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: spax for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Processing Dependency: /usr/sbin/sendmail for package: redhat-lsb-core-4.1-27.el7.centos.1.x86_64
--> Running transaction check
---> Package krb5-devel.x86_64 0:1.15.1-37.el7_7.2 will be installed
--> Processing Dependency: libkadm5(x86-64) = 1.15.1-37.el7_7.2 for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: krb5-libs(x86-64) = 1.15.1-37.el7_7.2 for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64
---> Package postfix.x86_64 2:2.10.1-7.el7 will be installed
---> Package redhat-lsb-submod-security.x86_64 0:4.1-27.el7.centos.1 will be installed
---> Package spax.x86_64 0:1.5.2-13.el7 will be installed
---> Package zlib-devel.x86_64 0:1.2.7-18.el7 will be installed
--> Running transaction check
---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed
---> Package krb5-libs.x86_64 0:1.15.1-37.el7_6 will be updated
---> Package krb5-libs.x86_64 0:1.15.1-37.el7_7.2 will be an update
---> Package libcom_err-devel.x86_64 0:1.42.9-16.el7 will be installed
---> Package libkadm5.x86_64 0:1.15.1-37.el7_7.2 will be installed
---> Package libselinux-devel.x86_64 0:2.5-14.1.el7 will be installed
--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-14.1.el7.x86_64
--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-14.1.el7.x86_64
--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-14.1.el7.x86_64
---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed
--> Running transaction check
---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed
---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
 Package                                                    Arch                                   Version                                                  Repository                                        Size
===================================================================================================================================================================================================================
Installing:
 cloudera-manager-agent                                     x86_64                                 6.3.0-1281944.el7                                        cloudera-manager                                  10 M
 cloudera-manager-daemons                                   x86_64                                 6.3.0-1281944.el7                                        cloudera-manager                                 1.1 G
 cloudera-manager-server                                    x86_64                                 6.3.0-1281944.el7                                        cloudera-manager                                  11 k
Installing for dependencies:
 MySQL-python                                               x86_64                                 1.2.5-1.el7                                              base                                              90 k
 keyutils-libs-devel                                        x86_64                                 1.5.8-3.el7                                              base                                              37 k
 krb5-devel                                                 x86_64                                 1.15.1-37.el7_7.2                                        updates                                          272 k
 libcom_err-devel                                           x86_64                                 1.42.9-16.el7                                            base                                              32 k
 libkadm5                                                   x86_64                                 1.15.1-37.el7_7.2                                        updates                                          178 k
 libselinux-devel                                           x86_64                                 2.5-14.1.el7                                             base                                             187 k
 libsepol-devel                                             x86_64                                 2.5-10.el7                                               base                                              77 k
 libverto-devel                                             x86_64                                 0.2.5-4.el7                                              base                                              12 k
 mod_ssl                                                    x86_64                                 1:2.4.6-90.el7.centos                                    base                                             112 k
 openssl-devel                                              x86_64                                 1:1.0.2k-19.el7                                          base                                             1.5 M
 pcre-devel                                                 x86_64                                 8.32-17.el7                                              base                                             480 k
 postfix                                                    x86_64                                 2:2.10.1-7.el7                                           base                                             2.4 M
 postgresql-libs                                            x86_64                                 9.2.24-1.el7_5                                           base                                             234 k
 python-psycopg2                                            x86_64                                 2.5.1-4.el7                                              base                                             132 k
 redhat-lsb-core                                            x86_64                                 4.1-27.el7.centos.1                                      base                                              38 k
 redhat-lsb-submod-security                                 x86_64                                 4.1-27.el7.centos.1                                      base                                              15 k
 spax                                                       x86_64                                 1.5.2-13.el7                                             base                                             260 k
 zlib-devel                                                 x86_64                                 1.2.7-18.el7                                             base                                              50 k
Updating for dependencies:
 krb5-libs                                                  x86_64                                 1.15.1-37.el7_7.2                                        updates                                          805 k

Transaction Summary
===================================================================================================================================================================================================================
Install  3 Packages (+18 Dependent packages)
Upgrade             (  1 Dependent package)

Total download size: 1.1 G
Downloading packages:
No Presto metadata available for updates
(1/22): cloudera-manager-agent-6.3.0-1281944.el7.x86_64.rpm                                                                                                                                 |  10 MB  00:00:00     
(2/22): cloudera-manager-server-6.3.0-1281944.el7.x86_64.rpm                                                                                                                                |  11 kB  00:00:00     
(3/22): MySQL-python-1.2.5-1.el7.x86_64.rpm                                                                                                                                                 |  90 kB  00:00:00     
(4/22): keyutils-libs-devel-1.5.8-3.el7.x86_64.rpm                                                                                                                                          |  37 kB  00:00:00     
(5/22): libcom_err-devel-1.42.9-16.el7.x86_64.rpm                                                                                                                                           |  32 kB  00:00:00     
(6/22): krb5-devel-1.15.1-37.el7_7.2.x86_64.rpm                                                                                                                                             | 272 kB  00:00:00     
(7/22): krb5-libs-1.15.1-37.el7_7.2.x86_64.rpm                                                                                                                                              | 805 kB  00:00:00     
(8/22): libkadm5-1.15.1-37.el7_7.2.x86_64.rpm                                                                                                                                               | 178 kB  00:00:00     
(9/22): libsepol-devel-2.5-10.el7.x86_64.rpm                                                                                                                                                |  77 kB  00:00:00     
(10/22): libselinux-devel-2.5-14.1.el7.x86_64.rpm                                                                                                                                           | 187 kB  00:00:00     
(11/22): libverto-devel-0.2.5-4.el7.x86_64.rpm                                                                                                                                              |  12 kB  00:00:00     
(12/22): mod_ssl-2.4.6-90.el7.centos.x86_64.rpm                                                                                                                                             | 112 kB  00:00:00     
(13/22): pcre-devel-8.32-17.el7.x86_64.rpm                                                                                                                                                  | 480 kB  00:00:00     
(14/22): openssl-devel-1.0.2k-19.el7.x86_64.rpm                                                                                                                                             | 1.5 MB  00:00:00     
(15/22): postgresql-libs-9.2.24-1.el7_5.x86_64.rpm                                                                                                                                          | 234 kB  00:00:00     
(16/22): python-psycopg2-2.5.1-4.el7.x86_64.rpm                                                                                                                                             | 132 kB  00:00:00     
(17/22): postfix-2.10.1-7.el7.x86_64.rpm                                                                                                                                                    | 2.4 MB  00:00:00     
(18/22): redhat-lsb-core-4.1-27.el7.centos.1.x86_64.rpm                                                                                                                                     |  38 kB  00:00:00     
(19/22): redhat-lsb-submod-security-4.1-27.el7.centos.1.x86_64.rpm                                                                                                                          |  15 kB  00:00:00     
(20/22): zlib-devel-1.2.7-18.el7.x86_64.rpm                                                                                                                                                 |  50 kB  00:00:00     
(21/22): spax-1.5.2-13.el7.x86_64.rpm                                                                                                                                                       | 260 kB  00:00:00     
(22/22): cloudera-manager-daemons-6.3.0-1281944.el7.x86_64.rpm                                                                                                                              | 1.1 GB  00:00:31     
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                               37 MB/s | 1.1 GB  00:00:31     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
icedtea-web-1.7.1-1.el7.x86_64 has missing requires of java-1.8.0-openjdk
  Updating   : krb5-libs-1.15.1-37.el7_7.2.x86_64                                                                                                                                                             1/23 
  Installing : postgresql-libs-9.2.24-1.el7_5.x86_64                                                                                                                                                          2/23 
  Installing : cloudera-manager-daemons-6.3.0-1281944.el7.x86_64                                                                                                                                              3/23 
  Installing : python-psycopg2-2.5.1-4.el7.x86_64                                                                                                                                                             4/23 
  Installing : libkadm5-1.15.1-37.el7_7.2.x86_64                                                                                                                                                              5/23 
  Installing : spax-1.5.2-13.el7.x86_64                                                                                                                                                                       6/23 
  Installing : libsepol-devel-2.5-10.el7.x86_64                                                                                                                                                               7/23 
  Installing : 1:mod_ssl-2.4.6-90.el7.centos.x86_64                                                                                                                                                           8/23 
  Installing : 2:postfix-2.10.1-7.el7.x86_64                                                                                                                                                                  9/23 
  Installing : zlib-devel-1.2.7-18.el7.x86_64                                                                                                                                                                10/23 
  Installing : libverto-devel-0.2.5-4.el7.x86_64                                                                                                                                                             11/23 
  Installing : pcre-devel-8.32-17.el7.x86_64                                                                                                                                                                 12/23 
  Installing : libselinux-devel-2.5-14.1.el7.x86_64                                                                                                                                                          13/23 
  Installing : MySQL-python-1.2.5-1.el7.x86_64                                                                                                                                                               14/23 
  Installing : libcom_err-devel-1.42.9-16.el7.x86_64                                                                                                                                                         15/23 
  Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64                                                                                                                                                        16/23 
  Installing : krb5-devel-1.15.1-37.el7_7.2.x86_64                                                                                                                                                           17/23 
  Installing : 1:openssl-devel-1.0.2k-19.el7.x86_64                                                                                                                                                          18/23 
  Installing : redhat-lsb-submod-security-4.1-27.el7.centos.1.x86_64                                                                                                                                         19/23 
  Installing : redhat-lsb-core-4.1-27.el7.centos.1.x86_64                                                                                                                                                    20/23 
  Installing : cloudera-manager-agent-6.3.0-1281944.el7.x86_64                                                                                                                                               21/23 
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
  Installing : cloudera-manager-server-6.3.0-1281944.el7.x86_64                                                                                                                                              22/23 
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-server.service to /usr/lib/systemd/system/cloudera-scm-server.service.
  Cleanup    : krb5-libs-1.15.1-37.el7_6.x86_64                                                                                                                                                              23/23 
  Verifying  : redhat-lsb-submod-security-4.1-27.el7.centos.1.x86_64                                                                                                                                          1/23 
  Verifying  : keyutils-libs-devel-1.5.8-3.el7.x86_64                                                                                                                                                         2/23 
  Verifying  : krb5-devel-1.15.1-37.el7_7.2.x86_64                                                                                                                                                            3/23 
  Verifying  : libcom_err-devel-1.42.9-16.el7.x86_64                                                                                                                                                          4/23 
  Verifying  : MySQL-python-1.2.5-1.el7.x86_64                                                                                                                                                                5/23 
  Verifying  : pcre-devel-8.32-17.el7.x86_64                                                                                                                                                                  6/23 
  Verifying  : libverto-devel-0.2.5-4.el7.x86_64                                                                                                                                                              7/23 
  Verifying  : zlib-devel-1.2.7-18.el7.x86_64                                                                                                                                                                 8/23 
  Verifying  : 2:postfix-2.10.1-7.el7.x86_64                                                                                                                                                                  9/23 
  Verifying  : cloudera-manager-daemons-6.3.0-1281944.el7.x86_64                                                                                                                                             10/23 
  Verifying  : cloudera-manager-server-6.3.0-1281944.el7.x86_64                                                                                                                                              11/23 
  Verifying  : cloudera-manager-agent-6.3.0-1281944.el7.x86_64                                                                                                                                               12/23 
  Verifying  : krb5-libs-1.15.1-37.el7_7.2.x86_64                                                                                                                                                            13/23 
  Verifying  : python-psycopg2-2.5.1-4.el7.x86_64                                                                                                                                                            14/23 
  Verifying  : 1:mod_ssl-2.4.6-90.el7.centos.x86_64                                                                                                                                                          15/23 
  Verifying  : 1:openssl-devel-1.0.2k-19.el7.x86_64                                                                                                                                                          16/23 
  Verifying  : libsepol-devel-2.5-10.el7.x86_64                                                                                                                                                              17/23 
  Verifying  : spax-1.5.2-13.el7.x86_64                                                                                                                                                                      18/23 
  Verifying  : libkadm5-1.15.1-37.el7_7.2.x86_64                                                                                                                                                             19/23 
  Verifying  : libselinux-devel-2.5-14.1.el7.x86_64                                                                                                                                                          20/23 
  Verifying  : redhat-lsb-core-4.1-27.el7.centos.1.x86_64                                                                                                                                                    21/23 
  Verifying  : postgresql-libs-9.2.24-1.el7_5.x86_64                                                                                                                                                         22/23 
  Verifying  : krb5-libs-1.15.1-37.el7_6.x86_64                                                                                                                                                              23/23 

Installed:
  cloudera-manager-agent.x86_64 0:6.3.0-1281944.el7                    cloudera-manager-daemons.x86_64 0:6.3.0-1281944.el7                    cloudera-manager-server.x86_64 0:6.3.0-1281944.el7                   

Dependency Installed:
  MySQL-python.x86_64 0:1.2.5-1.el7              keyutils-libs-devel.x86_64 0:1.5.8-3.el7        krb5-devel.x86_64 0:1.15.1-37.el7_7.2               libcom_err-devel.x86_64 0:1.42.9-16.el7                       
  libkadm5.x86_64 0:1.15.1-37.el7_7.2            libselinux-devel.x86_64 0:2.5-14.1.el7          libsepol-devel.x86_64 0:2.5-10.el7                  libverto-devel.x86_64 0:0.2.5-4.el7                           
  mod_ssl.x86_64 1:2.4.6-90.el7.centos           openssl-devel.x86_64 1:1.0.2k-19.el7            pcre-devel.x86_64 0:8.32-17.el7                     postfix.x86_64 2:2.10.1-7.el7                                 
  postgresql-libs.x86_64 0:9.2.24-1.el7_5        python-psycopg2.x86_64 0:2.5.1-4.el7            redhat-lsb-core.x86_64 0:4.1-27.el7.centos.1        redhat-lsb-submod-security.x86_64 0:4.1-27.el7.centos.1       
  spax.x86_64 0:1.5.2-13.el7                     zlib-devel.x86_64 0:1.2.7-18.el7               

Dependency Updated:
  krb5-libs.x86_64 0:1.15.1-37.el7_7.2                                                                                                                                                                             

Complete!
[hadoop@manager ~]$

配置本地Parcel存储库

[root@manager ~]# mv /home/hadoop/Downloads/cloudera/parcel/* /opt/cloudera/parcel-repo
[root@manager ~]# cd /opt/cloudera/parcel-repo
[root@manager parcel-repo]# pwd
/opt/cloudera/parcel-repo
[root@manager ~]# sha1sum CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel | awk '{ print $1 }' > CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.sha
[root@manager ~]# chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*

初始化scm库

[root@manager parcel-repo]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm 123456Aa.
JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/java/jdk1.8.0_181-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Thu Oct 31 00:33:50 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

启动CM

[root@manager ~]#
systemctl stop cloudera-scm-server \
&& systemctl start cloudera-scm-server \
&& sleep 2 \
&& tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log | grep "INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server"
1分钟等待后,如果出现:
2019-10-31 00:39:00,113 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
则代表CM启动成功了
访问---》【http://192.168.88.31:7180】
账号密码为admin/admin
在整个启动过程中,可以新开一个窗口,然后tail -f看看日志有啥错误
[root@manager ~]# tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

如果在中间出现问题,要先
[root@manager ~]# systemctl stop cloudera-scm-server
再运行上面的启动命令。数据库密码【123456Aa.】

初始化CM

如果在中间报异常了,那么基本上就需要做回滚和清理动作了(还不如重新来个新的操作系统来得简单)

【一】、Cluster Basics

给个人看得懂的集群名字即可

【二】、Accept JDK License

接受,不接受走不下去

【三】、Enter Login Credentials

root用户可以安装(sudo用户????)安装 Cloudera 包需要有主机的 root 访问权限。此安装程序将通过 SSH 连接到您的主机,然后直接以 root 用户身份登录,或者以另一个具有变为 root 用户的无密码 sudo/pbrun 权限的用户身份登录。

【四】、Specify Hosts

192.168.88.[31-34],可以通过这样的表达式来搜索多台机器

【五】、Select Repository

填入本地的yum库---》http://192.168.88.31/cloudera-manager/
CDH-6.3.0-1.cdh6.3.0.p0.1279813,这里我们需要明确我们自己安装的版本,因为系统会把比较新的版本都显示出来,如果选择错了,就是从网络下载了,那个时间就超长了

【六】、Install Agents

耗时10、20分钟的样子,由于页面是js写的,因此不要刷新浏览器。如果出错了,根据错误修正接节点的问题,然后retry。如果我们在这里放弃了,基本上这个操作系统也就废掉了(删除清理,估计也是非常麻烦的)

【七】、Install Parcels

耗时几个小时,distubuted哪里会刷新传递分发的速度 xxM/s。这个阶段磁盘占用率非常高。页面上是有简单的进度条显示整体进度的。没有到100%,continue的按钮是点击不了的。

【八】、Inspect Cluster

点击Inspect Network Performance按钮
点击Inspect Hosts按钮,可以参考这里:https://blog.****.net/qq_39680564/article/details/100013275,相关的报错汇总
主机健康检查、I understand the risks, let me continue with cluster setup.,点选这个走下去。
各个组件的验证

【选择安装的服务】

impala、hdfs、hbase等组件

【自行分配角色至主机】

这里官方文档中,有按照规模建议指定机器的分配方式

【测试数据库链接】

这里的数据库实例名,用户名、密码都是上面的脚本创建的,这里是直接使用已经创建好的库,不能乱填的,要按照上面的创建mysql库的脚本来进行填写。

常见报错1:JDBC driver cannot be found. Unable to find the JDBC database jar on host
[hadoop@manager mysql-jdbc]$
sudo mkdir -p /usr/share/java/ \
&& cd /home/hadoop/Downloads/cloudera/mysql-jdbc/;tar -zxvf mysql-connector-java-5.1.47.tar.gz \
&& sudo cp /home/hadoop/Downloads/cloudera/mysql-jdbc/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47-bin.jar /usr/share/java/mysql-connector-java.jar

常见报错2:Cloudera 建议将 /proc/sys/vm/swappiness 设置为最大值 10
临时生效:
sysctl vm.swappiness=10 && cat /proc/sys/vm/swappiness
echo 'vm.swappiness=10'>> /etc/sysctl.conf

永久生效:
echo 'vm.swappiness=10'>> /etc/sysctl.conf

常见报错3:已启用透明大页面压缩,可能会导致重大性能问题。
临时生效:
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local
echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local

永久生效:
echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local
echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local

【配置实例的安装位置】

kudu的配置参数
Kudu Master WAL Directory:/kudumaster/wal
Kudu Master Data Directories:/kudumaster/data
Kudu Tablet Server WAL Directory:/kudutablet/wal
Kudu Tablet Server Data Directories:/kudutablet/data
HDFS 块大小 128
接受的 DataNode 失败的卷:0
DataNode 数据目录 /dfs/dn
NameNode 数据目录 /dfs/nn
HDFS 检查点目录 /dfs/snn
Host Monitor 存储目录 /var/lib/cloudera-host-monitor
Service Monitor 存储目录 /var/lib/cloudera-service-monitor
NodeManager 本地目录 /yarn/nm
数据目录 /var/lib/zookeeper
事务日志目录 /var/lib/zookeeper

【安装服务】

各种错误,可以参照:
https://www.cnblogs.com/felixzh/p/9082344.html
https://blog.****.net/sardtass/article/details/93723610
来进行问题解决。解决后,不停重试安装即可。

【安装完成成功提示】

关闭所有服务:
在页面Cluster集群的cluster1后面有个下拉三角,点击出下拉列表后,点击stop,这个stop也就是把集群中的所有服务,全部定制
在Cloudera Management Service后面有个下拉三角,点击出下拉列表后,点击stop,这个stop也就是把集群中的所有服务,全部定制

仅仅在主机上运行
[root@master cloudera]# service cloudera-scm-server stop
Redirecting to /bin/systemctl stop cloudera-scm-server.service
每台机器都要运行这个指令
[root@master cloudera]# service cloudera-scm-agent stop
Redirecting to /bin/systemctl stop cloudera-scm-agent.service

[root@manager ~]# 
service cloudera-scm-server stop \
&& service cloudera-scm-agent stop

重新启动整个集群:
service cloudera-scm-server start
所有都需要运行service cloudera-scm-agent start

先需要启动CM
[root@manager ~]#
systemctl stop cloudera-scm-server \
&& systemctl start cloudera-scm-server \
&& sleep 2 \
&& tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log | grep "INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server"

[root@manager ~]# 
service cloudera-scm-server restart \
&& service cloudera-scm-agent restart

在Cloudera Management Service后面有个下拉三角,点击出下拉列表后,点击stop,这个start也就是把集群中的所有服务,全部定制
在页面Cluster集群的cluster1后面有个下拉三角,点击出下拉列表后,点击stop,这个start也就是把集群中的所有服务,全部定制

service cloudera-scm-server status
service cloudera-scm-agent status

[root@manager ~]# 
service cloudera-scm-server status \
&& service cloudera-scm-agent status
上一篇:CentOS7中搭建nodejs10(VUE)开发环境踩坑指南


下一篇:深入浅出:Hadoop的start-balancer.sh与hdfs balancer分布式数据均衡