基于yum的方式安装Cloudera Manager Server（使用Mysql 8.0版本）
作者：尹正杰
版权声明：原创作品，谢绝转载！否则将追究法律责任。
一.安装和配置元数据库
1>.部署MySQL数据库
我们应启动的5台服务器中选出来2台服务做MySQL主从同步。我们把mysql服务安装在node101.yinzhengjie.org.cn上。至于MySQL的部署方式并不是本片博客的重点，不过可参考我之前的笔记：使用二进制安装包的方式单机部署MySQL8.0.13。
由于这个元数据库还是比较重要的，在生产环境使用中，建议使用MySQL的复制机制部署一个主从复制。或者使用Keepalived+MySQL部署一个主主双活的数据库，以提升其可用性。
2>.配置元数据库
创建所需的表和账号。首先我们修改root的密码。接着创建scm,rman,hive,oozie,hue,sentry等6套库和用户，分别用户Cloudera Manager，Reports Manager，hive，oozie，hue，sentry存储数据。
[root@node101 ~]# mysql -uroot -pyinzhengjie
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 13
Server version: 8.0.15 MySQL Community Server - GPL Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql>
mysql> CREATE DATABASE scm DEFAULT CHARACTER SET UTF8;
Query OK, 1 row affected, 1 warning (0.01 sec) mysql>
mysql> CREATE USER scm@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie';
Query OK, 0 rows affected (0.01 sec) mysql>
mysql> GRANT ALL PRIVILEGES ON scm.* TO scm@'%';
Query OK, 0 rows affected (0.01 sec) mysql>
mysql> CREATE DATABASE rman DEFAULT CHARACTER SET UTF8;
Query OK, 1 row affected, 1 warning (0.01 sec) mysql> CREATE USER rman@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie';
Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL PRIVILEGES ON rman.* TO rman@'%';
Query OK, 0 rows affected (0.00 sec) mysql>
创建Reports Manager所需要的库和用户
mysql> CREATE DATABASE hive DEFAULT CHARACTER SET UTF8;
Query OK, 1 row affected, 1 warning (0.01 sec) mysql> CREATE USER hive@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie';
Query OK, 0 rows affected (0.00 sec) mysql> GRANT ALL PRIVILEGES ON hive.* TO hive@'%';
Query OK, 0 rows affected (0.00 sec) mysql>
创建hive所需要的库和用户
mysql> CREATE DATABASE oozie DEFAULT CHARACTER SET UTF8;
Query OK, 1 row affected, 1 warning (0.00 sec) mysql> CREATE USER oozie@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie';
Query OK, 0 rows affected (0.01 sec) mysql> GRANT ALL PRIVILEGES ON oozie.* TO oozie@'%';
Query OK, 0 rows affected (0.00 sec) mysql>
创建oozie所需要的库和用户
mysql> CREATE DATABASE hue DEFAULT CHARACTER SET UTF8;
Query OK, 1 row affected, 1 warning (0.00 sec) mysql> CREATE USER hue@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie';
Query OK, 0 rows affected (0.01 sec) mysql> GRANT ALL PRIVILEGES ON hue.* TO hue@'%';
Query OK, 0 rows affected (0.01 sec) mysql>
创建hue所需要的库和用户
mysql> CREATE DATABASE sentry DEFAULT CHARACTER SET UTF8;
Query OK, 1 row affected, 1 warning (0.00 sec) mysql> CREATE USER sentry@'%' IDENTIFIED WITH mysql_native_password BY 'yinzhengjie';
Query OK, 0 rows affected (0.01 sec) mysql> GRANT ALL PRIVILEGES ON sentry.* TO sentry@'%';
Query OK, 0 rows affected (0.00 sec) mysql>
创建sentry所需要的库和用户
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec) mysql>
mysql> SELECT user,host,plugin FROM mysql.user;
+------------------+-----------+-----------------------+
| user | host | plugin |
+------------------+-----------+-----------------------+
| hive | % | mysql_native_password |
| hue | % | mysql_native_password |
| oozie | % | mysql_native_password |
| rman | % | mysql_native_password |
| scm | % | mysql_native_password |
| sentry | % | mysql_native_password |
| mysql.infoschema | localhost | caching_sha2_password |
| mysql.session | localhost | caching_sha2_password |
| mysql.sys | localhost | caching_sha2_password |
| root | localhost | caching_sha2_password |
+------------------+-----------+-----------------------+
10 rows in set (0.00 sec) mysql>
[root@node101 ~]# mysql -uscm -pyinzhengjie #测试是否可以正常登陆
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 24
Server version: 8.0.15 MySQL Community Server - GPL Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> quit
Bye
[root@node101 ~]#
[root@node101 ~]# mysql -h node101.yinzhengjie.org.cn -uscm -pyinzhengjie
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 25
Server version: 8.0.15 MySQL Community Server - GPL Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> quit
Bye
[root@node101 ~]#
测试创建的用户是否可以正常登陆
二.Cloudera Manager 部署方式
1>.yum方式安装方式
yum安装方式一：（下载Cloudera Manager的源，修改配置文件并执行安装，这种方式是通过网络的方法安装，效率较低！）
下载地址：https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/。
[root@node101 ~]# ll /etc/yum.repos.d/ | grep cloudera-manager.repo
[root@node101 ~]#
[root@node101 ~]# wget -P /etc/yum.repos.d/ https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo
--2019-02-21 11:47:07-- https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo
Resolving archive.cloudera.com (archive.cloudera.com)... 151.101.108.167
Connecting to archive.cloudera.com (archive.cloudera.com)|151.101.108.167|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 290 [binary/octet-stream]
Saving to: ‘/etc/yum.repos.d/cloudera-manager.repo’ 100%[========================================================================================================================================================================>] 290 --.-K/s in 0s 2019-02-21 11:47:07 (48.4 MB/s) - ‘/etc/yum.repos.d/cloudera-manager.repo’ saved [290/290] [root@node101 ~]#
[root@node101 ~]# ll /etc/yum.repos.d/ | grep cloudera-manager.repo
-rw-r--r-- 1 root root 290 Dec 14 00:32 cloudera-manager.repo
[root@node101 ~]#
[root@node101 ~]#
[root@node101 ~]# wget -P /etc/yum.repos.d/ https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo #下载CM的源
[root@node101 ~]# grep baseurl /etc/yum.repos.d/cloudera-manager.repo
baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5/
[root@node101 ~]#
[root@node101 ~]# sed -i s'#baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5/#baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/#' /etc/yum.repos.d/cloudera-manager.repo #修改为你想要安装的CM版本
[root@node101 ~]#
[root@node101 ~]# grep baseurl /etc/yum.repos.d/cloudera-manager.repo #我这里改为5.16.1版本
baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/
[root@node101 ~]#
[root@node101 ~]# yum -y install cloudera-manager-daemons cloudera-manager-server #
按照上面的方式按照是可以按照成功，但是网速想我这样慢的小伙伴就不推荐使用这种方式，我们可以提前把rpm包下载下来，然后手动安装即可。
yum安装方式二：（如果你发现下载速度很慢，可以在自己电脑上下载安装包在上传到虚拟机内，本片博客采用的就是这种方式！大家也看到了，上面的下载速度太慢了！）
下载地址：https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.16.1/RPMS/x86_64/。
[root@node101 ~]# ll -lht
total 754M
-rwxr-xr-x 1 root root 11K Feb 21 12:21 cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
-rwxr-xr-x 1 root root 8.5K Feb 21 12:21 cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
-rwxr-xr-x 1 root root 754M Feb 21 12:21 cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
[root@node101 ~]#
[root@node101 ~]# ll -lht
total 754M
-rwxr-xr-x 1 root root 11K Feb 21 12:21 cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
-rwxr-xr-x 1 root root 8.5K Feb 21 12:21 cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
-rwxr-xr-x 1 root root 754M Feb 21 12:21 cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
[root@node101 ~]#
[root@node101 ~]# yum -y install cloudera-manager-*.rpm
Loaded plugins: fastestmirror
Examining cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm: cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64
Marking cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm to be installed
Examining cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm: cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64
Marking cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm to be installed
Examining cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm: cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64
Marking cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package cloudera-manager-daemons.x86_64 0:5.16.1-1.cm5161.p0.1.el7 will be installed
---> Package cloudera-manager-server.x86_64 0:5.16.1-1.cm5161.p0.1.el7 will be installed
---> Package cloudera-manager-server-db-2.x86_64 0:5.16.1-1.cm5161.p0.1.el7 will be installed
--> Processing Dependency: postgresql-server >= 8.4 for package: cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64
Loading mirror speeds from cached hostfile
* base: mirrors.shu.edu.cn
* extras: mirror.jdcloud.com
* updates: centos.cs.nctu.edu.tw
--> Running transaction check
---> Package postgresql-server.x86_64 0:9.2.24-1.el7_5 will be installed
--> Processing Dependency: postgresql-libs(x86-64) = 9.2.24-1.el7_5 for package: postgresql-server-9.2.24-1.el7_5.x86_64
--> Processing Dependency: postgresql(x86-64) = 9.2.24-1.el7_5 for package: postgresql-server-9.2.24-1.el7_5.x86_64
--> Processing Dependency: libpq.so.5()(64bit) for package: postgresql-server-9.2.24-1.el7_5.x86_64
--> Running transaction check
---> Package postgresql.x86_64 0:9.2.24-1.el7_5 will be installed
---> Package postgresql-libs.x86_64 0:9.2.24-1.el7_5 will be installed
--> Finished Dependency Resolution Dependencies Resolved ======================================================================================================================================================================================================================================================
Package Arch Version Repository Size
======================================================================================================================================================================================================================================================
Installing:
cloudera-manager-daemons x86_64 5.16.1-1.cm5161.p0.1.el7 /cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64 935 M
cloudera-manager-server x86_64 5.16.1-1.cm5161.p0.1.el7 /cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64 13 k
cloudera-manager-server-db-2 x86_64 5.16.1-1.cm5161.p0.1.el7 /cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64 20 k
Installing for dependencies:
postgresql x86_64 9.2.24-1.el7_5 base 3.0 M
postgresql-libs x86_64 9.2.24-1.el7_5 base 234 k
postgresql-server x86_64 9.2.24-1.el7_5 base 3.8 M Transaction Summary
======================================================================================================================================================================================================================================================
Install 3 Packages (+3 Dependent packages) Total size: 942 M
Total download size: 7.1 M
Installed size: 968 M
Downloading packages:
postgresql-server-9.2.24-1.el7 FAILED
http://mirror.lzu.edu.cn/centos/7.6.1810/os/x86_64/Packages/postgresql-server-9.2.24-1.el7_5.x86_64.rpm: [Errno 14] curl#56 - "Recv failure: Connection reset by peer" ] 0.0 B/s | 0 B --:--:-- ETA
Trying other mirror.
(1/3): postgresql-9.2.24-1.el7_5.x86_64.rpm | 3.0 MB 00:00:03
(2/3): postgresql-server-9.2.24-1.el7_5.x86_64.rpm | 3.8 MB 00:00:05
(3/3): postgresql-libs-9.2.24-1.el7_5.x86_64.rpm | 234 kB 00:00:07
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 962 kB/s | 7.1 MB 00:00:07
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : postgresql-libs-9.2.24-1.el7_5.x86_64 1/6
Installing : postgresql-9.2.24-1.el7_5.x86_64 2/6
Installing : postgresql-server-9.2.24-1.el7_5.x86_64 3/6
Installing : cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64 4/6
Installing : cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64 5/6
Installing : cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64 6/6
Verifying : cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64 1/6
Verifying : cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64 2/6
Verifying : cloudera-manager-server-db-2-5.16.1-1.cm5161.p0.1.el7.x86_64 3/6
Verifying : postgresql-server-9.2.24-1.el7_5.x86_64 4/6
Verifying : postgresql-libs-9.2.24-1.el7_5.x86_64 5/6
Verifying : postgresql-9.2.24-1.el7_5.x86_64 6/6 Installed:
cloudera-manager-daemons.x86_64 0:5.16.1-1.cm5161.p0.1.el7 cloudera-manager-server.x86_64 0:5.16.1-1.cm5161.p0.1.el7 cloudera-manager-server-db-2.x86_64 0:5.16.1-1.cm5161.p0.1.el7 Dependency Installed:
postgresql.x86_64 0:9.2.24-1.el7_5 postgresql-libs.x86_64 0:9.2.24-1.el7_5 postgresql-server.x86_64 0:9.2.24-1.el7_5 Complete!
[root@node101 ~]#
[root@node101 ~]# yum -y install cloudera-manager-*.rpm #下载后安装
2>..离线方式安装
详情请参考：离线方式部署Cloudera Manager5.15.1。
三.配置并启动Cloudera Manager
1>.配置Cloudera Manager
[root@node101 ~]# grep CMF_JAVA_OPTS /etc/default/cloudera-scm-server #Cloudera Manager默认分配的JVM堆内存是2G，某些情况下会出现OOM的情况，我们可以适当将其调大。
export CMF_JAVA_OPTS="-Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"
[root@node101 ~]#
[root@node101 ~]# sed -i s'#Xmx2G#Xmx4G#' /etc/default/cloudera-scm-server #我们将Cloudera Manager的堆内存调大到4G。
[root@node101 ~]#
[root@node101 ~]# grep CMF_JAVA_OPTS /etc/default/cloudera-scm-server
export CMF_JAVA_OPTS="-Xmx4G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"
[root@node101 ~]#
[root@node101 ~]#
2>.初始化数据库配置
Cloudera已经提供了一个sh脚本用户配置Cloudera Manager连接元数据库，官方文档参考：https://www.cloudera.com/documentation/enterprise/5-14-x/topics/cm_ig_installing_configuring_dbs.html#concept_i2r_m3m_hn。
我们的安装规划下，元数据库和Cloudera Manager运行于同一个节点，且已经预先创建了scm库和用户，因此，命令很简单：
[root@node101 ~]# /usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm yinzhengjie
JAVA_HOME=/usr/java/jdk1.
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1./bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly! #注意，如果你看到了这一行提示，说明配置完成，否则需要根据提示解决相应的错误！
[root@node101 ~]#
3>.启动Cloudera Manager
[root@node101 ~]# systemctl status cloudera-scm-server
● cloudera-scm-server.service - LSB: Cloudera SCM Server
Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-server; bad; vendor preset: disabled)
Active: inactive (dead)
Docs: man:systemd-sysv-generator() Feb :: node101.yinzhengjie.org.cn systemd[]: [/run/systemd/generator.late/cloudera-scm-server.service:] Failed to add dependency on +cloudera-scm-server-db.service, ignoring: Invalid argument
Feb :: node101.yinzhengjie.org.cn systemd[]: [/run/systemd/generator.late/cloudera-scm-server.service:] Failed to add dependency on +cloudera-scm-server-db.service, ignoring: Invalid argument
[root@node101 ~]#
[root@node101 ~]# systemctl start cloudera-scm-server #启动Cloudera Manager
[root@node101 ~]#
[root@node101 ~]# systemctl status cloudera-scm-server
● cloudera-scm-server.service - LSB: Cloudera SCM Server
Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-server; bad; vendor preset: disabled)
Active: active (exited) since Thu -- :: CST; 2min 38s ago
Docs: man:systemd-sysv-generator()
Process: ExecStart=/etc/rc.d/init.d/cloudera-scm-server start (code=exited, status=/SUCCESS) Feb :: node101.yinzhengjie.org.cn systemd[]: Starting LSB: Cloudera SCM Server...
Feb :: node101.yinzhengjie.org.cn su[]: PAM unable to dlopen(/usr/lib64/security/pam_fprintd.so): /usr/lib64/security/pam_fprintd.so: cannot open shared object file: No such file or directory
Feb :: node101.yinzhengjie.org.cn su[]: PAM adding faulty module: /usr/lib64/security/pam_fprintd.so
Feb :: node101.yinzhengjie.org.cn su[]: (to cloudera-scm) root on none
Feb :: node101.yinzhengjie.org.cn cloudera-scm-server[]: Starting cloudera-scm-server: [ OK ]
Feb :: node101.yinzhengjie.org.cn systemd[]: Started LSB: Cloudera SCM Server.
[root@node101 ~]#
4>.查看Cloudera Manager的日志
[root@node101 ~]# tail -10f /var/log/cloudera-scm-server/cloudera-scm-server.log #查看Cloudera Manager的服务的日志
-- ::, INFO SearchRepositoryManager-:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Initializing SearchTemplateManager:--21T04::.540Z
-- ::, INFO SearchRepositoryManager-:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating entities:--21T04::.673Z
-- ::, INFO SearchRepositoryManager-:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num entities:
-- ::, INFO SearchRepositoryManager-:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating documents:--21T04::.721Z
-- ::, INFO SearchRepositoryManager-:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num docs:
-- ::, INFO SearchRepositoryManager-:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Constructing repo:--21T04::.823Z
-- ::, INFO WebServerImpl:org.mortbay.log: jetty-6.1..cloudera.
-- ::, INFO WebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7180 #注意，如果出现这一行提示，说明你的服务启动成功啦！默认端口为7180，下面有WebServerImpl：Started Jetty server提示！
-- ::, INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
-- ::, INFO SearchRepositoryManager-:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Finished constructing repo:--21T04::.827Z
5>.访问Cloudera Manager 的WebUI界面
打开浏览器，推荐使用Chrome并将英语设置为首选语言，访问：http://node101.yinzhengjie.org.cn:7180/cmf/login