《OD大数据实战》Hive环境搭建

2023-01-19 19:33:39

一、搭建hadoop环境

《OD大数据实战》hadoop伪分布式环境搭建

二、Hive环境搭建

1. 准备安装文件

下载地址：

http://archive.cloudera.com/cdh5/cdh/5/

hive-0.13.1-cdh5.3.6.tar.gz

2. 解压

tar -zxvf hive-0.13.-cdh5.3.6.tar.gz -C /opt/modules/cdh/

3. 修改配置

cd /opt/modules/cdh/hive-0.13.-cdh5.3.6/conf

mv hive-env.sh.template hive-env.sh

mv hive-default.xml.template hive-site.xml

mv hive-exec-log4j.properties.template hive-exec-log4j.properties

mv hive-log4j.properties.template hive-log4j.properties

1）修改hive-env.sh

#增加一行

export JAVA_HOME=/opt/modules/jdk1..0_67

HADOOP_HOME=/opt/modules/cdh/hadoop-2.5.-cdh5.3.6

export HIVE_CONF_DIR=/opt/modules/cdh/hive-0.13.-cdh5.3.6/conf

2）修改hive-log4j.properties

hive.log.dir=/opt/modules/cdh/hive-0.13.-cdh5.3.6/logs

3）修改hive-exec-log4j.properties

hive.log.dir=/opt/modules/cdh/hive-0.13.-cdh5.3.6/logs

4）修改hive-site.xml

<property>

  <name>hive.lazysimple.extended_boolean_literal</name>

  <value>false</value>

  <description>

    LazySiimpleSerde uses this properties to determine if it treats 'T', 't', 'F', 'f',

    '1', and '0' as extened, legal boolean literal, in addition to 'TRUE' and 'FALSE'.

    The default is false, which means only 'TRUE' and 'FALSE' are treated as legal

    boolean literal.

  </description>

</property>

<property>

  <name>hive.mapjoin.optimized.hashtable</name>

  <value>true</value>

  <description>Whether Hive should use memory-optimized hash table for MapJoin. Only works on Tez, because memory-optimized hashtable cannot be serialized.</description>

</property>

4. 验证hive环境结果

bin/hive

dfs -ls /;

三、mysql环境搭建

1. 目标是安装mysql 5.1.17

2. 在官网下载yum源

http://dev.mysql.com/downloads/repo/yum/

http://repo.mysql.com//mysql57-community-release-el6-8.noarch.rpm

3. 安装yum源到/etc/yum.repos.d/目录

sudo rpm -Uvh mysql57-community-release-el6-8.noarch.rpm

cd /etc/yum.repos.d/

4. 修改yum源配置

修改文件：mysql-community.repo 和mysql-community-resource.repo

5.6 enable = 1

5.7 enable = 0

5. 安装mysql

sudo yum -y install mysql-community-server

6. mysql安全性设置

sudo mysql_secure_installation

grant all privileges on *.* to 'root'@'%' identified by 'beifeng' with grant option

7. 验证mysql安装结果

进入命令行： mysql -uroot -p

四、本地mysql作为metastore模式

1. copy mysql驱动到${HIVE_HOME}/lib中

cp mysql-connector-java-5.1.-bin.jar /opt/modules/cdh/hive-0.13.-cdh5.3.6/lib/

2. 修改hive-site.xml

<property>

  <name>javax.jdo.option.ConnectionURL</name>

  <value>jdbc:mysql://localhost:3306/cdh_hive_local_hive?createDatabaseIfNotExist=true</value>

  <description>JDBC connect string for a JDBC metastore</description>

</property>

<property>

  <name>javax.jdo.option.ConnectionDriverName</name>

  <value>com.mysql.jdbc.Driver</value>

  <description>Driver class name for a JDBC metastore</description>

</property>

<property>

  <name>javax.jdo.option.ConnectionUserName</name>

  <value>root</value>

  <description>username to use against metastore database</description>

</property>

<property>

  <name>javax.jdo.option.ConnectionPassword</name>

  <value>beifeng</value>

  <description>password to use against metastore database</description>

</property>

3. 运行bin/hive命令

4. 查看mysql数据库，发现多了一个cdh_hive_local_hive数据库

五、远程mysql作为metastore模式

1. copy mysql驱动到${HIVE_HOME}/lib中

cp mysql-connector-java-5.1.27-bin.jar /opt/modules/cdh/hive-0.13.1-cdh5.3.6/lib/

2. 启动metastore服务器

nohup hive --service metastore > /home/beifeng/hive_metastore.run.log 2>&1 &

系统日志输出级别： 2 错误，1正常

查看进程信息： ps -ef | grep HiveMetaStore

关闭Hive

kill -9 processId

kill -9 `ps -ef | grep HiveMetaStore | awk '{print $2'} | head -n 1`

3. 修改hive-site.xml

<property>

  <name>hive.metastore.uris</name>

  <value>thrift://beifeng-hadoop-02:9083</value>

  <description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>

</property>

<property>

  <name>javax.jdo.option.ConnectionURL</name>

  <value>jdbc:mysql://localhost:3306/cdh_hive_remote_hive?createDatabaseIfNotExist=true</value>

  <description>JDBC connect string for a JDBC metastore</description>

</property>

<property>

  <name>javax.jdo.option.ConnectionDriverName</name>

  <value>com.mysql.jdbc.Driver</value>

  <description>Driver class name for a JDBC metastore</description>

</property>

<property>

  <name>javax.jdo.option.ConnectionUserName</name>

  <value>root</value>

  <description>username to use against metastore database</description>

</property>

<property>

  <name>javax.jdo.option.ConnectionPassword</name>

  <value>beifeng</value>

  <description>password to use against metastore database</description>

</property>

4. 运行bin/hive命令

5. 查看mysql数据库，发现多了一个cdh_hive_local_hive数据库

六、JDBC连接hive

1. 修改hive-site.xml

<property>

  <name>hive.server2.thrift.port</name>

  <value>10000</value>

  <description>Port number of HiveServer2 Thrift interface.

  Can be overridden by setting $HIVE_SERVER2_THRIFT_PORT</description>

</property>

<property>

  <name>hive.server2.thrift.bind.host</name>

  <value>0.0.0.0</value>

  <description>Bind host on which to run the HiveServer2 Thrift interface.

  Can be overridden by setting $HIVE_SERVER2_THRIFT_BIND_HOST</description>

</property>

2. 启动hiveserver2服务器

nohup hive --service hiveserver2 > /home/beifeng/hiveserver2.run.log >& &

ps -ef | grep HiveServer2

netstat -tlnup | grep

3. 进入beeline客户端

beeline

4. 连接hive

beeline> !connect jdbc:hive2://beifeng-hadoop-02:10000

scan complete in 5ms

Connecting to jdbc:hive2://beifeng-hadoop-02:10000

Enter username for jdbc:hive2://beifeng-hadoop-02:10000: beifeng

Enter password for jdbc:hive2://beifeng-hadoop-02:10000: *******

5. 修改配置

<property>

  <name>hive.server2.long.polling.timeout</name>

  <value></value>

  <description>Time in milliseconds that HiveServer2 will wait, before responding to asynchronous calls that use long polling</description>

</property>

码农公寓