Hadoop手把手逐级搭建(4) Hadoop高可用+联邦+视图文件系统(HA+Federation+ViewFs)

2021-10-03 02:02:56

前置步骤:

第四阶段: Hadoop高可用+联邦(HA+Federation)

0. 步骤概述

1). 为高可用保存hadoop配置

2). 增加federation配置

3). 首次启动HA+Federation集群part1：启动journalnode和zookeeper，格式化zookeeper集群

4). 首次启动HA+Federation集群part2：格式化第一组的namenode，即hadoop1

5). 首次启动HA+Federation集群part3：格式化第一组的namenode，即hadoop3

6). 首次启动HA+Federation集群part4：启动ZKFC, datanode和yarn

7). 常规启动HA+Federation集群

8). 在HA+Federation集群上测试wordcount程序

9). 为HA+Federation(高可用+联邦)配置viewfs

10). 在HA+Federation+viewFs集群上测试wordcount程序

1. 为高可用保存hadoop配置

1.1 进入$HADOOP_HOME/etc/目录

[root@hadoop1 ~]# cd /opt/test/hadoop-2.6.5/etc

1.2 备份hadoop高可用配置，供以后使用

[root@hadoop1 etc]# cp -r hadoop/ hadoop-ha

1.3 查看$HADOOP_HOME/etc/目录，备份成功

[root@hadoop1 etc]# ls
hadoop hadoop-full hadoop-ha

# hadoop-full保留了已有配置，接下来高可用的配置继续在hadoop文件夹内修改

2. 增加federation配置

2.0 在hadoop1上进入$HADOOP_HOME/etc/hadoop目录

[root@hadoop1 ~]# cd /opt/test/hadoop-2.6.5/etc/hadoop

2.1 在hadoop1上修改hdfs-site.xml文件，将原有配置替换如下

[root@hadoop1 hadoop]# vim hdfs-site.xml
<configuration>
<property>
   <name>dfs.replication</name>
   <value>3</value>
</property>
<!--定义nameservices逻辑名称-->
<property>
  <name>dfs.nameservices</name>
  <value>mycluster,mycluster2</value>
</property>
<!--映射nameservices逻辑名称到namenode逻辑名称-->
<property>
  <name>dfs.ha.namenodes.mycluster</name>
  <value>nn1,nn2</value>
</property>
<property>
  <name>dfs.ha.namenodes.mycluster2</name>
  <value>nn3,nn4</value>
</property>

<!--映射namenode逻辑名称到真实主机名称(RPC) mycluster -->
<property>
  <name>dfs.namenode.rpc-address.mycluster.nn1</name>
  <value>hadoop1:8020</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.mycluster.nn2</name>
  <value>hadoop2:8020</value>
</property>
<!--映射namenode逻辑名称到真实主机名称(RPC) mycluster2 -->
<property>
  <name>dfs.namenode.rpc-address.mycluster2.nn3</name>
  <value>hadoop3:8020</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.mycluster2.nn4</name>
  <value>hadoop4:8020</value>
</property>

<!--映射namenode逻辑名称到真实主机名称(HTTP) mycluster-->
<property>
  <name>dfs.namenode.http-address.mycluster.nn1</name>
  <value>hadoop1:50070</value>
</property>
<property>
  <name>dfs.namenode.http-address.mycluster.nn2</name>
  <value>hadoop2:50070</value>
</property>
<!--映射namenode逻辑名称到真实主机名称(HTTP) mycluster2-->
<property>
  <name>dfs.namenode.http-address.mycluster2.nn3</name>
  <value>hadoop3:50070</value>
</property>
<property>
  <name>dfs.namenode.http-address.mycluster2.nn4</name>
  <value>hadoop4:50070</value>
</property>
<!--配置journalnode集群位置信息及目录-->
<property>
  <name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoop1:8485;hadoop2:8485;hadoop3:8485/mycluster</value>
</property>
<property>
  <name>dfs.journalnode.edits.dir</name>
  <value>/var/test/hadoop/fed/jn</value>
</property>
<!--配置故障切换实现类-->
<property>
  <name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
  <name>dfs.client.failover.proxy.provider.mycluster2</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!--指定切换方式为SSH免密钥方式-->
<property>
  <name>dfs.ha.fencing.methods</name>
  <value>sshfence</value>
</property>
<property>
  <name>dfs.ha.fencing.ssh.private-key-files</name>
  <value>/root/.ssh/id_dsa</value>
</property>
<!--设置自动切换-->
<property>
   <name>dfs.ha.automatic-failover.enabled.mycluster</name>
   <value>true</value>
</property>
<property>
   <name>dfs.ha.automatic-failover.enabled.mycluster2</name>
   <value>true</value>
</property>
</configuration>

2.2 在hadoop1上修改core-site.xml文件，将原有配置替换如下
[root@hadoop1 hadoop]# vim core-site.xml
<configuration>
<property>
    <name>fs.defaultFS</name>
        <value>hdfs://mycluster</value>
</property>
<!--设置zookeeper数据存放目录-->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/var/test/hadoop/fed</value>
</property>
<!--设置zookeeper位置信息-->
<property>
        <name>ha.zookeeper.quorum</name>
        <value>hadoop2:2181,hadoop3:2181,hadoop4:2181</value>
    </property>
</configuration>

2.3 在hadoop1上操作，将更新后的hdfs-site.xml,core-site.xml分发到其他节点

[root@hadoop1 hadoop]# scp hdfs-site.xml core-site.xml hadoop2:`pwd`
[root@hadoop1 hadoop]# scp hdfs-site.xml core-site.xml hadoop3:`pwd`
[root@hadoop1 hadoop]# scp hdfs-site.xml core-site.xml hadoop4:`pwd`

2.4 修改hadoop3上的hdfs-site.xml和core-site.xml文件

2.4.1 进入$HADOOP_HOME/etc/hadoop目录
[root@hadoop3 ~]# cd /opt/test/hadoop-2.6.5/etc/hadoop

2.4.2 在hadoop3上修改hdfs-site.xml文件的dfs.namenode.shared.edits.dir属性
[root@hadoop3 hadoop]# vim hdfs-site.xml
<!--配置journalnode集群位置信息及目录-->
<property>
  <name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoop1:8485;hadoop2:8485;hadoop3:8485/mycluster2</value>
</property>

2.4.3 在haoop3上修改core-site.xml文件的fs.defaultFS属性
[root@hadoop3 hadoop]# vim core-site.xml
<property>
    <name>fs.defaultFS</name>
        <value>hdfs://mycluster2</value>
</property>

3. 首次启动HA+Federation集群part1：启动journalnode和zookeeper，格式化zookeeper集群

3.1 在hadoop1,hadoop2,hadoop3上启动journalnode

[root@hadoop1 ~]# hadoop-daemon.sh start journalnode
[root@hadoop2 ~]# hadoop-daemon.sh start journalnode
[root@hadoop3 ~]# hadoop-daemon.sh start journalnode

3.1.1 hadoop1, hadoop2, hadoop3, hadoop4进程显示如下
[root@hadoop1 ~]# jps
[root@hadoop2 ~]# jps
[root@hadoop3 ~]# jps
1*** JournalNode
1*** Jps

[root@hadoop3 ~]# jps
jps

3.2 在hadoop2,hadoop3,hadoop4上分别启动zookeeper

[root@hadoop2 ~]# zkServer.sh start
[root@hadoop3 ~]# zkServer.sh start
[root@hadoop4 ~]# zkServer.sh start

3.3 在hadoop1和hadoop3上格式化zookeeper

[root@hadoop1 ~]# hdfs zkfc -formatZK
[root@hadoop3 ~]# hdfs zkfc -formatZK

3.3.1 格式化zookeeper后在hadoop2,hadoop3,hadoop4查看zookeeper进程
[root@hadoop2 ~]# zkCli.sh 
[root@hadoop2 ~]# zkCli.sh 
[root@hadoop2 ~]# zkCli.sh 
Connecting to localhost:2181

[zk: localhost:2181(CONNECTED) 0] ls /
[hadoop-ha, zookeeper]
[zk: localhost:2181(CONNECTED) 1] ls /hadoop-ha
[mycluster, mycluster2]
[zk: localhost:2181(CONNECTED) 2]

4. 首次启动HA+Federation集群part2：格式化第一组的namenode，即hadoop1

4.1 在hadoop1上操作，指定clusterid格式化namenode,

4.1.1 命令
hadoop namenode -format -clusterid ${CLUSTER_ID}
${CLUSTER_ID}为自行指定的clusterID，本例中使用cluster1

4.1.2 执行命令
[root@hadoop1 ~]# hadoop namenode -format -clusterid cluster1

4.2 格式化完成后在hadoop1上启动namenode

[root@hadoop1 ~]# hadoop-daemon.sh start namenode
starting namenode, logging to /opt/test/hadoop-2.6.5/logs/hadoop-root-namenode-hadoop1.out

4.3 hadoop1进程显示如下

[root@hadoop1 ~]# jps
**** Jps
**** JournalNode
**** NameNode

4.4 在hadoop2，即另一台namenode上同步hadoop1的CID等信息

[root@hadoop2 ~]# hdfs namenode -bootstrapStandby

4.5 在备用namenode，即hadoop2上启动namenode

[root@hadoop2 ~]# hadoop-daemon.sh start namenode

4.5.1 在hadoop2上查看进程
[root@hadoop2 ~]# jps
1406 JournalNode
1476 QuorumPeerMain
1710 NameNode
1791 Jps

5. 首次启动HA+Federation集群part3：格式化第一组的namenode，即hadoop3

#备注：步骤4和步骤5除操作的虚拟机不同，过程完全相同

5.1 在hadoop3上操作，指定clusterid格式化namenode,

[root@hadoop3 ~]# hadoop namenode -format -clusterid cluster1

5.2 格式化完成后在hadoop3上启动namenode

[root@hadoop3 ~]# hadoop-daemon.sh start namenode
starting namenode, logging to /opt/test/hadoop-2.6.5/logs/hadoop-root-namenode-hadoop3.out

5.3 hadoop3进程显示如下

[root@hadoop3 ~]# jps
**** Jps
**** JournalNode
**** NameNode

5.4 在hadoop4，即该组另一台namenode上同步hadoop3的CID等信息

[root@hadoop4 ~]# hdfs namenode -bootstrapStandby

5.5 在备用namenode上启动namenode

[root@hadoop4 ~]# hadoop-daemon.sh start namenode

5.5.1 在hadoop4上查看进程
[root@hadoop4 ~]# jps
1407 QuorumPeerMain
1615 NameNode
1696 Jps

6. 首次启动HA+Federation集群part4：启动ZKFC, datanode和yarn

6.1 启动zkfc

6.1.1 在hadoop1,hadoop2,hadoop3,hadoop4上启动zkfc
[root@hadoop1 ~]# hadoop-daemon.sh start zkfc
[root@hadoop2 ~]# hadoop-daemon.sh start zkfc
[root@hadoop3 ~]# hadoop-daemon.sh start zkfc
[root@hadoop4 ~]# hadoop-daemon.sh start zkfc
starting zkfc, logging to /opt/test/hadoop-2.6.5/logs/hadoop-root-zkfc-hadoop*.out

6.1.2 启动zkfc后在hadoop2,hadoop3,hadoop4查看已有进程
[root@hadoop1 ~]# jps
1404 JournalNode
1727 DFSZKFailoverController
1601 NameNode
1794 Jps

[root@hadoop2 ~]# jps
1668 Jps
1406 JournalNode
1476 QuorumPeerMain
1623 DFSZKFailoverController

[root@hadoop3 ~]# jps
1836 Jps
1404 JournalNode
1474 QuorumPeerMain
1664 NameNode
1769 DFSZKFailoverController

[root@hadoop4 ~]# jps
1407 QuorumPeerMain
1546 DFSZKFailoverController
1591 Jps

6.2 启动datanode

6.2.1 在active的namenode上启动datanode
[root@hadoop1 ~]# hadoop-daemons.sh start datanode

# 因为hadoop1已经启动了所有datanode，不用在hadoop3上重复启动

6.2.2 启动datanode后查看hadoop1,hadoop2,hadoop3,hadoop4进程
[root@hadoop1 ~]# jps
1404 JournalNode
1885 Jps
1727 DFSZKFailoverController
1601 NameNode

[root@hadoop2 ~]# jps
1406 JournalNode
1476 QuorumPeerMain
2007 Jps
1710 NameNode
1911 DataNode
1623 DFSZKFailoverController

[root@hadoop3 ~]# jps
1404 JournalNode
1991 Jps
1474 QuorumPeerMain
1664 NameNode
1904 DataNode
1769 DFSZKFailoverController

[root@hadoop4 ~]# jps
1407 QuorumPeerMain
1546 DFSZKFailoverController
1615 NameNode
1811 DataNode
1908 Jps

6.3 启动yarn

6.3.1 在hadoop1上启动yarn
[root@hadoop1 ~]# start-yarn.sh

6.3.2 启动yarn后查看hadoop1, hadoop2, hadoop3, hadoop4进程
[root@hadoop1 ~]# jps
1404 JournalNode
1727 DFSZKFailoverController
1601 NameNode
1947 ResourceManager
2199 Jps

[root@hadoop2 ~]# jps
1406 JournalNode
1476 QuorumPeerMain
2114 Jps
1710 NameNode
1911 DataNode
1623 DFSZKFailoverController
2078 NodeManager

[root@hadoop3 ~]# jps
1404 JournalNode
2035 NodeManager
2068 Jps
1474 QuorumPeerMain
1664 NameNode
1904 DataNode
1769 DFSZKFailoverController

[root@hadoop4 ~]# jps
1407 QuorumPeerMain
2035 Jps
1546 DFSZKFailoverController
1970 NodeManager
1615 NameNode
1811 DataNode

6.4 通过web查看集群运行状态,

http://192.168.111.211:50070/dfshealth.html#tab-overview
'hadoop1:8020' (active)
http://192.168.111.212:50070/dfshealth.html#tab-overview
'hadoop1:8020' (standby)
http://192.168.111.213:50070/dfshealth.html#tab-overview
'hadoop1:8020' (active)
http://192.168.111.214:50070/dfshealth.html#tab-overview
'hadoop1:8020' (standby)

7. 常规启动HA+Federation集群

7.1 在hadoop2, hadoop3, hadoop4上启动zookeeper

[root@hadoop2 ~]# zkServer.sh start
[root@hadoop3 ~]# zkServer.sh start
[root@hadoop4 ~]# zkServer.sh start
JMX enabled by default
Using config: /opt/test/zookeeper-3.4.6/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

7.1.1 在hadoop2, hadoop3, hadoop4上查看进程
[root@hadoop2 ~]# jps
[root@hadoop3 ~]# jps
[root@hadoop4 ~]# jps
2*** Jps
2*** QuorumPeerMain

7.2 在hadoop1上执行start-dfs.sh

[root@hadoop1 ~]# start-dfs.sh
Starting namenodes on [hadoop1 hadoop2 hadoop3 hadoop4]
hadoop*: starting namenode, logging to /opt/test/hadoop-2.6.5/logs/hadoop-root-namenode-hadoop*.out
hadoop*: starting datanode, logging to /opt/test/hadoop-2.6.5/logs/hadoop-root-datanode-hadoop*.out
Starting journal nodes [hadoop1 hadoop2 hadoop3]
Hadoop*: starting journalnode, logging to /opt/test/hadoop-2.6.5/logs/hadoop-root-journalnode-hadoop*.out

7.2.1 在hadoop1, hadoop2, hadoop3, hadoop4上查看进程
[root@hadoop1 mapreduce]# jps
3243 Jps
2926 NameNode 
3121 JournalNode

[root@hadoop2 ~]# jps
2565 Jps
2240 QuorumPeerMain 
2305 NameNode
2462 JournalNode
2371 DataNode

[root@hadoop3 ~]# jps
3095 Jps 
2767 QuorumPeerMain
2832 NameNode
2989 JournalNode
2901 DataNode

[root@hadoop4 ~]# jps
2389 Jps
2133 QuorumPeerMain
2204 NameNode 
2270 DataNode

7.3 在hadoop1上启动yarn

[root@hadoop1 ~]# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/test/hadoop-2.6.5/logs/yarn-root-resourcemanager-hadoop1.out
hadoop*: starting nodemanager, logging to /opt/test/hadoop-2.6.5/logs/yarn-root-nodemanager-hadoop*.out

7.3.1 在hadoop1,hadoop2,hadoop3,hadoop4上查看进程
[root@hadoop1 mapreduce]# jps
3588 Jps
2926 NameNode
3121 JournalNode
3313 ResourceManager

[root@hadoop2 ~]# jps
2773 Jps
2240 QuorumPeerMain
2305 NameNode
2462 JournalNode
2371 DataNode
2626 NodeManager

[root@hadoop3 ~]# jps
3308 Jps
2767 QuorumPeerMain
2832 NameNode
2989 JournalNode
2901 DataNode
3155 NodeManager

[root@hadoop4 ~]# jps
2598 Jps
2133 QuorumPeerMain
2204 NameNode
2270 DataNode
2451 NodeManager

7.4 在hadoop1,hadoop2,hadoop3,hadoop4上启动zkfc

[root@hadoop1 ~]# hadoop-daemon.sh start zkfc
[root@hadoop2 ~]# hadoop-daemon.sh start zkfc
[root@hadoop3 ~]# hadoop-daemon.sh start zkfc
[root@hadoop4 ~]# hadoop-daemon.sh start zkfc
starting zkfc, logging to /opt/test/hadoop-2.6.5/logs/hadoop-root-zkfc-hadoop*.out

7.4.1 在hadoop1,hadoop2,hadoop3,hadoop4上查看进程
[root@hadoop1 mapreduce]# jps
3588 Jps
2926 NameNode
3121 JournalNode
3313 ResourceManager
3641 DFSZKFailoverController

[root@hadoop2 ~]# jps
2773 Jps
2240 QuorumPeerMain
2305 NameNode
2462 JournalNode
2371 DataNode
2626 NodeManager
2826 DFSZKFailoverController

[root@hadoop3 ~]# jps
3308 Jps
2767 QuorumPeerMain
2832 NameNode
2989 JournalNode
2901 DataNode
3155 NodeManager
3362 DFSZKFailoverController

[root@hadoop4 ~]# jps
2598 Jps
2133 QuorumPeerMain
2204 NameNode
2270 DataNode
2451 NodeManager
2651 DFSZKFailoverController

7.5 通过web查看集群运行状态,

http://192.168.111.211:50070/dfshealth.html#tab-overview
'hadoop1:8020' (active)
http://192.168.111.212:50070/dfshealth.html#tab-overview
'hadoop1:8020' (standby)
http://192.168.111.213:50070/dfshealth.html#tab-overview
'hadoop1:8020' (active)
http://192.168.111.214:50070/dfshealth.html#tab-overview
'hadoop1:8020' (standby)

8. 在HA+Federation集群上测试wordcount程序

8.1 从hadoop1或hadoop3进入$HADOOP_HOME/share/hadoop/mapreduce/
目录，本例选择hadoop1

[root@hadoop1 ~]# cd /opt/test/hadoop-2.6.5/share/hadoop/mapreduce/

8.2 上传test.txt文件到根目录

8.2.1 默认上传
[root@hadoop1 mapreduce]# hadoop fs -put test.txt /

8.2.2 也可以指定blocksize
[root@hadoop1 mapreduce]# hdfs dfs -D dfs.blocksize=1048576 -put test.txt /

8.3 运行wordcount测试程序，输出到/output

[root@hadoop1 mapreduce]# 
hadoop jar hadoop-mapreduce-examples-2.6.5.jar wordcount /test.txt /output
#运行时会首先看到如下信息
INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032

8.4 查看mapreduce运行结果

[root@hadoop1 mapreduce]# hadoop dfs -text /output/part-*
hello    100003
world    200002
“hello    100000

9. 为HA+Federation(高可用+联邦)配置viewfs

9.1 进入$HADOOP_HOME/etc/hadoop目录

[root@hadoop1 ~]# cd /opt/test/hadoop-2.6.5/etc/hadoop

9.2 在hadoop1上修改core-site.xml文件，将原有配置替换如下

[root@hadoop1 hadoop]# vim core-site.xml
<configuration xmlns:xi="http://www.w3.org/2001/XInclude"> 
<!--cmt.xml前使用绝对路径-->
<xi:include href="/opt/test/hadoop-2.6.5/etc/hadoop/cmt.xml" />
<property>
        <name>fs.default.name</name>
        <value>viewfs://clusterX</value>
</property>
<!--设置zookeeper数据存放目录-->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/var/test/hadoop/fed</value>
</property>
<!--设置zookeeper位置信息-->
<property>
        <name>ha.zookeeper.quorum</name>
        <value>hadoop2:2181,hadoop3:2181,hadoop4:2181</value>
    </property>
</configuration>

9.3 在hadoop1的/opt/test/hadoop-2.6.5/etc/hadoop目录下新增cmt.xml文件

[root@hadoop1 hadoop]# vim cmt.xml
<configuration>
  <property>
    <name>fs.viewfs.mounttable.clusterX.link./ns1</name>
    <value>hdfs://mycluster</value>
  </property>
  <property>
    <name>fs.viewfs.mounttable.clusterX.link./ns2</name>
    <value>hdfs://mycluster2</value>
  </property>
<property> 
    <!-- 指定 /tmp 目录，许多依赖hdfs的组件可能会用到此目录 -->
    <name>fs.viewfs.mounttable.clusterX.link./tmp</name>  
    <value>hdfs://mycluster/tmp</value> 
  </property>
</configuration>

10. 在HA+Federation+viewFs集群上测试wordcount程序

10.1 从hadoop1或hadoop3进入$HADOOP_HOME/share/hadoop/mapreduce/目录，本例选择hadoop1

[root@hadoop1 ~]# cd /opt/test/hadoop-2.6.5/share/hadoop/mapreduce/

10.2上传test.txt文件到根目录

10.2.1 默认上传
[root@hadoop1 mapreduce]# hadoop fs -put test.txt /

10.2.2 也可以指定blocksize
[root@hadoop1 mapreduce]# hdfs dfs -D dfs.blocksize=1048576 -put test.txt /

10.3 运行wordcount测试程序，输出到/output

[root@hadoop1 mapreduce]# 
hadoop jar hadoop-mapreduce-examples-2.6.5.jar wordcount /ns1/test.txt /ns1/output

# 运行时会首先看到如下信息
INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032

10.4 查看mapreduce运行结果

[root@hadoop1 mapreduce]# hadoop dfs -text /ns1/output/part-*
hello    100003
world    200002
“hello    100000

参考建议

a). 《HDFS Federation和NameNode HA的搭建》

1. 整个集群,4个虚拟机hadoop1,haoop2,hadoop3,hadoop4共用一个clusterID;
2. 保证两个NameService可以共享所有的DataNodes;
否则两个NameService在format之后，生成的clusterid不一致，DataNode会随机注册到不同的NameNode上

b). 《HDFS Federation机制》

1. 新建Hadoop集群的时候,最好使用同一个cluster Id来format,以后要归并入同一个集群也会很容易
2. 管理多套独立的集群与HDFS Federation模式下的集群不同
2.1 在HDFS Federation下,可以统一进行节点的下线,上线,Balance等操作,完全就是在一个集群内的运维操作;
2.2 对于多套独立的集群,这意味着重复操作
2.3 对于HDFS Federation引入的多nameservice,会让客户端程序维护多个nameservice,以及这些对应namespace所存储的具体文件目录
2.4 namespace多了,一个优化的做法是用viewFs来解决
2.5 在客户端配置上增加一个mount table.让客户端访问的是一个逻辑意义上的file system,无须更改目标指向的file system.
2.6 这样可以同时应用HDFS Federation和viewFs的优势

c). hadoop HA+Federation（高可用联邦）模式搭建指南

d). Hadoop 2.0 NameNode HA和Federation实践

e). HDFS Federation客户端(viewfs)配置攻略

码农公寓