前言：
本文章是在 Hadoop2.7.6_07_HA高可用 的基础上完成的，所以不清楚的可参见这篇文章。
1. Hadoop的federation机制
文件的元数据是放在namenode上的，只有一个Namespace（命名空间）。随着HDFS的数据越来越多，单个namenode的资源使用必然会达到上限，而且namenode的负载能力也会越来越高，限制HDFS的性能。
Federation即为“联邦”，该特性允许一个HDFS集群中存在多个NameNode同时对外提供服务，这些NameNode分管一部分目录（水平切分），彼此之间相互隔离，但共享底层的DataNode存储资源。
1.1. HDFS-federation图解
2. 主机规划
|
主机名称 |
外网IP |
内网IP |
操作系统 |
备注 |
安装软件 |
运行进程 |
|
mini01 |
10.0.0.111 |
172.16.1.111 |
CentOS 7.4 |
ssh port:22 |
jdk、hadoop |
NameNode、DFSZKFailoverController(zkfc) |
|
mini02 |
10.0.0.112 |
172.16.1.112 |
CentOS 7.4 |
ssh port:22 |
jdk、hadoop |
NameNode、DFSZKFailoverController(zkfc) |
|
mini03 |
10.0.0.113 |
172.16.1.113 |
CentOS 7.4 |
ssh port:22 |
jdk、hadoop、zookeeper |
NameNode、ResourceManager |
|
mini04 |
10.0.0.114 |
172.16.1.114 |
CentOS 7.4 |
ssh port:22 |
jdk、hadoop、zookeeper |
NameNode、ResourceManager |
|
mini05 |
10.0.0.115 |
172.16.1.115 |
CentOS 7.4 |
ssh port:22 |
jdk、hadoop、zookeeper |
DataNode、NodeManager、JournalNode、QuorumPeerMain |
|
mini06 |
10.0.0.116 |
172.16.1.116 |
CentOS 7.4 |
ssh port:22 |
jdk、hadoop、zookeeper |
DataNode、NodeManager、JournalNode、QuorumPeerMain |
|
mini07 |
10.0.0.117 |
172.16.1.117 |
CentOS 7.4 |
ssh port:22 |
jdk、hadoop、zookeeper |
DataNode、NodeManager、JournalNode、QuorumPeerMain |
注意：此规划是在HA的基础上进行的federation，即 mini01和mini02做HA，mini03和mini04做HA；之后再做federation。
HA的配置请参见：Hadoop2.7.6_07_HA高可用
Linux添加hosts信息，保证每台都可以相互ping通
[root@mini01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
:: localhost localhost.localdomain localhost6 localhost6.localdomain6 10.0.0.111 mini01
10.0.0.112 mini02
10.0.0.113 mini03
10.0.0.114 mini04
10.0.0.115 mini05
10.0.0.116 mini06
10.0.0.117 mini07
Windows的hosts文件修改
# 文件位置C:\Windows\System32\drivers\etc 在hosts中追加如下内容
…………………………………………
10.0.0.111 mini01
10.0.0.112 mini02
10.0.0.113 mini03
10.0.0.114 mini04
10.0.0.115 mini05
10.0.0.116 mini06
10.0.0.117 mini07
2.1. 注意事项
其中添加yun用户、实现yun用户免秘钥登录、Jdk【java8】、Zookeeper部署；请参见Hadoop2.7.6_07_HA高可用
3. Hadoop部署与配置修改
注意：每台机器的Hadoop以及配置相同
3.1. 部署
[yun@mini01 software]$ pwd
/app/software
[yun@mini01 software]$ ll
total
-rw-r--r-- yun yun Jun : CentOS-.4_hadoop-2.7..tar.gz
[yun@mini01 software]$ tar xf CentOS-.4_hadoop-2.7..tar.gz
[yun@mini01 software]$ mv hadoop-2.7./ /app/
[yun@mini01 software]$ cd
[yun@mini01 ~]$ ln -s hadoop-2.7./ hadoop
[yun@mini01 ~]$ ll
total
lrwxrwxrwx yun yun Jun : hadoop -> hadoop-2.7./
drwxr-xr-x yun yun Jun : hadoop-2.7.
lrwxrwxrwx yun yun May : jdk -> jdk1..0_112
drwxr-xr-x yun yun Sep jdk1..0_112
3.2. 环境变量
[root@mini01 profile.d]# pwd
/etc/profile.d
[root@mini01 profile.d]# vim hadoop.sh
export HADOOP_HOME="/app/hadoop"
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH [root@mini01 profile.d]# source /etc/profile # 生效
3.3. core-site.xml
[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
………………
--> <!-- Put site-specific property overrides in this file. --> <configuration>
<property>
<name>fs.defaultFS</name>
<value>viewfs:///</value>
</property> <property>
<name>fs.viewfs.mounttable.default.link./bi</name>
<value>hdfs://bi/</value>
</property> <property>
<name>fs.viewfs.mounttable.default.link./dt</name>
<value>hdfs://dt/</value>
</property> <!-- 指定hadoop临时目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
</property> <!-- 指定zookeeper地址 -->
<property>
<name>ha.zookeeper.quorum</name>
<value>mini03:,mini04:,mini05:,mini06:,mini07:</value>
</property> </configuration>
3.4. hdfs-site.xml
3.4.1. 在mini01和mini02中
# 在mini01、mini02、mini05、mini06、mini07 可使用如下配置
[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
……………………
--> <!-- Put site-specific property overrides in this file. --> <configuration>
<!--指定hdfs的nameservice为bi,dt，需要和core-site.xml中的保持一致 -->
<property>
<name>dfs.nameservices</name>
<value>bi,dt</value>
</property> <!-- bi下面有两个NameNode，分别是nn1，nn2 -->
<property>
<name>dfs.ha.namenodes.bi</name>
<value>nn1,nn2</value>
</property> <!-- dt下面有两个NameNode，分别是nn3，nn4 -->
<property>
<name>dfs.ha.namenodes.dt</name>
<value>nn3,nn4</value>
</property> <!-- nn1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.bi.nn1</name>
<value>mini01:</value>
</property>
<!-- nn1的http通信地址 -->
<property>
<name>dfs.namenode.http-address.bi.nn1</name>
<value>mini01:</value>
</property> <!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.bi.nn2</name>
<value>mini02:</value>
</property>
<!-- nn2的http通信地址 -->
<property>
<name>dfs.namenode.http-address.bi.nn2</name>
<value>mini02:</value>
</property> <!-- nn3的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.dt.nn3</name>
<value>mini03:</value>
</property>
<!-- nn3的http通信地址 -->
<property>
<name>dfs.namenode.http-address.dt.nn3</name>
<value>mini03:</value>
</property> <!-- nn4的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.dt.nn4</name>
<value>mini04:</value>
</property>
<!-- nn4的http通信地址 -->
<property>
<name>dfs.namenode.http-address.dt.nn4</name>
<value>mini04:</value>
</property> <!-- 指定NameNode的edits元数据在JournalNode上的存放位置 -->
<!-- 在bi名称空间的两个namenode中用如下配置 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://mini05:8485;mini06:8485;mini07:8485/bi</value>
</property>
<!-- 在dt名称空间的两个namenode中用如下配置 # 在mini01和mini02中注释掉 -->
<!--
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://mini05:8485;mini06:8485;mini07:8485/dt</value>
</property>
--> <!-- 指定JournalNode在本地磁盘存放数据的位置 -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/app/hadoop/journaldata</value>
</property> <!-- 开启NameNode失败自动切换 -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property> <!-- 配置失败自动切换实现方式 -->
<property>
<name>dfs.client.failover.proxy.provider.bi</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.dt</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property> <!-- 配置隔离机制方法，多个机制用换行分割，即每个机制暂用一行-->
<!-- 其中shell(/bin/true) 表示可执行一个脚本 比如 shell(/app/yunwei/hadoop_fence.sh) -->
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property> <!-- 使用sshfence隔离机制时需要ssh免登陆 -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/app/.ssh/id_rsa</value>
</property> <!-- 配置sshfence隔离机制超时时间 单位：毫秒 -->
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value></value>
</property> </configuration>
3.4.2. 在mini03和mini04中
# 在mini03、mini04 可使用如下配置
[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
……………………
--> <!-- Put site-specific property overrides in this file. --> <configuration>
<!--指定hdfs的nameservice为bi,dt，需要和core-site.xml中的保持一致 -->
<property>
<name>dfs.nameservices</name>
<value>bi,dt</value>
</property> <!-- bi下面有两个NameNode，分别是nn1，nn2 -->
<property>
<name>dfs.ha.namenodes.bi</name>
<value>nn1,nn2</value>
</property> <!-- dt下面有两个NameNode，分别是nn3，nn4 -->
<property>
<name>dfs.ha.namenodes.dt</name>
<value>nn3,nn4</value>
</property> <!-- nn1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.bi.nn1</name>
<value>mini01:</value>
</property>
<!-- nn1的http通信地址 -->
<property>
<name>dfs.namenode.http-address.bi.nn1</name>
<value>mini01:</value>
</property> <!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.bi.nn2</name>
<value>mini02:</value>
</property>
<!-- nn2的http通信地址 -->
<property>
<name>dfs.namenode.http-address.bi.nn2</name>
<value>mini02:</value>
</property> <!-- nn3的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.dt.nn3</name>
<value>mini03:</value>
</property>
<!-- nn3的http通信地址 -->
<property>
<name>dfs.namenode.http-address.dt.nn3</name>
<value>mini03:</value>
</property> <!-- nn4的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.dt.nn4</name>
<value>mini04:</value>
</property>
<!-- nn4的http通信地址 -->
<property>
<name>dfs.namenode.http-address.dt.nn4</name>
<value>mini04:</value>
</property> <!-- 指定NameNode的edits元数据在JournalNode上的存放位置 -->
<!-- 在bi名称空间的两个namenode中用如下配置 # 在mini03和mini04中注释掉 -->
<!--
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://mini05:8485;mini06:8485;mini07:8485/bi</value>
</property>
-->
<!-- 在dt名称空间的两个namenode中用如下配置 # 在mini01和mini02中注释掉 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://mini05:8485;mini06:8485;mini07:8485/dt</value>
</property> <!-- 指定JournalNode在本地磁盘存放数据的位置 -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/app/hadoop/journaldata</value>
</property> <!-- 开启NameNode失败自动切换 -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property> <!-- 配置失败自动切换实现方式 -->
<property>
<name>dfs.client.failover.proxy.provider.bi</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.dt</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property> <!-- 配置隔离机制方法，多个机制用换行分割，即每个机制暂用一行-->
<!-- 其中shell(/bin/true) 表示可执行一个脚本 比如 shell(/app/yunwei/hadoop_fence.sh) -->
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property> <!-- 使用sshfence隔离机制时需要ssh免登陆 -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/app/.ssh/id_rsa</value>
</property> <!-- 配置sshfence隔离机制超时时间 单位：毫秒 -->
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value></value>
</property> </configuration>
3.5. mapred-site.xml
# 较上篇文章无改变
[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ cp -a mapred-site.xml.template mapred-site.xml
[yun@mini01 hadoop]$ vim mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
……………………
--> <!-- Put site-specific property overrides in this file. --> <configuration>
<!-- 指定mr框架为yarn方式 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property> </configuration>
3.6. yarn-site.xml
# 较上篇文章无改变
[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim yarn-site.xml
<?xml version="1.0"?>
<!--
……………………
-->
<configuration> <!-- Site specific YARN configuration properties -->
<!-- 开启RM高可用 -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property> <!-- 指定RM的cluster id -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yrc</value>
</property> <!-- 指定RM的名字 -->
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property> <!-- 分别指定RM的地址 -->
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>mini03</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>mini04</value>
</property> <!-- 指定zk集群地址 -->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>mini03:,mini04:,mini05:,mini06:,mini07:</value>
</property> <!-- reduce 获取数据的方式 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property> </configuration>
3.7. 修改slaves
slaves是指定子节点的位置，因为要在mini01上启动HDFS、在mini03启动yarn，所以mini01上的slaves文件指定的是datanode的位置，mini03上的slaves文件指定的是nodemanager的位置
# 较上篇文章无改变
[yun@mini01 hadoop]$ pwd
/app/hadoop/etc/hadoop
[yun@mini01 hadoop]$ vim slaves
mini05
mini06
mini07
PS：改后配置后，将这些配置拷到其他Hadoop机器
4. 启动相关服务
注意：第一次启动时严格按照下面的步骤！！！！！！！
4.1. 启动zookeeper集群
前面已经启动了，这里就不说了
4.2. 启动journalnode
# 根据规划在mini05、mini06、mini07 启动
# 在第一次格式化的时候需要先启动journalnode 之后就不必了
[yun@mini05 ~]$ hadoop-daemon.sh start journalnode # 已经配置环境变量，所以不用进入到响应的目录
starting journalnode, logging to /app/hadoop-2.7./logs/hadoop-yun-journalnode-mini05.out
[yun@mini05 ~]$ jps
QuorumPeerMain
Jps
JournalNode
4.3. 在bi的格式化
4.3.1. 格式化HDFS
# 在mini01上执行命令
[yun@mini01 ~]$ hdfs namenode -format -clusterID CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f # 其中UUID可以在网上生成一个
// :: INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = mini01/10.0.0.111
STARTUP_MSG: args = [-format, -clusterID, CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f]
STARTUP_MSG: version = 2.7.6
STARTUP_MSG: classpath = ………………
STARTUP_MSG: build = Unknown -r Unknown; compiled by 'root' on 2018-06-08T08:30Z
STARTUP_MSG: java = 1.8.0_112
************************************************************/
// :: INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
// :: INFO namenode.NameNode: createNameNode [-format, -clusterID, CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f]
Formatting using clusterid: CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f
// :: INFO namenode.FSNamesystem: No KeyProvider found.
// :: INFO namenode.FSNamesystem: fsLock is fair: true
// :: INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
// :: INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=
// :: INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
// :: INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to :::00.000
// :: INFO blockmanagement.BlockManager: The block deletion will start around Jul ::
// :: INFO util.GSet: Computing capacity for map BlocksMap
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
// :: INFO blockmanagement.BlockManager: defaultReplication =
// :: INFO blockmanagement.BlockManager: maxReplication =
// :: INFO blockmanagement.BlockManager: minReplication =
// :: INFO blockmanagement.BlockManager: maxReplicationStreams =
// :: INFO blockmanagement.BlockManager: replicationRecheckInterval =
// :: INFO blockmanagement.BlockManager: encryptDataTransfer = false
// :: INFO blockmanagement.BlockManager: maxNumBlocksToLog =
// :: INFO namenode.FSNamesystem: fsOwner = yun (auth:SIMPLE)
// :: INFO namenode.FSNamesystem: supergroup = supergroup
// :: INFO namenode.FSNamesystem: isPermissionEnabled = true
// :: INFO namenode.FSNamesystem: Determined nameservice ID: bi
// :: INFO namenode.FSNamesystem: HA Enabled: true
// :: INFO namenode.FSNamesystem: Append Enabled: true
// :: INFO util.GSet: Computing capacity for map INodeMap
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO namenode.FSDirectory: ACLs enabled? false
// :: INFO namenode.FSDirectory: XAttrs enabled? true
// :: INFO namenode.FSDirectory: Maximum size of an xattr:
// :: INFO namenode.NameNode: Caching file names occuring more than times
// :: INFO util.GSet: Computing capacity for map cachedBlocks
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes =
// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.extension =
// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets =
// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users =
// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = ,,
// :: INFO namenode.FSNamesystem: Retry cache on namenode is enabled
// :: INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is millis
// :: INFO util.GSet: Computing capacity for map NameNodeRetryCache
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO namenode.FSImage: Allocated new BlockPoolId: BP--10.0.0.111-
// :: INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted.
// :: INFO namenode.FSImageFormatProtobuf: Saving image file /app/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
// :: INFO namenode.FSImageFormatProtobuf: Image file /app/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size bytes saved in seconds.
// :: INFO namenode.NNStorageRetentionManager: Going to retain images with txid >=
// :: INFO util.ExitUtil: Exiting with status
// :: INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at mini01/10.0.0.111
************************************************************/
拷贝到mini02
#格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文件，这里我配置的是/app/hadoop/tmp，然后将/app/hadoop/tmp拷贝到mini02的/app/hadoop/下。
# 方法1：
[yun@mini01 hadoop]$ pwd
/app/hadoop
[yun@mini01 hadoop]$ scp -r tmp/ yun@mini02:/app/hadoop
VERSION % .4KB/s :
seen_txid % .0KB/s :
fsimage_0000000000000000000.md5 % .7KB/s :
fsimage_0000000000000000000 % .1KB/s : ##########################
# 方法2：##也可以这样，建议hdfs namenode -bootstrapStandby # 不过需要mini02的Hadoop起来才行
4.3.2. 格式化ZKFC
# 在mini01上执行一次即可
[yun@mini01 current]$ hdfs zkfc -formatZK
// :: INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at mini01/10.0.0.111:
// :: INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.-, built on // : GMT
// :: INFO zookeeper.ZooKeeper: Client environment:host.name=mini01
// :: INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_112
// :: INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
// :: INFO zookeeper.ZooKeeper: Client environment:java.home=/app/jdk1..0_112/jre
// :: INFO zookeeper.ZooKeeper: Client environment:java.class.path=……………………
// :: INFO zookeeper.ZooKeeper: Client environment:java.library.path=/app/hadoop-2.7./lib/native
// :: INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
// :: INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
// :: INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
// :: INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
// :: INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.-.el7.x86_64
// :: INFO zookeeper.ZooKeeper: Client environment:user.name=yun
// :: INFO zookeeper.ZooKeeper: Client environment:user.home=/app
// :: INFO zookeeper.ZooKeeper: Client environment:user.dir=/app/hadoop-2.7.
// :: INFO zookeeper.ZooKeeper: Initiating client connection, connectString=mini03:,mini04:,mini05:,mini06:,mini07: sessionTimeout= watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@7f3b84b8
// :: INFO zookeeper.ClientCnxn: Opening socket connection to server mini04/10.0.0.114:. Will not attempt to authenticate using SASL (unknown error)
// :: INFO zookeeper.ClientCnxn: Socket connection established to mini04/10.0.0.114:, initiating session
// :: INFO zookeeper.ClientCnxn: Session establishment complete on server mini04/10.0.0.114:, sessionid = 0x464538f087b0003, negotiated timeout =
// :: INFO ha.ActiveStandbyElector: Session connected.
// :: INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/bi in ZK.
// :: INFO zookeeper.ZooKeeper: Session: 0x464538f087b0003 closed
// :: INFO zookeeper.ClientCnxn: EventThread shut down
4.4. 在dt的格式化
4.4.1. 格式化HDFS
# 在mini03上执行命令
[yun@mini03 hadoop]$ hdfs namenode -format -clusterID CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f # clusterID必须与bi的相同
// :: INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = mini03/10.0.0.113
STARTUP_MSG: args = [-format, -clusterID, CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f]
STARTUP_MSG: version = 2.7.6
STARTUP_MSG: classpath = ……………………
STARTUP_MSG: build = Unknown -r Unknown; compiled by 'root' on 2018-06-08T08:30Z
STARTUP_MSG: java = 1.8.0_112
************************************************************/
// :: INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
// :: INFO namenode.NameNode: createNameNode [-format, -clusterID, CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f]
Formatting using clusterid: CID-8c9c1d4b-5aa3-4d12-b717-31a56884da7f
// :: INFO namenode.FSNamesystem: No KeyProvider found.
// :: INFO namenode.FSNamesystem: fsLock is fair: true
// :: INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
// :: INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=
// :: INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
// :: INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to :::00.000
// :: INFO blockmanagement.BlockManager: The block deletion will start around Jul ::
// :: INFO util.GSet: Computing capacity for map BlocksMap
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
// :: INFO blockmanagement.BlockManager: defaultReplication =
// :: INFO blockmanagement.BlockManager: maxReplication =
// :: INFO blockmanagement.BlockManager: minReplication =
// :: INFO blockmanagement.BlockManager: maxReplicationStreams =
// :: INFO blockmanagement.BlockManager: replicationRecheckInterval =
// :: INFO blockmanagement.BlockManager: encryptDataTransfer = false
// :: INFO blockmanagement.BlockManager: maxNumBlocksToLog =
// :: INFO namenode.FSNamesystem: fsOwner = yun (auth:SIMPLE)
// :: INFO namenode.FSNamesystem: supergroup = supergroup
// :: INFO namenode.FSNamesystem: isPermissionEnabled = true
// :: INFO namenode.FSNamesystem: Determined nameservice ID: dt
// :: INFO namenode.FSNamesystem: HA Enabled: true
// :: INFO namenode.FSNamesystem: Append Enabled: true
// :: INFO util.GSet: Computing capacity for map INodeMap
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO namenode.FSDirectory: ACLs enabled? false
// :: INFO namenode.FSDirectory: XAttrs enabled? true
// :: INFO namenode.FSDirectory: Maximum size of an xattr:
// :: INFO namenode.NameNode: Caching file names occuring more than times
// :: INFO util.GSet: Computing capacity for map cachedBlocks
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes =
// :: INFO namenode.FSNamesystem: dfs.namenode.safemode.extension =
// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets =
// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users =
// :: INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = ,,
// :: INFO namenode.FSNamesystem: Retry cache on namenode is enabled
// :: INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is millis
// :: INFO util.GSet: Computing capacity for map NameNodeRetryCache
// :: INFO util.GSet: VM type = -bit
// :: INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
// :: INFO util.GSet: capacity = ^ = entries
// :: INFO namenode.FSImage: Allocated new BlockPoolId: BP--10.0.0.113-
// :: INFO common.Storage: Storage directory /app/hadoop/tmp/dfs/name has been successfully formatted.
// :: INFO namenode.FSImageFormatProtobuf: Saving image file /app/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
// :: INFO namenode.FSImageFormatProtobuf: Image file /app/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size bytes saved in seconds.
// :: INFO namenode.NNStorageRetentionManager: Going to retain images with txid >=
// :: INFO util.ExitUtil: Exiting with status
// :: INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at mini03/10.0.0.113
************************************************************/
拷贝到mini04
#格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文件，这里我配置的是/app/hadoop/tmp，然后将/app/hadoop/tmp拷贝到mini04的/app/hadoop/下。
# 方法1：
[yun@mini03 hadoop]$ pwd
/app/hadoop
[yun@mini03 hadoop]$ scp -r tmp/ yun@mini04:/app/hadoop
VERSION % .4KB/s :
seen_txid % .0KB/s :
fsimage_0000000000000000000.md5 % .7KB/s :
fsimage_0000000000000000000 % .1KB/s : ##########################
# 方法2：##也可以这样，建议hdfs namenode -bootstrapStandby # 不过需要mini04的Hadoop起来才行
4.4.2. 格式化ZKFC
# 在mini03上执行一次即可
[yun@mini03 hadoop]$ hdfs zkfc -formatZK
// :: INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at mini03/10.0.0.113:
// :: INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.-, built on // : GMT
// :: INFO zookeeper.ZooKeeper: Client environment:host.name=mini03
// :: INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_112
// :: INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
// :: INFO zookeeper.ZooKeeper: Client environment:java.home=/app/jdk1..0_112/jre
// :: INFO zookeeper.ZooKeeper: Client environment:java.class.path=……………………
// :: INFO zookeeper.ZooKeeper: Client environment:java.library.path=/app/hadoop-2.7./lib/native
// :: INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
// :: INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
// :: INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
// :: INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
// :: INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.-.el7.x86_64
// :: INFO zookeeper.ZooKeeper: Client environment:user.name=yun
// :: INFO zookeeper.ZooKeeper: Client environment:user.home=/app
// :: INFO zookeeper.ZooKeeper: Client environment:user.dir=/app/hadoop-2.7.
// :: INFO zookeeper.ZooKeeper: Initiating client connection, connectString=mini03:,mini04:,mini05:,mini06:,mini07: sessionTimeout= watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@7f3b84b8
// :: INFO zookeeper.ClientCnxn: Opening socket connection to server mini05/10.0.0.115:. Will not attempt to authenticate using SASL (unknown error)
// :: INFO zookeeper.ClientCnxn: Socket connection established to mini05/10.0.0.115:, initiating session
// :: INFO zookeeper.ClientCnxn: Session establishment complete on server mini05/10.0.0.115:, sessionid = 0x56455467ae50003, negotiated timeout =
// :: INFO ha.ActiveStandbyElector: Session connected.
// :: INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/dt in ZK.
// :: INFO zookeeper.ZooKeeper: Session: 0x56455467ae50003 closed
// :: INFO zookeeper.ClientCnxn: EventThread shut down
4.5. 启动HDFS
# 在mini01操作一次即可
[yun@mini01 hadoop]$ start-dfs.sh
Starting namenodes on [mini01 mini02 mini03 mini04]
mini01: starting namenode, logging to /app/hadoop-2.7./logs/hadoop-yun-namenode-mini01.out
mini02: starting namenode, logging to /app/hadoop-2.7./logs/hadoop-yun-namenode-mini02.out
mini04: starting namenode, logging to /app/hadoop-2.7./logs/hadoop-yun-namenode-mini04.out
mini03: starting namenode, logging to /app/hadoop-2.7./logs/hadoop-yun-namenode-mini03.out
mini06: starting datanode, logging to /app/hadoop-2.7./logs/hadoop-yun-datanode-mini06.out
mini07: starting datanode, logging to /app/hadoop-2.7./logs/hadoop-yun-datanode-mini07.out
mini05: starting datanode, logging to /app/hadoop-2.7./logs/hadoop-yun-datanode-mini05.out
Starting journal nodes [mini05 mini06 mini07]
mini05: journalnode running as process . Stop it first.
mini07: journalnode running as process . Stop it first.
mini06: journalnode running as process . Stop it first.
Starting ZK Failover Controllers on NN hosts [mini01 mini02 mini03 mini04]
mini04: starting zkfc, logging to /app/hadoop-2.7./logs/hadoop-yun-zkfc-mini04.out
mini01: starting zkfc, logging to /app/hadoop-2.7./logs/hadoop-yun-zkfc-mini01.out
mini02: starting zkfc, logging to /app/hadoop-2.7./logs/hadoop-yun-zkfc-mini02.out
mini03: starting zkfc, logging to /app/hadoop-2.7./logs/hadoop-yun-zkfc-mini03.out
4.6. 启动YARN
#####注意#####：是在mini03上执行start-yarn.sh，把namenode和resourcemanager分开是因为性能问题
# 因为他们都要占用大量资源，所以把他们分开了，他们分开了就要分别在不同的机器上启动
[yun@mini03 ~]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /app/hadoop-2.7./logs/yarn-yun-resourcemanager-mini03.out
mini07: starting nodemanager, logging to /app/hadoop-2.7./logs/yarn-yun-nodemanager-mini07.out
mini06: starting nodemanager, logging to /app/hadoop-2.7./logs/yarn-yun-nodemanager-mini06.out
mini05: starting nodemanager, logging to /app/hadoop-2.7./logs/yarn-yun-nodemanager-mini05.out ################################
# 在mini04启动 resourcemanager
[yun@mini04 ~]$ yarn-daemon.sh start resourcemanager # 也可用start-yarn.sh
starting resourcemanager, logging to /app/hadoop-2.7./logs/yarn-yun-resourcemanager-mini04.out
4.7. 启动说明
# 第一次启动的时候请严格按照上面的步骤【第一次涉及格式化问题】
# 第二次以及之后，步骤为： 启动zookeeper、HDFS、YARN
5. 浏览器访问
5.1. bi访问
http://mini01:50070
http://mini02:50070
5.2. dt访问
http://mini03:50070
http://mini04:50070
6. 简单操作
6.1. HDFS创建目录与上传
[yun@mini02 software]$ pwd
/app/software
[yun@mini02 software]$ ll
total
-rw-r--r-- yun yun Jun : CentOS-.4_hadoop-2.7..tar.gz
-rw-r--r-- yun yun May : eclipse-jee-oxygen-3a-win32-x86_64.zip
-rw-r--r-- yun yun Oct jdk1..0_112.tar.gz
-rw-rw-r-- yun yun Jul : test
[yun@mini02 software]$ hadoop fs -ls / # 通过命令行可见，是进行了逻辑分区的
Found items
-r-xr-xr-x - yun yun -- : /bi
-r-xr-xr-x - yun yun -- : /dt
[yun@mini02 software]$ hadoop fs -mkdir /bi/software # 在bi创建目录
[yun@mini02 software]$
[yun@mini02 software]$ hadoop fs -mkdir /dt/software # 在dt创建目录
[yun@mini02 software]$
[yun@mini02 software]$ hadoop fs -put jdk1..0_112.tar.gz /bi/software # 上传文件到bi
[yun@mini02 software]$ hadoop fs -put eclipse-jee-oxygen-3a-win32-x86_64.zip /dt/software # 上传文件到dt
[yun@mini02 software]$
[yun@mini02 software]$
[yun@mini02 software]$ hadoop fs -ls /bi/software
Found items
-rw-r--r-- yun supergroup -- : /bi/software/jdk1..0_112.tar.gz
[yun@mini02 software]$
[yun@mini02 software]$
[yun@mini02 software]$ hadoop fs -ls /dt/software
Found items
-rw-r--r-- yun supergroup -- : /dt/software/eclipse-jee-oxygen-3a-win32-x86_64.zip
6.2. Datanode目录结构
[yun@mini05 current]$ pwd
/app/hadoop/tmp/dfs/data/current
[yun@mini05 current]$ ll
total
drwx------ yun yun Jul : BP--10.0.0.111-
drwx------ yun yun Jul : BP--10.0.0.113-
-rw-rw-r-- yun yun Jul : VERSION
[yun@mini05 current]$ tree
.
├── BP--10.0.0.111-
│ ├── current
│ │ ├── finalized
│ │ │ └── subdir0
│ │ │ └── subdir0
│ │ │ ├── blk_1073741825
│ │ │ ├── blk_1073741825_1001.meta
│ │ │ ├── blk_1073741826
│ │ │ └── blk_1073741826_1002.meta
│ │ ├── rbw
│ │ └── VERSION
│ ├── scanner.cursor
│ └── tmp
├── BP--10.0.0.113-
│ ├── current
│ │ ├── finalized
│ │ │ └── subdir0
│ │ │ └── subdir0
│ │ │ ├── blk_1073741825
│ │ │ ├── blk_1073741825_1001.meta
│ │ │ ├── blk_1073741826
│ │ │ ├── blk_1073741826_1002.meta
│ │ │ ├── blk_1073741827
│ │ │ └── blk_1073741827_1003.meta
│ │ ├── rbw
│ │ └── VERSION
│ ├── scanner.cursor
│ └── tmp
└── VERSION directories, files