Hadoop 2.2 YARN分布式集群搭建配置流程

搭建环境准备:JDK1.6,SSH免密码通信

系统:CentOS 6.3

集群配置:NameNode和ResourceManager在一台服务器上,三个数据节点

搭建用户:YARN

Hadoop2.2下载地址:http://www.apache.org/dyn/closer.cgi/hadoop/common/

步骤一:上传Hadoop 2.2 并解压到/export/yarn/hadoop-2.2.0

  • 外层的启动脚本在sbin目录
  • 内层的被调用脚本在bin目录
  • Native的so文件都在lib/native目录
  • 配置程序文件都放置在libexec
  • 配置文件都在etc目录,对应以前版本的conf目录
  • 所有的jar包都在share/hadoop目录下面

步骤二:配置环境变量

  在~/.bashrc文件中添加以下配置:

export JAVA_HOME=/export/servers/jdk1..0_25/
export HADOOP_DEV_HOME=/export/yarn/hadoop-2.2.
export HADOOP_MAPARED_HOME=${HADOOP_DEV_HOME}
export HADOOP_COMMON_HOME=${HADOOP_DEV_HOME}
export HADOOP_HDFS_HOME=${HADOOP_DEV_HOME}
export YARN_HOME=${HADOOP_DEV_HOME}
export HADOOP_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop
export HDFS_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop
export YARN_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop
export PATH=$PATH:$HADOOP_DEV_HOME/bin:$JAVA_HOME/bin:$HADOOP_DEV_HOME/sbin

  配置完成后,执行source ~/.bashrc命令

步骤三:core-site.xml hdfs-site.xml mapred-site.xml yarn-site.xml配置

  •   Core-site.xml配置
<configuration >
<property>
<name>fs.defaultFS</name>
<value>hdfs://master1:9101</value>
<description></description>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/export/yarn/hadoop-log/</value>
<description>tmp临时目录</description>
</property>
<property>
<name>io.compression.codecs</name>
<value>com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value>
<description>压缩编码,这里配置了LZO</description>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
<description>LZO对应类</description>
</property>
<property>
<name>io.native.lib.available</name>
<value>true</value>
<description>是否启用本地native库</description>
</property>
</configuration>
  • Hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/export/yarn/hadoop-log/nd</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/export/yarn/hadoop-log/dd</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>0.0.0.0:60176</value>
<description>namenode http 地址</description>
</property>
<property>
<name>dfs.datanode.address</name>
<value>0.0.0.0:60116</value>
</property>
<property>
<name>dfs.datanode.ipc.address</name>
<value>0.0.0.0:60126</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>0.0.0.0:60176</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>0.0.0.0:60196</value>
</property>
</configuration>
  • Mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
  • yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.address</name>
<value>master1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master1:8088</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master1:8031</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
<property>
<name>yarn.scheduler.fair.allocation.file</name>
<value>fair-scheduler.xml</value>
</property>
</configuration>

  注:这里配置了Hadoop 2.2 FairScheduler调度器

步骤四:slaves配置

将三个数据节点配置到slaves中

步骤五:将配置好的Hadoop 2.2 分发同步到各个数据节点

步骤六:格式化NameNode

执行命令:hdfs namenode –format

或者 hadoop namenode –format

步骤七:启动hdfs和yarn

启动Hdfs: start-dfs.sh

启动yarn: start-yarn.sh

或者可以执行start-all.sh一起启动hdfs和yarn

步骤八:测试

Hdfs测试:

向hdfs中上传文件:hdfs dfs –put abc /input

查看hdfs文件目录:hdfs dfs –ls /

Yarn测试:

运行WordCount测试程序:

     hadoop jar /export/yarn/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /input /out

欢迎加入Hadoop技术群进行交流:147681830

Hadoop 2.2 YARN分布式集群搭建配置流程

上一篇:安装openssh


下一篇:HDU - 1255 覆盖的面积(线段树求矩形面积交 扫描线+离散化)