Hadoop 系列文章(三) 配置部署启动YARN及在YARN上运行MapReduce程序

2023-01-02 08:48:56

　　这篇文章里我们将用配置 YARN，在 YARN 上运行 MapReduce。

　　1、修改 yarn-env.sh 环境变量里的 JAVA_HOME 路径

　　[bamboo@hadoop-senior hadoop-2.5.0]$ vim etc/hadoop/yarn-env.sh

　　export JAVA_HOME=/opt/modules/jdk1.7.0_67

　　2、修改 etc/hadoop/yarn-site.xml，添加如下的两个属性

<configuration>

    <property>

        <name>yarn.resourcemanager.hostname</name>

        <value>hadoop-senior.bamboo.com</value>

    </property>

    <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

</configuration>

　　修改 slave 的 hostname(即把 localhost 改为 hostname)

　　vim /etc/hadoop/slaves

　　把 localhost 换为 hostname

　　hadoop-senior.bamboo.com

　　3、启动 yarn

[bamboo@hadoop-senior hadoop-2.5.0]$ sbin/yarn-daemon.sh start resourcemanager

starting resourcemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-bamboo-resourcemanager-hadoop-senior.bamboo.com.out

[bamboo@hadoop-senior hadoop-2.5.0]$ sbin/yarn-daemon.sh start nodemanager

starting nodemanager, logging to /opt/modules/hadoop-2.5.0/logs/yarn-bamboo-nodemanager-hadoop-senior.bamboo.com.out

[bamboo@hadoop-senior hadoop-2.5.0]$ jps

10131 NameNode

11931 NodeManager

12033 Jps

10467 DataNode

11679 ResourceManager

　　4、修改 mapred-env.sh 和 mapred-site.xml

　　mapred-env.sh (修改 JAVA_HOME 变量地址)

　　mapred-site.xml 文件，添加如下的属性

<configuration>

    <property>

        <name>mapreduce.framework.name</name>

        <value>yarn</value>

    </property>

</configuration>

　　5、在 yarn 上运行 mapreduce

运行前需要删除 output 目录，否则运行的时候会报错

[bamboo@hadoop-senior hadoop-2.5.0]$ bin/hdfs dfs -rm -R /user/bamboo/mapreduce/wordcount/output

17/12/31 19:42:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

17/12/31 19:42:25 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes.

Deleted /user/bamboo/mapreduce/wordcount/output

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar wordcount /user/bamboo/mapreduce/wordcount/input /user/bamboo/mapreduce/wordcount/output

[bamboo@hadoop-senior hadoop-2.5.0]$ bin/hdfs dfs -cat /user/bamboo/mapreduce/wordcount/output/*

17/12/31 19:46:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

hadoop 4

hdfs 1

mapreduce 1

nodemanager 1

resourcemanager 1

yarn 2

Ok,YARN 的配置以及启动就到这里。

码农公寓

相关文章