hadoop 2.7.3伪分布式安装

2023-07-25 13:22:10

hadoop 2.7.3伪分布式安装

hadoop集群的伪分布式部署由于只需要一台服务器，在测试，开发过程中还是很方便实用的，有必要将搭建伪分布式的过程记录下来，好记性不如烂笔头。

hadoop 2.7.3

JDK 1.8.91

到Apache的官网下载hadoop的二进制安装包。

cd /home/fuxin.zhao/soft

tar -czvf hadoop 2.7.3.tar.gz

cd hadoop-2.7.3

cd etc/hadoop/

pwd

1. 建立本机到本机的免密登录

ssh-keygen -t rsa -P ""

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

ssh localhost

1. 修改hadoop的配置文件

位于$HADOOP_HOME/conf目录下的修改四个配置文件：slaves、core-site.xml

hdfs-site.xml 、mapred-site.xml 、 yarn-site.xml

vi etc/hadoop/yarn-env.sh

export JAVA_HOME=/usr/local/jdk

vi etc/hadoop/hadoop-env.sh

export JAVA_HOME=/usr/local/jdk

vi slaves

##加入本机的hostname

fuxin.zhao@ubuntuServer01:~/soft/hadoop-2.7.3/etc/hadoop$ vi slaves

ubuntuServer01

vi core-site.xml

<configuration>

 <property>

   <name>fs.defaultFS</name>

   <value>hdfs://ubuntuServer01:9000</value>

 </property>

 <property>

   <name>hadoop.tmp.dir</name>

   <value>file:/home/fuxin.zhao/hadoop/tmp</value>

   <description>Abase for other temporary directories.</description>

 </property>

</configuration>

vi hdfs-site.xml：

<configuration>

    <property>

         <name>dfs.replication</name>

         <value>1</value>

    </property>

    <property>

         <name>dfs.namenode.name.dir</name>

         <value>file:/home/fuxin.zhao/hadoop/tmp/dfs/name</value>

    </property>

    <property>

         <name>dfs.datanode.data.dir</name>

         <value>file:/home/fuxin.zhao/hadoop/tmp/dfs/data</value>

    </property>

   <property>

    <name>dfs.block.size</name>

    <value>67108864</value>

   </property>

</configuration>

vi yarn-site.xml

<configuration>

<property>

  <name>yarn.nodemanager.aux-services</name>

  <value>mapreduce_shuffle</value>

</property>

<property>

  <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>

  <value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property>

  <name>yarn.scheduler.minimum-allocation-mb</name>

  <value>512</value>

</property>

<property>

  <name>yarn.scheduler.maximum-allocation-mb</name>

  <value>2048</value>

</property>

<property>

  <name>yarn.scheduler.minimum-allocation-vcores</name>

  <value>1</value>

</property>

<property>

  <name>yarn.scheduler.maximum-allocation-vcores</name>

  <value>2</value>

</property>

</configuration>

vi mapred-site.xml

<configuration>

<property>

	<name>mapreduce.framework.name</name>

	<value>yarn</value>

</property>

<property>

	<name>yarn.app.mapreduce.am.resource.mb</name>

	<value>512</value>

</property>

<property>

	<name>mapreduce.map.memory.mb</name>

	<value>512</value>

</property>

<property>

	<name>mapreduce.reduce.memory.mb</name>

	<value>512</value>

</property>

</configuration>

vi .bashrc

export JAVA_HOME=/usr/local/jdk

export HADOOP_HOME=/home/fuxin.zhao/soft/hadoop-2.7.3

export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH

配置完成后，执行 NameNode 的格式化:

./bin/hdfs namenode -format

./sbin/start-dfs.sh

./sbin/start-yarn.sh

mr-jobhistory-daemon.sh start historyserver

查看hdfs的web页面：

http://ubuntuserver01:50070/

http://ubuntuserver01:8088/

hadoop fs -ls /

hadoop fs -mkdir /user

hadoop fs -mkdir /user/fuxin.zhao

hadoop fs -touchz textFile

运行官方自带的测试job（teragen and terasort）：

测试job（teragen and terasort）

#在/tmp/terasort/1000000下生成100M数据

hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar teragen 1000000 /tmp/terasort/1000000-input

#排序，输出到/tmp/terasort/1000000-output

hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar terasort /tmp/terasort/1000000-input /tmp/terasort/1000000-output

#删除临时文件

hadoop fs -rm -r /tmp/terasort/1000000-input

hadoop fs -rm -r /tmp/terasort/1000000-output

码农公寓