hadoop ——完全分布式环境搭建

2022-12-03 11:55:05

hadoop 完全分布式环境搭建

1、虚拟机角色分配：

192.168.44.184 hadoop02 NameNode/DataNode ResourceManager/NodeManager

192.168.44.185 hadoop03 DataNode NodeManager

192.168.44.186 hadoop04 DataNode NodeManager

每台机器配置好hostname,hosts：/etc/hostname 中配置好hostname

/etc/hosts 中配置好hosts：

192.168.44.184 hadoop02

192.168.44.185 hadoop03

192.168.44.186 hadoop04

2、sskey免登录设置

1）ssh-keygen -t rsa （会在.ssh目录生成公钥和私钥,每台机器都要生成）

2）以hadoop02为主，将公钥copy到各个机器

ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop02

    ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop03

    ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop04

3、jdk的安装(在hadoop02上配置，另外两台机器稍后复制过去)

1）直接下载jdk包放到安装目录即可（jdk-8u131-linux-x64.tar.gz解压到安装目录 tar zxvf ./jdk-8u131-linux-x64.tar.gz -C /home/hadoop/app/）

2）配置环境变量直接在 .bash_profile配置

export JAVA_HOME=/home/hadoop/app/jdk1.8.0_131

    export PATH=$JAVA_HOME/bin:$PATH

4.下载安装hadoop-2.6.0-cdh5.7.0(在hadoop02上配置，另外两台机器稍后复制过去)

cdh下载地址：（http://archive.cloudera.com/cdh5/cdh/5/)

1）直接下载hadoop-2.6.0-cdh5.7.0.tar.gz包放到安装目录即可

2）配置环境变量

 export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0

    export PATH=$HADOOP_HOME/bin:$JAVA_HOME/bin:$PATH

5、配置修改hadoop相关配置文件(在hadoop02上配置，另外两台机器稍后复制过去)

1）hadoop_home/etc/hadoop/hadoop-env.sh 配置jdk

export JAVA_HOME=/home/hadoop/app/jdk1.8.0_131

2）etc/hadoop/core-site.xml

<property>

            <name>fs.defaultFS</name>

            <value>hdfs://hadoop02:8020</value>

        </property>

        <property>

            <name>hadoop.tmp.dir</name>

            <value>/home/hadoop/app/tmp</value>

        </property>

3）etc/hadoop/hdfs-site.xml 设置副本系数和name、data的存放路径

<property>

            <name>dfs.replication</name>

            <value>3</value>

        </property>

        <property>

            <name>dfs.namenode.name.dir</name>

            <value>/home/hadoop/app/tmp/dfs/name</value>

        </property>

        <property>

           <name>dfs.datanode.data.dir</name>

           <value>/home/hadoop/app/tmp/dfs/data</value>

        </property>

4）etc/hadoop/yarn-site.xml

<property>

            <name>yarn.nodemanager.aux-services</name>

            <value>mapreduce_shuffle</value>

        </property>

        <property>

            <name>yarn.resourcemanager.hostname</name>

            <value>hadoop02</value>

        </property>

5）etc/hadoop/mapred-site.xml

<property>

            <name>mapreduce.framework.name</name>

            <value>yarn</value>

        </property>

6）/etc/hadoop/slaves(标明从节点)

hadoop02

        hadoop03

        hadoop04

6、分发安装包到 hadoop03、hadoop03节点上

scp -r ~/app hadoop@hadoop03:~/

        scp -r ~/app hadoop@hadoop04:~/

        scp  ~/.bash_profile hadoop@hadoop03:~/

        scp  ~/.bash_profile hadoop@hadoop04:~/

在hadoop03,hadoop04上分别source下.bash_profile文件

7、NameNode格式化：只要在hadoop02上格式化

bin/hdfs namenode -format

8、启动hadoop集群。在NameNode启动

sbin/start-all.sh

启动日志：

[hadoop@hadoop02 hadoop-2.6.0-cdh5.7.0]$ ./sbin/start-all.sh

    This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

    18/08/19 23:20:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

    Starting namenodes on [hadoop02]

    hadoop02: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop02.out

    hadoop02: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop02.out

    hadoop04: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop04.out

    hadoop03: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop03.out

    Starting secondary namenodes [0.0.0.0]

    The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.

    ECDSA key fingerprint is 78:05:94:d6:0b:e2:78:7d:93:3c:90:d0:ab:7c:ed:e2.

    Are you sure you want to continue connecting (yes/no)? yes

    0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.

    0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop02.out

    18/08/19 23:21:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

    starting yarn daemons

    starting resourcemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-hadoop02.out

    hadoop03: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop03.out

    hadoop04: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop04.out

    hadoop02: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop02.out

9、验证

1）jps验证：

[hadoop@hadoop02 hadoop-2.6.0-cdh5.7.0]$ jps

    3862 SecondaryNameNode

    4086 NodeManager

    3591 NameNode

    3991 ResourceManager

    3678 DataNode

    4399 Jps

[hadoop@hadoop03 ~]$ jps

    3460 NodeManager

    3576 Jps

    3372 DataNode

[hadoop@hadoop04 ~]$ jps

    3284 NodeManager

    3400 Jps

    3196 DataNode

2）web页面验证：http://hadoop02:50070 （hdfs）

http://hadoop02:8088 （yarn）

10、停止

sbin/stop-all.sh

码农公寓

相关文章