centos7配置hadoop集群

2022-09-18 18:06:07

一：测试环境搭建规划：

主机名称	IP	用户	HDFS	YARN
hadoop11	192.168.1.101	hadoop	NameNode,DataNode	NodeManager
hadoop12	192.168.1.102	hadoop	DataNode	NodeManager
hadoop13	192.168.1.103	hadoop	DataNode,SecondaryNameNode	NodeManager
hadoop14	192.168.1.104	hadoop	DataNode	ResourceManager,NodeManager

二：hadoop安装：

1:新建hadoop用户：

useradd hadoop

passwd hadoop

su hadoop

mkdir /home/hadoop/apps

mkdir /home/hadoop/data

三：关闭防火墙：

systemctl stop firewalld

systemctl disable firewalld

四：关闭selinux：

vim /etc/sysconfig/selinux

修改SELINUX=enforcing为SELINUX=disabled

五：安装Java：

5-1下载JDK：

https://www.oracle.com/technetwork/java/javase/downloads/index.html

https://www.oracle.com/technetwork/java/javase/downloads/jdk11-downloads-5066655.html

点选 Accept License Agreement

下载jdk-11.0.1_linux-x64_bin.tar.gz

把下载好的jdk-11.0.1_linux-x64_bin.tar.gz，上传到服务器 /usr/local目录

tar -zxvf jdk-11.0.1_linux-x64_bin.tar.gz

5-2添加环境变量：

vim /etc/profile

在结尾添加：

#java
export JAVA_HOME='/usr/local/jdk-11.0.1'
export PATH=$JAVA_HOME/bin:$PATH

保存退出后，执行：

source /etc/profile

查看版本，确定环境变量是否生效：

java -version

六：安装hadoop

6-1下载hadoop：

https://hadoop.apache.org/releases.html

选择binary文件

拷贝镜像地址，下载

wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.8.5/hadoop-2.8.5.tar.gz

6-2 解压到指定目录：

tar -zxvf hadoop-2.8.5.tar.gz -C /home/hadoop/apps

6-3修改配置文件：

cd /home/hadoop/apps/hadoop-2.8.5/etc/hadoop

配置Hadoop JDK路径修改hadoop-env.sh、mapred-env.sh、yarn-env.sh文件中的JDK路径：

#export JAVA_HOME='/usr/local/jdk-11.0.1'

export JAVA_HOME='/usr/local/jdk1.8.0_191'

6-3-1 vim core-site.xml

<!-- Put site-specific property overrides in this file. -->

<configuration>

 <property>

   <name>fs.defaultFS</name>

   <value>hdfs://hadoop11.zm.com:8020</value>

 </property>

 <property>

   <name>hadoop.tmp.dir</name>

   <value>/home/hadoop/data/hadoop/tmp</value>

 </property>

</configuration>

fs.fefaultFS是NameNode的地址，

hadoop.tmp.dir是hadoop临时目录的地址，

应保证目录是已存在的，如不存在，需先创建。

6-3-2 vim hdfs-site.xml

<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/data/hadoop/name</value>
<description>为了保证元数据的安全一般配置多个不同目录</description>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/data/hadoop/data</value>
<description>datanode 的数据存储目录</description>
</property>

<property>
<name>dfs.replication</name>
<value>2</value>
<description>HDFS 的数据块的副本存储个数, 默认是3</description>
</property>

<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop13.zm.com:50090</value>
</property>

</configuration>

dfs.namenode.name.dir：namenode数据的存放地点。也就是namenode元数据存放的地方，记录了hdfs系统中文件的元数据。

dfs.datanode.data.dir： datanode数据的存放地点。也就是block块存放的目录了。

dfs.replication：hdfs的副本数设置。也就是上传一个文件，其分割为block块后，每个block的冗余副本个数，默认配置是3。

dfs.secondary.http.address：secondarynamenode 运行节点的信息，和 namenode 不同节点

dfs.namenode.secondary.http-address是指定secondaryNameNode的http访问地址和端口号，因为在规划中，我们将hadoop13规划为SecondaryNameNode服务器。

所以这里设置为：hadoop13.zm.com:50090

6-3-3 配置slave文件,slaves文件是指定HDFS上有哪些DataNode节点：

cd /home/hadoop/apps/hadoop-2.8.5

[hadoop@hadoop11 hadoop-2.8.5]$ vim slaves

hadoop11.zm.com

hadoop12.zm.com

hadoop13.zm.com

hadoop14.zm.com

6-3-4 cd /home/hadoop/apps/hadoop-2.8.5/etc/hadoop

vim yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->

<property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

    <property>

        <name>yarn.resourcemanager.hostname</name>

        <value>hadoop14.zm.com</value>

    </property>

    <property>

        <name>yarn.log-aggregation-enable</name>

        <value>true</value>

    </property>

    <property>

        <name>yarn.log-aggregation.retain-seconds</name>

        <value></value>

    </property>

</configuration>

yarn.resourcemanager.hostname这个指定resourcemanager服务器指向hadoop14.zm.com，

yarn.log-aggregation-enable是配置是否启用日志聚集功能，

yarn.log-aggregation.retain-seconds是配置聚集的日志在HDFS上最多保存多长时间。

6-3-5 cp mapred-site.xml.template mapred-site.xml

vim mapred-site.xml

<!-- Put site-specific property overrides in this file. -->

<configuration>

    <property>

        <name>mapreduce.framework.name</name>

        <value>yarn</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.address</name>

        <value>hadoop11.zm.com:</value>

    </property>

    <property>

        <name>mapreduce.jobhistory.webapp.address</name>

        <value>hadoop11.zm.com:</value>

    </property>

</configuration>

mapreduce.framework.name设置mapreduce任务运行在yarn上。

mapreduce.jobhistory.address是设置mapreduce的历史服务器安装在hadoop11机器上。

mapreduce.jobhistory.webapp.address是设置历史服务器的web页面地址和端口号。

七：配置hosts文件：

回到root权限：su

vim /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.1.201     hadoop11     hadoop11.zm.com

192.168.1.202     hadoop12     hadoop12.zm.com

192.168.1.203     hadoop13     hadoop13.zm.com 

192.168.1.204     hadoop14     hadoop14.zm.com

八：SSH免密登陆

su hadoop

所有hadoop集群服务器：

su hadoop

cd /home/hadoop/.ssh

ssh-keygen -t rsa

将所有终端公钥文件拷贝到hadoop11：

　　所有终端运行：

　　ssh-copy-id -i hadoop11

登陆hadoop11终端的/home/hadoop/.ssh修改权限：

　　chmod 600 authorized_keys

分发授权文件到其他终端：

　　scp /home/hadoop/.ssh/authorized_keys hadoop12:/home/hadoop/.ssh/

scp /home/hadoop/.ssh/authorized_keys hadoop13:/home/hadoop/.ssh/

scp /home/hadoop/.ssh/authorized_keys hadoop14:/home/hadoop/.ssh/

每台机器用ssh命令连接下所有机器，第一次连接要输入yes，以后不用输入

九：hadoop分发到其他终端

拷贝hadoop11 /home/hadoop/apps目录内容到其他终端:

scp -rq data hadoop12:/home/hadoop/

拷贝hadoop11 /home/hadoop/data目录内容到其他终端:

scp -rq data hadoop12:/home/hadoop/

十：配置hadoop环境变量（所有节点均需配置）

注意：

1、如果使用root用户进行安装。 vi /etc/profile 即可系统变量

2、如果使用普通用户进行安装。 vi ~/.bashrc 用户变量

vim .bashrc

# User specific aliases and functions

export HADOOP_HOME=/home/hadoop/apps/hadoop-2.8.

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

[hadoop@hadoop11 ~]$ source .bashrc

查看hadoop版本：hadoop version

[hadoop@hadoop11 ~]$ hadoop version

Hadoop 2.8.

Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 0b8464d75227fcee2c6e7f2410377b3d53d3d5f8

Compiled by jdu on --10T03:32Z

Compiled with protoc 2.5.

From source with checksum 9942ca5c745417c14e318835f420733

This command was run using /home/hadoop/apps/hadoop-2.8./share/hadoop/common/hadoop-common-2.8..jar

十一：hadoop初始化

HDFS初始化只能在主节点上进行

如果需要重新初始化，先删除相关目录文件，

粗略做法是删除/home/hadoop/data/hadoop路径下

[hadoop@hadoop14 hadoop]$ ll

总用量

drwx------.  hadoop hadoop  10月  : data

drwxrwxr-x.  hadoop hadoop  10月  : name

drwxrwxr-x.  hadoop hadoop  10月  : tmp

[hadoop@hadoop14 hadoop]$ pwd

/home/hadoop/data/hadoop

[hadoop@hadoop11 ~]$ hadoop namenode -format

// :: INFO namenode.FSImage: Allocated new BlockPoolId: BP--198.8.8.201-

// :: INFO common.Storage: Storage directory /home/hadoop/data/hadoop/name has been successfully formatted.

// :: INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/data/hadoop/name/current/fsimage.ckpt_0000000000000000000 using no compression

// :: INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/data/hadoop/name/current/fsimage.ckpt_0000000000000000000 of size  bytes saved in  seconds.

// :: INFO namenode.NNStorageRetentionManager: Going to retain  images with txid >=

// :: INFO util.ExitUtil: Exiting with status

// :: INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at hadoop11/198.8.8.201

************************************************************/

初始化后会自动在/home/hadoop/data/hadoop/name路径

建立current目录

生成：

-rw-rw-r--  hadoop hadoop  10月  : fsimage_0000000000000000000

-rw-rw-r--  hadoop hadoop   10月  : fsimage_0000000000000000000.md5

-rw-rw-r--  hadoop hadoop    10月  : seen_txid

-rw-rw-r--  hadoop hadoop  10月  : VERSION

十二：启动HDFS：

可以在集群中任何节点执行：

[hadoop@hadoop12 ~]$ start-dfs.sh

报错

[hadoop@hadoop12 ~]$ start-dfs.sh

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/home/hadoop/apps/hadoop-2.8./share/hadoop/common/lib/hadoop-auth-2.8..jar) to method sun.security.krb5.Config.getInstance()

WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

Starting namenodes on [hadoop11.zm.com]

The authenticity of host 'hadoop11.zm.com (198.8.8.201)' can't be established.

ECDSA key fingerprint is SHA256:RlH2tSlCWblQ5jZ6DMuCt4+yEmYuWo5MZuqUTGesM9I.

ECDSA key fingerprint is MD5:de:e9:::c7:::0c:e6:8a::0e::5f:2b:8d.

Are you sure you want to continue connecting (yes/no)? no

关闭HDFS:

stop-dfs.sh

降低java版本到jdk1.8.0_191

如步骤6-3修改集群所有节点配置文件：

cd /home/hadoop/apps/hadoop-2.8.5/etc/hadoop

配置Hadoop JDK路径修改hadoop-env.sh、mapred-env.sh、yarn-env.sh文件中的JDK路径：

#export JAVA_HOME='/usr/local/jdk-11.0.1'

export JAVA_HOME='/usr/local/jdk1.8.0_191'

再次启动HDFS（任何节点均可执行）:

start-dfs.sh

[hadoop@hadoop11 sbin]$ start-dfs.sh

Starting namenodes on [hadoop11.zm.com]

hadoop11.zm.com: starting namenode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-namenode-hadoop11.out

hadoop11: starting datanode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-datanode-hadoop11.out

hadoop13: starting datanode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-datanode-hadoop13.out

hadoop14: starting datanode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-datanode-hadoop14.out

hadoop12: starting datanode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-datanode-hadoop12.out

Starting secondary namenodes [hadoop13.zm.com]

hadoop13.zm.com: starting secondarynamenode, logging to /home/hadoop/apps/hadoop-2.8./logs/hadoop-hadoop-secondarynamenode-hadoop13.out

在其他节点启动时会报询问信息

The authenticity of host 'hadoop13.zm.com (198.8.8.203)' can't be established.

ECDSA key fingerprint is SHA256:RlH2tSlCWblQ5jZ6DMuCt4+yEmYuWo5MZuqUTGesM9I.

ECDSA key fingerprint is MD5:de:e9:::c7:::0c:e6:8a::0e::5f:2b:8d.

Are you sure you want to continue connecting (yes/no)?

解决办法（所有节点执行）：

进入root用户

vim /etc/ssh/ssh_config

加入如下两句

#   StrictHostKeyChecking ask

StrictHostKeyChecking no

UserKnownHostsFile /dev/null

start-dfs.sh执行后：

在/home/hadoop/data/hadoop/data中自动生成

[hadoop@hadoop11 data]$ ll

总用量

drwxrwxr-x  hadoop hadoop  10月  : current

-rw-rw-r--  hadoop hadoop  10月  : in_use.lock

在/home/hadoop/data/hadoop/name中自动生成

[hadoop@hadoop11 name]$ ll

总用量

drwxrwxr-x  hadoop hadoop  10月  : current

-rw-rw-r--  hadoop hadoop   10月  : in_use.lock

十三：启动YARN（只能在ResourceManager主节点执行）：

[hadoop@hadoop14 ~]$ start-yarn.sh

[hadoop@hadoop14 ~]$ start-yarn.sh

starting yarn daemons

starting resourcemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-resourcemanager-hadoop11.out

hadoop12: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop12.out

hadoop13: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop13.out

hadoop14: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop14.out

hadoop11: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop11.out

start-yarn.sh执行后：

会在/home/hadoop/data/hadoop/tmp中自动生成

[hadoop@hadoop14 tmp]$ ll

总用量

drwxr-xr-x  hadoop hadoop  10月  : nm-local-dir

[hadoop@hadoop14 nm-local-dir]$ ll

总用量

drwxr-xr-x  hadoop hadoop  10月  : filecache

drwx------  hadoop hadoop  10月  : nmPrivate

drwxr-xr-x  hadoop hadoop  10月  : usercache

此配置启动后，ResourceManager没有启动，无法访问：http://198.8.8.204:8088/

需要在所有节点添加ResourceManager配置：

vim vim yarn-site.xml

  See the License for the specific language governing permissions and

  limitations under the License. See accompanying LICENSE file.

-->

<configuration>

<!-- Site specific YARN configuration properties -->

<property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

    <property>

        <name>yarn.resourcemanager.hostname</name>

        <value>hadoop14</value>

    </property>

    <property>

        <name>yarn.resourcemanager.address</name>

        <value>${yarn.resourcemanager.hostname}:</value>

#ResourceManager 对客户端暴露的地址。客户端通过该地址向RM提交应用程序，杀死应用程序等。

    </property>

    <property>

        <name>yarn.resourcemanager.scheduler.address</name>

        <value>${yarn.resourcemanager.hostname}:</value>

#ResourceManager 对ApplicationMaster暴露的访问地址。ApplicationMaster通过该地址向RM申请资源、释放资源等。

    </property>

    <property>

        <name>yarn.resourcemanager.resource-tracker.address</name>

        <value>${yarn.resourcemanager.hostname}:</value>

#ResourceManager 对NodeManager暴露的地址。NodeManager通过该地址向RM汇报心跳，领取任务等。

    </property>

    <property>

        <name>yarn.resourcemanager.admin.address</name>

        <value>${yarn.resourcemanager.hostname}:</value>

#ResourceManager 对管理员暴露的访问地址。管理员通过该地址向RM发送管理命令等。

    </property>

    <property>

        <name>yarn.resourcemanager.webapp.address</name>

#        <value>198.8.8.204:</value>

        <value>${yarn.resourcemanager.hostname}:</value>

    </property>

    <property>

        <name>yarn.log-aggregation-enable</name>

        <value>true</value>

    </property>

    <property>

        <name>yarn.log-aggregation.retain-seconds</name>

        <value></value>

    </property>

</configuration>

保存配置，

在ResourceManager主节点hadoop14重启yarn：

stop-yarn.sh

start-yarn.sh

[hadoop@hadoop14 hadoop]$ start-yarn.sh

starting yarn daemons

starting resourcemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-resourcemanager-hadoop14.out

hadoop11: Warning: Permanently added 'hadoop11,198.8.8.201' (ECDSA) to the list of known hosts.

hadoop12: Warning: Permanently added 'hadoop12,198.8.8.202' (ECDSA) to the list of known hosts.

hadoop14: Warning: Permanently added 'hadoop14,198.8.8.204' (ECDSA) to the list of known hosts.

hadoop13: Warning: Permanently added 'hadoop13,198.8.8.203' (ECDSA) to the list of known hosts.

hadoop14: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop14.out

hadoop11: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop11.out

hadoop13: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop13.out

hadoop12: starting nodemanager, logging to /home/hadoop/apps/hadoop-2.8./logs/yarn-hadoop-nodemanager-hadoop12.out

[hadoop@hadoop14 hadoop]$ jps

 ResourceManager

 DataNode

 NodeManager

 Jps

可以访问：http://198.8.8.204:8088/

十四：查看所有节点进程

jps

[hadoop@hadoop11 nm-local-dir]$ jps

 NodeManager

 DataNode

 Jps

 NameNode

[hadoop@hadoop12 nm-local-dir]$ jps

 NodeManager

 DataNode

 Jps

[hadoop@hadoop13 nm-local-dir]$ jps

 NodeManager

 DataNode

 Jps

 SecondaryNameNode

[hadoop@hadoop14 nm-local-dir]$ jps

 DataNode

 NodeManager

 Jps

十五：HDFS和YARN的WEB管理界面：

http://198.8.8.201:50070

十六：hadoop简单应用：

创建目录

[hadoop@hadoop11 ~]$ hadoop fs -mkdir -p /test/input

查看创建文件

hadoop fs -ls /

新建文件

vim words.txt

hello zhangsan

hello lisi

hello wangwu

上传文件

hadoop fs -put ~/words.txt /test/input

查看是否上传成功

hadoop fs -ls /test/input

下载文件

hadoop fs -get /test/input/words.txt /home/hadoop/

查看是否下载成功

cd /home/hadoop

用wordcount测试：

[hadoop@hadoop11 hadoop]$ hadoop jar /home/hadoop/apps/hadoop-2.8.5/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.5.jar wordcount /test/input /test/output

// :: INFO client.RMProxy: Connecting to ResourceManager at hadoop14/198.8.8.204:

// :: INFO input.FileInputFormat: Total input files to process :

// :: INFO mapreduce.JobSubmitter: number of splits:

// :: INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1541142769004_0001

// :: INFO impl.YarnClientImpl: Submitted application application_1541142769004_0001

// :: INFO mapreduce.Job: The url to track the job: http://hadoop14:8088/proxy/application_1541142769004_0001/

// :: INFO mapreduce.Job: Running job: job_1541142769004_0001

// :: INFO mapreduce.Job: Job job_1541142769004_0001 running in uber mode : false

// :: INFO mapreduce.Job:  map % reduce %

// :: INFO mapreduce.Job:  map % reduce %

// :: INFO mapreduce.Job:  map % reduce %

// :: INFO mapreduce.Job: Job job_1541142769004_0001 completed successfully

// :: INFO mapreduce.Job: Counters:

    File System Counters

        FILE: Number of bytes read=

        FILE: Number of bytes written=

        FILE: Number of read operations=

        FILE: Number of large read operations=

        FILE: Number of write operations=

        HDFS: Number of bytes read=

        HDFS: Number of bytes written=

        HDFS: Number of read operations=

        HDFS: Number of large read operations=

        HDFS: Number of write operations=

    Job Counters

        Launched map tasks=

        Launched reduce tasks=

        Data-local map tasks=

        Total time spent by all maps in occupied slots (ms)=

        Total time spent by all reduces in occupied slots (ms)=

        Total time spent by all map tasks (ms)=

        Total time spent by all reduce tasks (ms)=

        Total vcore-milliseconds taken by all map tasks=

        Total vcore-milliseconds taken by all reduce tasks=

        Total megabyte-milliseconds taken by all map tasks=

        Total megabyte-milliseconds taken by all reduce tasks=

    Map-Reduce Framework

        Map input records=

        Map output records=

        Map output bytes=

        Map output materialized bytes=

        Input split bytes=

        Combine input records=

        Combine output records=

        Reduce input groups=

        Reduce shuffle bytes=

        Reduce input records=

        Reduce output records=

        Spilled Records=

        Shuffled Maps =

        Failed Shuffles=

        Merged Map outputs=

        GC time elapsed (ms)=

        CPU time spent (ms)=

        Physical memory (bytes) snapshot=

        Virtual memory (bytes) snapshot=

        Total committed heap usage (bytes)=

    Shuffle Errors

        BAD_ID=

        CONNECTION=

        IO_ERROR=

        WRONG_LENGTH=

        WRONG_MAP=

        WRONG_REDUCE=

    File Input Format Counters

        Bytes Read=

    File Output Format Counters

        Bytes Written=

查看输出文件：

[hadoop@hadoop11 hadoop]$ hadoop fs -ls /test/output

[hadoop@hadoop11 hadoop]$ hadoop fs -ls /test/output

Found  items

-rw-r--r--    hadoop supergroup           -- : /test/output/_SUCCESS

-rw-r--r--    hadoop supergroup          -- : /test/output/part-r-

[hadoop@hadoop11 hadoop]$ hadoop fs -cat /test/output/part-r-

hello

lisi

wangwu

zhangsan

参考文档：

https://www.cnblogs.com/qingyunzong/p/8496127.html

https://blog.csdn.net/hliq5399/article/details/78193113

码农公寓

相关文章