多个线程运行MR程序时hadoop出现的问题

2022-10-28 16:27:24

夜间多个任务同时并行，总有几个随机性有任务失败，查看日志：刷选关键词 Caused by 或者 FAILED

  cat -n ads_channel.log |grep "Caused by"

      Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot overwrite non empty destination directory /tmp/hadoop-hdfs/mapred/local/

      Caused by: java.io.IOException: Rename cannot overwrite non empty destination directory /tmp/hadoop-hdfs/mapred/local/

In order to change the average load for a reducer (in bytes):

  set hive.exec.reducers.bytes.per.reducer=<number>

In order to limit the maximum number of reducers:

  set hive.exec.reducers.max=<number>

In order to set a constant number of reducers:

  set mapreduce.job.reduces=<number>

java.io.IOException: java.util.concurrent.ExecutionException: org.apache.hadoop.fs.FileAlreadyExistsException: rename destination /tmp/hadoop-hdfs/mapred/local/ already exists.

    at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:)

    at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:)

    at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:)

    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:)

    at org.apache.hadoop.mapreduce.Job$.run(Job.java:)

    at org.apache.hadoop.mapreduce.Job$.run(Job.java:)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:)

    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

    at org.apache.hadoop.mapreduce.Job.submit(Job.java:)

    at org.apache.hadoop.mapred.JobClient$.run(JobClient.java:)

    at org.apache.hadoop.mapred.JobClient$.run(JobClient.java:)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:)

    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:)

    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:)

    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:)

    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:)

    at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:)

    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:)

    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:)

    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:)

    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:)

    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:)

    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:)

    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:)

    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:)

    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:)

    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:)

    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:)

    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:)

    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:)

    at java.lang.reflect.Method.invoke(Method.java:)

    at org.apache.hadoop.util.RunJar.run(RunJar.java:)

    at org.apache.hadoop.util.RunJar.main(RunJar.java:)

Caused by: java.util.concurrent.ExecutionException: org.apache.hadoop.fs.FileAlreadyExistsException: rename destination /tmp/hadoop-hdfs/mapred/local/ already exists.

    at java.util.concurrent.FutureTask.report(FutureTask.java:)

    at java.util.concurrent.FutureTask.get(FutureTask.java:)

    at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:)

    ...  more

Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: rename destination /tmp/hadoop-hdfs/mapred/local/ already exists.

    at org.apache.hadoop.fs.FileSystem.rename(FileSystem.java:)

    at org.apache.hadoop.fs.DelegateToFileSystem.renameInternal(DelegateToFileSystem.java:)

    at org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:)

    at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:)

    at org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:)

    at org.apache.hadoop.fs.FileContext.rename(FileContext.java:)

    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:)

    at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:)

    at java.util.concurrent.FutureTask.run(FutureTask.java:)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)

    at java.lang.Thread.run(Thread.java:)

Job Submission failed with exception 'java.io.IOException(java.util.concurrent.ExecutionException: org.apache.hadoop.fs.FileAlreadyExistsException: rename destination /tmp/hadoop-hdfs/mapred/local/1579374502408 already exists.)'

FAILED: Execution Error, return code  from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

扩展：

cat -n ads_channel.log |grep "Caused by" 或者 grep ads_channel.log  -e "Caused by" 或者 grep -E "Caused by|FAILED"  ads_channel.log #两个关键词

grep "2019-12-21" ads_channel.log | grep "Caused by" ads_channel.log

cat ads_channel.log | grep "Caused by" -B  ##根据关键字查看前20行日志

cat ads_channel.log | grep "Caused by" -A  ##根据关键字查看后20行日志

cat ads_channel.log | grep "Caused by" -C  #根据关键字查看前后10行日志

说明：

-A 表示关键字之后，After

-B 表示关键字之前，Before

-C 表示关键字前后，Context

vim ads_channel.log

:set nu  : (跳转到指定行数)

实时查询多个关键字的日志信息

命令：tail -f ads_channel.log |grep -E “Caused by"

问题原因：

当多个线程运行MR程序时hadoop出现的问题：
https://issues.apache.org/jira/browse/MAPREDUCE-6992
https://issues.apache.org/jira/browse/MAPREDUCE-6441

hdfs会创建一个以当前时间的时间戳命名的文件.当两个mr任务在同一毫秒提交,造成了文件的并发访问问题.

yarn的运行模式:

1-本地模式（LocalJobRunner实现）
mapreduce.framework.name设置为local，则不会使用YARN集群来分配资源，在本地节点执行。在本地模式运行的任务，无法发挥集群的优势。注：在web UI是查看不到本地模式运行的任务。

对 hive有些了解的人都会知道，hive 会将 SQL 语句最终转化成分布式执行的 mapreduce 任务计划。对于大数量集的数据启动 mapreduce 所花费的时间是渺小的。因为数据量大，并且分布再不同的机器上，在不同的机器上处理，这样做是 hive 的优势之一。然而当处理小数量，并且数据都聚集再一台机器上时，那么启动本地模式是非常有意的，不可避免的启动 mapreduce，将数据拉回客户端，本地处理，这样减少了分处理后合并花费的时间。如此一来，对数据量比较小的操作，就可以在本地执行，这样要比提交任务到集群执行效率要快很多。
启动本地模式，需要配置如下参数：

     hive.exec.mode.local.auto                    决定 Hive 是否应该自动地根据输入文件大小，在本地运行。

     hive.exec.mode.local.auto.inputbytes.max     最大输入数据量,当输入数据量小于这个值的时候将会启动本地模式,默认是 128M。

     hive.exec.mode.local.auto.tasks.max          最大输入文件个数,当输入文件个数小于这个值的时候将会启动本地模式。(默认4)

当一个job满足如下条件才能真正使用本地模式：

  .job的输入数据大小必须小于参数：hive.exec.mode.local.auto.inputbytes.max(默认128MB)

  .job的map数必须小于参数：hive.exec.mode.local.auto.tasks.max(默认4)

  .job的reduce数必须为0或者1

2-Yarn模式（YARNRunner实现）
mapreduce.framework.name设置为yarn，当客户端配置mapreduce.framework.name为yarn时, 客户端会使用YARNRunner与服务端通信, 而YARNRunner真正的实现是通过ClientRMProtocol与RM交互, 包括提交Application, 查询状态等功能。但是根据任务的特性，分为两种方式执行任务

3-Uber模式：

为降低小作业延迟而设计的一种模式，所有任务，不管是Map Task，还是Reduce Task，均在同一个Container中顺序执行，这个Container其实也是MRAppMaster所在Container

4-Non-Uber模式：

对于运行时间较长的大作业，先为Map Task申请资源，当Map Task运行完成数目达到一定比例后再为Reduce Task申请资源。

解决办法：

1-在不改源代码的情况下，取消自动启动本地模式,根据集群环境,临时在运行程序时设置:

set hive.exec.mode.local.auto = false

2-在调度系统中设置设置失败重试.

azkaban配置失败重试如下:

type =command

command = xxxxxx

retries=

retry.backoff= #毫秒数

参考：https://blog.csdn.net/weixin_39445556/article/details/80348976

在官网找到了这个bug，在2.7.1版本中已经修复了这个bug，对集群进行升级：

This is a bug in Hadoop 2.6.0. It's been marked as fixed but it still happens occasionally (see: https://issues.apache.org/jira/browse/YARN-2624).

https://*.com/questions/30857413/hadoop-complains-about-attempting-to-overwrite-nonempty-destination-directory

[hdfs@el-hadoop- logs]$ hadoop dfsadmin -report  ##查看hadoop状况：

DEPRECATED: Use of this script to execute hdfs command is deprecated.

Instead use the hdfs command for it.

Configured Capacity:  (1.13 TB)

Present Capacity:  (1.05 TB)

DFS Remaining:  (1.02 TB)

DFS Used:  (27.28 GB)

DFS Used%: 2.54%

Under replicated blocks:

Blocks with corrupt replicas:

Missing blocks:

Missing blocks (with replication factor ): 

-------------------------------------------------

Live datanodes ():

Name: 172.26.0.106: (el-hadoop-)

Hostname: el-hadoop-

Rack: /default

Decommission Status : Normal

Configured Capacity:  (385.73 GB)

DFS Used:  (9.07 GB)

Non DFS Used:  (20.54 GB)

DFS Remaining:  (335.73 GB)

DFS Used%: 2.35%

DFS Remaining%: 87.04%

Configured Cache Capacity:  ( GB)

Cache Used:  ( B)

Cache Remaining:  ( GB)

Cache Used%: 0.00%

Cache Remaining%: 100.00%

Xceivers:

Last contact: Sat Dec  :: CST 

Name: 172.26.0.108: (el-hadoop-)

Hostname: el-hadoop-

Rack: /default

Decommission Status : Normal

Configured Capacity:  (385.73 GB)

DFS Used:  (9.10 GB)

Non DFS Used:  ( B)

DFS Remaining:  (356.24 GB)

DFS Used%: 2.36%

DFS Remaining%: 92.35%

Configured Cache Capacity:  ( GB)

Cache Used:  ( B)

Cache Remaining:  ( GB)

Cache Used%: 0.00%

Cache Remaining%: 100.00%

Xceivers:

Last contact: Sat Dec  :: CST 

Name: 172.26.0.109: (el-hadoop-)

Hostname: el-hadoop-

Rack: /default

Decommission Status : Normal

Configured Capacity:  (385.73 GB)

DFS Used:  (9.10 GB)

Non DFS Used:  ( B)

DFS Remaining:  (356.24 GB)

DFS Used%: 2.36%

DFS Remaining%: 92.35%

Configured Cache Capacity:  ( GB)

Cache Used:  ( B)

Cache Remaining:  ( GB)

Cache Used%: 0.00%

Cache Remaining%: 100.00%

Xceivers:

Last contact: Sat Dec  :: CST

码农公寓

解决办法：

相关文章