参考文章:http://www.jianshu.com/p/791137760c14
运行SparkStreming程序一段时间后,发现产生了异常:
ERROR JobScheduler: Error running job streaming job ms.
org.apache.spark.SparkException:
Job aborted due to stage failure:
Task in stage 37560.0 failed times,
most recent failure: Lost task 13.3 in stage 37560.0
(TID , 192.169.2.33, executor ):
kafka.common.OffsetOutOfRangeException
如果消息体太大了,超过 fetch.message.max.bytes=1m
的默认配置,那么Spark Streaming会直接抛出OffsetOutOfRangeException异常,然后停止服务。
解决方案:Kafka consumer中设置fetch.message.max.bytes为大一点的内存
比如设置为50M:1024*1024*50
fetch.message.max.bytes=52428800