Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

Spark submit任务到Spark集群时,会出现如下异常:

Exception 1:Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

查看Spark logs文件spark-Spark-org.apache.spark.deploy.master.Master-1-hadoop1.out发现:

Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

此时的Spark Web UI界面如下:

Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

Reason:

之前在Spark配置文件spark-env.sh中,SPARK_LOCAL_IP的配置是localhost,内存为512M,所以在Spark UI界面中Workers显示均对应到主机hadoop1默认的的localhost,只是给它分配的workers对应不同的端口而已,而之前最大内存为2.9G,所以5 * 512M > 2.9G的,故上述报错。

Solution:

修改Spark配置文件spark-env.sh,将SPARK_LOCAL_IP的localhost修改为对应的主机名称(hadoop1,hadoop2...),并修改SPARK_WORKER_MEMORY的内存配置 < 对应机器分配的内存 即可

Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

提交任务(WordCount)到Spark集群中,相应脚本如下:

Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

执行脚本,运行Spark任务,过程如下:

./runSpark.sh

Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

对应的WordCount结果如下:

Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

此时对应的Spark运行UI界面如下:

Spark之submit任务时的Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

上一篇:学校的统一订书80%该烧掉——IT推荐书单


下一篇:CI 结合 vue.js 的搜索功能模块