解决Spark本地测试异常之 System memory 259522560 must be at least 471859200
一、异常如下
java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:217)
at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:199)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:332)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:432)
at com.temptag.app.TagGenerator$.main(TagGenerator.scala:14)
at com.temptag.app.TagGenerator.main(TagGenerator.scala)
19/04/08 20:31:19 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:217)
at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:199)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:332)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:432)
at com.temptag.app.TagGenerator$.main(TagGenerator.scala:14)
at com.temptag.app.TagGenerator.main(TagGenerator.scala)
二、抛出异常原因
Spark是非常依赖内存的计算框架，在Win环境下使用local模式时，实际上是使用多线程的形式模拟集群进行计算，因而对于计算机的内存有一定要求，这是典型的因为计算机内存不足而抛出的异常。
三、解决办法
在设置SparkContext时，多设置一个如下参数，来指定分配给Spark的内存，这样问题就解决啦！
val sparkConf = new SparkConf()
.set("spark.testing.memory","2147480000")