服务器集群上用spark-shell做简单测试

spark-shell --jars xxx.jar xx.jar

>>>:paste


import
org.datasyslab.geospark.serde.GeoSparkKryoRegistrator import org.datasyslab.geosparkviz.core.Serde.GeoSparkVizKryoRegistrator import org.datasyslab.geosparksql.utils.GeoSparkSQLRegistrator import org.apache.spark.sql.SparkSession import org.apache.spark.SparkConf import org.apache.spark.SparkContext import org.apache.spark.serializer.KryoSerializer import org.datasyslab.geospark.serde.GeoSparkKryoRegistrator
var sparkBuild
= SparkSession .builder() .appName("testsql2") .config("spark.serializer", classOf[KryoSerializer].getName) .config("spark.kryo.registrator", classOf[GeoSparkKryoRegistrator].getName) .enableHiveSupport() sparkBuild.config("spark.driver.allowMultipleContexts","true") sparkBuild.config("spark.sql.small.file.combine","true") sparkBuild.config("spark.sql.small.file.split.size","512000") val spaksson: SparkSession = sparkBuild.getOrCreate() GeoSparkSQLRegistrator.registerAll(spaksson) spaksson.sql(s""" select (ST_GeomFromWKT(agg_env)) as mulitenv from xxx where xxx=‘2021-07-8‘""".stripMargin).show()

 

服务器集群上用spark-shell做简单测试

上一篇:ubuntu20.04启用root账户登录


下一篇:Linux 下安装 Nginx