如果运行出现以下报错:
Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.ClassTag$.Int()Lscala/reflect/ManifestFactory$IntManifest;
是Scala版本不匹配造成,请参见:
Spark3.2教程(前置)关于Spark3.2.0与Scala版本的坑
在IDEA中开发:
package com.alan
import org.apache.spark.{SparkConf, SparkContext}
object Test1 {
def main(args: Array[String]): Unit = {
//创建SparkContexta
//本地运行
// val conf=new SparkConf().setMaster("local[2]").setAppName("WordCount")
//提交运行
val conf = new SparkConf().setAppName("WordCound");
val sc = new SparkContext(conf)
//加载文件
val file = sc.textFile("hdfs://hp301:9000/ee/b.txt")
//val file=sc.textFile("file:///d:/test/words.txt")
//处理
val flatFile = file.flatMap(item => item.split(" "))
val flatFileMap = flatFile.map(item => (item, 1))
val aggCount = flatFileMap.reduceByKey((curr, agg) => curr + agg)
//得到结果
aggCount.foreach(item => println(item))
}
}
注意:本次是从HDFS中取数据进行计算,如果没有配置Hadoop,可以改成从Linux本地读取。
配置pom.xml中打包插件,为了将依赖都打入包中:
<bulid>
<!--指定源码包和测试包的位置-->
<sourceDirectory>src/main/scala</sourceDirectory>
<plugins>
<!-- 该插件用于将 Scala 代码编译成 class 文件 -->
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<executions>
<execution>
<!-- 声明绑定到 maven 的 compile 阶段 -->
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<args>
<arg>-target:jvm-1.8</arg>
</args>
</configuration>
</execution>
</executions>
</plugin>
<!--maven打包的插件-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>com.alan.Test1</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
双击package命令,执行Maven打包:
将打好的包上传到linux的/apps文件夹下,执行命令:
spark-submit --master spark://hp301:7077 /apps/test_s2-1.0-SNAPSHOT-jar-with-dependencies.jar
在Web UI中可以看到结果: