spark 从数据库中读取数据

spark 从数据库中读取数据的几种方式,以Mysql 为例

创建spark
val spark = SparkSession.builer().appName("用户标签").master("local[*]").getOrCreate()
val url = "jdbc:mysql://数据库地址/数据库名"
val driver = "mysql 数据库驱动”
val tableName = "表名"
val user = "数据库用户名"
val password = "数据库密码"

方式一: 直连
val df: DataFrame = spark.read.format("jdbc").options(Map(
      "url" -> url,
      "driver" -> driver
      "dbtable" -> tableName,
      "user"-> user,
      "password" -> password
      )).load()

方式二: 直连,执行指定sql
val sql = "select id, username from tableName"
val df: DataFrame = spark.read.format("jdbc").options(Map(
      "url" -> url,
      "driver" -> driver
      "dbtable" -> sql,
      "user"-> user,
      "password" -> password
      )).load()

方式三:使用jdbc 的properties
val prop = new Properties()
prop.setProperty("user", user)
prop.setProperty("password", password)
val df: DataFrame = spark.read.jdbc(url, tableName, prop)

方式四:传统的数据库驱动,查询结果是ResultSet 类型
try{
      Class.forName(driver)
var conn: Connection = DriverManager.getConnection(url, user, password)
val statement = conn.createStatement()
val res = statement.executeQuery(select * from tableName)
}

方式五:hive 数据库,与spark 天然结合(需要配置hive 作为数据源)
spark.sql("select * from tableName)

spark 从数据库中读取数据

上一篇:SQL Server中Substring的用法


下一篇:Ubuntu18.04下MySQL5.7的安装