spark 从数据库中读取数据的几种方式,以Mysql 为例
val spark = SparkSession.builer().appName("用户标签").master("local[*]").getOrCreate()
val url = "jdbc:mysql://数据库地址/数据库名"
val driver = "mysql 数据库驱动”
val tableName = "表名"
val user = "数据库用户名"
val password = "数据库密码"
val df: DataFrame = spark.read.format("jdbc").options(Map(
"url" -> url,
"driver" -> driver
"dbtable" -> tableName,
"user"-> user,
"password" -> password
)).load()
val sql = "select id, username from tableName"
val df: DataFrame = spark.read.format("jdbc").options(Map(
"url" -> url,
"driver" -> driver
"dbtable" -> sql,
"user"-> user,
"password" -> password
)).load()
val prop = new Properties()
prop.setProperty("user", user)
prop.setProperty("password", password)
val df: DataFrame = spark.read.jdbc(url, tableName, prop)
方式四:传统的数据库驱动,查询结果是ResultSet 类型 |
try{
Class.forName(driver)
var conn: Connection = DriverManager.getConnection(url, user, password)
val statement = conn.createStatement()
val res = statement.executeQuery(select * from tableName)
}
方式五:hive 数据库,与spark 天然结合(需要配置hive 作为数据源) |
spark.sql("select * from tableName)
spark 从数据库中读取数据