spark 2.4
spark sql中执行
set hive.exec.max.dynamic.partitions=10000;
后再执行sql依然会报错:
org.apache.hadoop.hive.ql.metadata.HiveException:
Number of dynamic partitions created is 1001, which is more than 1000.
To solve this try to set hive.exec.max.dynamic.partitions to at least 1001.
这个参数hive.exec.max.dynamic.partitions的默认值是1000,修改没有生效,
原因如下:
`HiveClient` does not know new value 1001. There is no way to change the default value of `hive.exec.max.dynamic.partitions` of `HiveCilent` with `SET` command.
The root cause is that `hive` parameters are passed to `HiveClient` on creating. So, the workaround is to use `--hiveconf` when starting `spark-shell`.
解决方法是在启动spark-sql时设置hiveconf
spark-sql --hiveconf hive.exec.max.dynamic.partitions=10000
参考:
https://issues.apache.org/jira/browse/SPARK-19881