之前同事处理的一个case,记录下
hive升级到0.13之后,在创建表时,发现报锁竞争的问题,信息如下:
1
|
conflicting lock present for vipdw mode EXCLUSIVE
|
hive中有锁是没有问题,但是这里的锁却是数据库层面的锁!而且是排他锁!这个锁的粒度就太大了,这个锁会导致所有的关于这个库的hive操作都要等待这个锁的释放.
这个应该是去拿表的锁啊。。怎么可以拿库的锁。。
根据之前的分析,hive的锁由DummyTxnManager类实现:
在DummyTxnManager类中,关于EXCLUSIVE锁的获取如下
代码路径:
1
|
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java |
原始代码:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
for (WriteEntity output : plan.getOutputs()) {
LOG.debug( "Adding " + output.getName() + " to list of lock outputs" );
List<HiveLockObj> lockObj = null ;
if (output.getType() == WriteEntity.Type.DATABASE) {
lockObjects.addAll(getLockObjects(plan, output.getDatabase(), null ,
null ,
output.isComplete() ? HiveLockMode.EXCLUSIVE : HiveLockMode.SHARED));
} else if (output.getTyp() == WriteEntity.Type.TABLE) {
lockObj = getLockObjects(plan, null , output.getTable(), null ,
output.isComplete() ? HiveLockMode.EXCLUSIVE : HiveLockMode.SHARED);
} else if (output.getTyp() == WriteEntity.Type.PARTITION) {
lockObj = getLockObjects(plan, null , null , output.getPartition(),
HiveLockMode.EXCLUSIVE);
}
// In case of dynamic queries, it is possible to have incomplete dummy partitions
else if (output.getTyp() == WriteEntity.Type.DUMMYPARTITION) {
lockObj = getLockObjects(plan, null , null , output.getPartition(),
HiveLockMode.SHARED);
}
if (lockObj != null ) {
lockObjects.addAll(lockObj);
ctx.getOutputLockObjects().put(output, lockObj);
}
}
|
更改源码,把数据库的排它锁这一段去掉:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
for (WriteEntity output : plan.getOutputs()) {
LOG.debug( "Adding " + output.getName() + " to list of lock outputs" );
List<HiveLockObj> lockObj = null ;
if (output.getType() == WriteEntity.Type.DATABASE) {
LOG.debug( "Output has database:" +output.getDatabase());
LOG.debug( "Removing unnecessary output lock aquirement on " +output .getDatabase()+ " database" );
// lockObjects.addAll(getLockObjects(plan, output.getDatabase(), null,
// null,
// output.isComplete() ? HiveLockMode.EXCLUSIVE : HiveLockMode.SHARED));
} else if (output.getTyp() == WriteEntity.Type.TABLE) {
lockObj = getLockObjects( plan, null , output.getTable(), null ,
output.isComplete() ? HiveLockMode.EXCLUSIVE : HiveLockMode .SHARED);
} else if (output.getTyp() == WriteEntity.Type.PARTITION) {
lockObj = getLockObjects( plan, null , null , output.getPartition(),
HiveLockMode.EXCLUSIVE);
}
// In case of dynamic queries, it is possible to have incomplete dummy partitions
else if (output.getTyp() == WriteEntity.Type.DUMMYPARTITION) {
lockObj = getLockObjects( plan, null , null , output.getPartition(),
HiveLockMode.SHARED);
}
if (lockObj != null ) {
lockObjects.addAll(lockObj);
ctx.getOutputLockObjects().put(output , lockObj);
}
}
|
本文转自菜菜光 51CTO博客,原文链接:http://blog.51cto.com/caiguangguang/1554541,如需转载请自行联系原作者