hive权限有两层，hive本身的验证和hadoop的验证。自定义hive的proxy功能时，hive层面的相关验证更改在
http://caiguangguang.blog.51cto.com/1652935/1587251
中已经提过，这里说下hive和hadoop以及本地文件系统相关的几个出现用户的地方：
1.job的log文件
session初始化时会初始化日志文件，主要在SessionState的start方法中：
|
1
2
3
4
5
6
7
8
9
10
11
12
|
public static SessionState start(SessionState startSs) {
setCurrentSessionState(startSs);
if(startSs.hiveHist == null){
if (startSs.getConf().getBoolVar(HiveConf.ConfVars.HIVE_SESSION_HISTORY_ENABLED)) {
// 如果设置hive.session.history.enabled为true，则会初始化日志文件,默认为false
startSs.hiveHist = new HiveHistoryImpl(startSs); // 由HiveHistoryImpl 产生日志文件
}else {
//Hive history is disabled, create a no-op proxy
startSs.hiveHist = HiveHistoryProxyHandler.getNoOpHiveHistoryProxy();
}
}
...
|
再来看org.apache.hadoop.hive.ql.history.HiveHistoryImpl类的构造函数，定义了日志的路径，如果日志目录不存在，则创建目录：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
|
public HiveHistoryImpl(SessionState ss) {
try {
console = new LogHelper(LOG);
String conf_file_loc = ss.getConf().getVar(
HiveConf.ConfVars.HIVEHISTORYFILELOC);
//HIVEHISTORYFILELOC("hive.querylog.location", System.getProperty("java.io.tmpdir") + File.separator + System.getProperty("user.name")),
默认值是/tmp/${user.name}/目录
if ((conf_file_loc == null) || conf_file_loc.length() == 0) {
console.printError("No history file location given");
return;
}
// Create directory
File histDir = new File(conf_file_loc);
if (!histDir.exists()) { //创建日志目录
if (!histDir.mkdirs()) {
console.printError("Unable to create log directory " + conf_file_loc);
return;
}
}
do {
histFileName = conf_file_loc + File.separator + "hive_job_log_" + ss.getSessionId() + "_"
+ Math.abs(randGen.nextInt()) + ".txt";
// 日志文件的完整路径 比如 /tmp/hdfs/hive_job_log_4f96f470-a6c1-41ae-9d30-def308e5412f_564454280.txt/tmp/hdfs/hive_job_log_sessionid_随机数.txt } while (! new File(histFileName).createNewFile());
console.printInfo("Hive history file=" + histFileName);
histStream = new PrintWriter(histFileName);
HashMap<String, String> hm = new HashMap<String, String>();
hm.put(Keys.SESSION_ID.name(), ss.getSessionId());
log(RecordTypes.SessionStart, hm);
} catch (IOException e) {
console.printError("FAILED: Failed to open Query Log : " + histFileName
+ " " + e.getMessage(), "\n"
+ org.apache.hadoop.util.StringUtils.stringifyException(e));
}
} |
2.job的中间文件
hive执行过程中保存在hdfs的路径，由hive.exec.scratchdir和hive.exec.local.scratchdir定义
scratch文件是在org.apache.hadoop.hive.ql.Context类的构造方法中获取
关于scratch目录的相关配置：
|
1
2
3
4
|
SCRATCHDIR("hive.exec.scratchdir", "/tmp/hive-" + System.getProperty("user.name")),
//默认值为/tmp/hive-当前登录用户LOCALSCRATCHDIR("hive.exec.local.scratchdir", System.getProperty("java.io.tmpdir") + File.separator + System.etProperty("user.name")),
SCRATCHDIRPERMISSION("hive.scratch.dir.permission", "700"),
|
在org.apache.hadoop.hive.ql.Context类的构造方法中
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
// scratch path to use for all non-local (ie. hdfs) file system tmp folders
private final Path nonLocalScratchPath;
// scratch directory to use for local file system tmp folders
private final String localScratchDir ;
// the permission to scratch directory (local and hdfs )
private final String scratchDirPermission ;
...public Context(Configuration conf, String executionId) {
this.conf = conf;
this.executionId = executionId;
// local & non-local tmp location is configurable. however it is the same across
// all external file systems
nonLocalScratchPath =
new Path(HiveConf.getVar(conf, HiveConf.ConfVars.SCRATCHDIR),
executionId);
localScratchDir = new Path(HiveConf.getVar(conf, HiveConf.ConfVars.LOCALSCRATCHDIR),
executionId).toUri().getPath();
scratchDirPermission= HiveConf.getVar(conf, HiveConf.ConfVars.SCRATCHDIRPERMISSION);
} |
在Driver的compile方法中会初始化这个对象。
3.job提交的用户
|
1
2
3
4
5
6
7
8
|
JobClient的init方法 UserGroupInformation clientUgi;
.... public void init( JobConf conf) throws IOException {
setConf(conf);
cluster = new Cluster(conf);
clientUgi = UserGroupInformation.getCurrentUser();
}
|
这里增加proxy比较容易，用UserGroupInformation的createRemoteUser方法即可：
比如把init方法改为：
|
1
2
3
4
5
6
7
8
9
10
11
12
|
public void init(JobConf conf) throws IOException {
setConf(conf);
cluster = new Cluster(conf);
if (conf.getBoolean("use.custom.proxy",false))
{
String proxyUser = conf.get("custom.proxy.user");
clientUgi = UserGroupInformation.createRemoteUser(proxyUser);
}else{
clientUgi = UserGroupInformation.getCurrentUser();
}
LOG.warn("clientUgi is " + clientUgi);
} |
本文转自菜菜光 51CTO博客，原文链接：http://blog.51cto.com/caiguangguang/1589874，如需转载请自行联系原作者