DBeaver连接Hive

2023-09-30 10:30:52

DBeaver连接Hive

准备工作

hadoop分布式或者伪分布已经安装好

hive已经安装好并且已经初始化

DBeaver客户端工具

可参考：https://blog.csdn.net/cww468/article/details/122328122?spm=1001.2014.3001.5501

启动Hadoop：

start-dfs.sh

启动Hive服务

hive --service metastore &  #启动Metastore服务

hive --service hiveserver2 & #启动hiveserver2服务

从服务器下载jar包

从服务器$HAOOP_HOME/share/hadoop/common/目录下下载jar包，我这里是hadoop-common-2.9.2
从服务器$HAOOP_HOME/share/hadoop/common/lib目录下下载protobuf的jar包，我这里是protobuf-java-2.5.0.jar
从服务器$HIVE_HOME/jdbc/目录下下载jar包，我这里是hive-jdbc-2.3.7-standalone.jar

使用DBeaver连接

打开DBeaver工具，点击 数据库 -- 新建连接 -- 选择Apache Hive 后，填写主机名称、端口、连接数据库、用户名、密码

点击 编辑驱动设置 -- 库 -- 添加文件，添加上一步下载的jar包，点击找到类 ，确定后点击 测试链接，没有问题后就可以成功连接Hive

可能遇到的问题

问题1：用户root不允许伪装成anonymous（hive的默认用户），hadoop引入了一个安全伪装机制，使得hadoop 不允许上层系统直接将实际用户传递到hadoop层，而是将实际用户传递给一个超级代理，由此代理在hadoop上执行操作，避免任意客户端随意操作hadoop。

问题描述：org.apache.hadoop.ipc.RemoteException: User: root is not allowed to impersonate root

解决方法：通过httpfs协议访问rest接口，以root用户包装默认用户的方式操作HDFS

vim hdfs-site.xml

#新增以下配置
<property>  
	<name>dfs.webhdfs.enabled</name>  
	<value>true</value>  
</property>

#将文件权限检查配置为false
<property>
    <name>dfs.permissions</name>
    <value>false</value>
</property>

vim core-site.xml

#新增以下配置
<property>
	<name>hadoop.proxyuser.root.hosts</name> #root是用户名，异常[User: root is not allowed to impersonate anonymous]中User后面的用户是什么，在这里就配置什么。
	<value>*</value>                         # *表示任意节点使用 hadoop 集群的代理用户 root 都能访问 hdfs 集群
</property>
<property>
	<name>hadoop.proxyuser.root.groups</name># *表示代理用户的组所属
	<value>*</value>
</property>

#以上配置完成后重启
	stop-dfs.sh
	stop-yarn.sh
	start-dfs.sh
	start-yarn.sh

问题2：hive驱动版本的问题
问题描述：
Required field ‘serverProtocolVersion’ is unset! Struct:TOpenSessionResp(status:TStatus(statusCode:ERROR_STATUS, infoMessages:[*org.apache.hive.service.cli.HiveSQLException:Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: caimh is not allowed to impersonate root:14:13, org.apache.hive.service.cli.session.SessionManager:createSession:SessionManager.java:419, org.apache.hive.service.cli.session.SessionManager:openSession:SessionManager.java:362, org.apache.hive.service.cli.CLIService:openSessionWithImpersonation:CLIService.java:193, org.apache.hive.service.cli.thrift.ThriftCLIService:getSessionHandle:ThriftCLIService.java:440, org.apache.hive.service.cli.thrift.ThriftCLIService:OpenSession:ThriftCLIService.java:322, org.apache.hive.service.rpc.thrift.TCLIService Processor OpenSession:getResult:TCLIService.java:1377, org.apache.hive.service.rpc.thrift.TCLIServiceProcessor OpenSession:getResult:TCLIService.java:1362, org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39, org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39, org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56, org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:286, java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149, java.util.concurrent.ThreadPoolExecutor Worker:run:ThreadPoolExecutor.java:624, java.lang.Thread:run:Thread.java:748, *java.lang.RuntimeException:java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: caimh is not allowed to impersonate root:22:8, org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:89, org.apache.hive.service.cli.session.HiveSessionProxy:access000:HiveSessionProxy.java:36, org.apache.hive.service.cli.session.HiveSessionProxy1:run:HiveSessionProxy.java:63, java.security.AccessController:doPrivileged:AccessController.java:-2, javax.security.auth.Subject:doAs:Subject.java:422, org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1746, org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:59, com.sun.proxy.Proxy37:open::-1, org.apache.hive.service.cli.session.SessionManager:createSession:SessionManager.java:410, *java.lang.RuntimeException:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: caimh is not allowed to impersonate root:29:7, org.apache.hadoop.hive.ql.session.SessionState:start:SessionState.java:610, org.apache.hadoop.hive.ql.session.SessionState:start:SessionState.java:548, org.apache.hive.service.cli.session.HiveSessionImpl:open:HiveSessionImpl.java:164, sun.reflect.NativeMethodAccessorImpl:invoke0:NativeMethodAccessorImpl.java:-2, sun.reflect.NativeMethodAccessorImpl:invoke:NativeMethodAccessorImpl.java:62, sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43, java.lang.reflect.Method:invoke:Method.java:498, org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:78, *org.apache.hadoop.ipc.RemoteException:User: caimh is not allowed to impersonate root:49:20, org.apache.hadoop.ipc.Client:call:Client.java:1476, org.apache.hadoop.ipc.Client:call:Client.java:1413, org.apache.hadoop.ipc.ProtobufRpcEngineInvoker:invoke:ProtobufRpcEngine.java:229, com.sun.proxy.Proxy30:getFileInfo::-1, org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB:getFileInfo:ClientNamenodeProtocolTranslatorPB.java:776, sun.reflect.NativeMethodAccessorImpl:invoke0:NativeMethodAccessorImpl.java:-2, sun.reflect.NativeMethodAccessorImpl:invoke:NativeMethodAccessorImpl.java:62, sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43, java.lang.reflect.Method:invoke:Method.java:498, org.apache.hadoop.io.retry.RetryInvocationHandler:invokeMethod:RetryInvocationHandler.java:191, org.apache.hadoop.io.retry.RetryInvocationHandler:invoke:RetryInvocationHandler.java:102, com.sun.proxy.Proxy31:getFileInfo::-1, org.apache.hadoop.hdfs.DFSClient:getFileInfo:DFSClient.java:2108, org.apache.hadoop.hdfs.DistributedFileSystem22:doCall:DistributedFileSystem.java:1305, org.apache.hadoop.hdfs.DistributedFileSystem22:doCall:DistributedFileSystem.java:1301, org.apache.hadoop.fs.FileSystemLinkResolver:resolve:FileSystemLinkResolver.java:81, org.apache.hadoop.hdfs.DistributedFileSystem:getFileStatus:DistributedFileSystem.java:1317, org.apache.hadoop.fs.FileSystem:exists:FileSystem.java:1426, org.apache.hadoop.hive.ql.session.SessionState:createRootHDFSDir:SessionState.java:708, org.apache.hadoop.hive.ql.session.SessionState:createSessionDirs:SessionState.java:654, org.apache.hadoop.hive.ql.session.SessionState:start:SessionState.java:586], errorCode:0, errorMessage:Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: caimh is not allowed to impersonate root), serverProtocolVersion:null)

解决方法：删除之前在DBeaver中的所有驱动，确保按照上面步骤从服务器下载jar包并添加到DBeaver中

问题3：缺少protobuf-jar包
问题描述：com/google/protobuf/ProtocolMessageEnum
解决方法：确保按照上面步骤从服务器下载protobuf的jar包并添加到DBeaver中

文章参考

https://blog.csdn.net/qq_32252917/article/details/102915507?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522164146183016781683953501%2522%252C%2522scm%2522%253A%252220140713.130102334…%2522%257D&request_id=164146183016781683953501&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2_allsobaiduend~default-3-102915507.first_rank_v2_pc_rank_v29&utm_term=dbeaver+hive&spm=1018.2226.3001.4187

https://blog.csdn.net/qq_43225978/article/details/97781612?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522164153924416780255245076%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fall.%2522%257D&request_id=164153924416780255245076&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2_allfirst_rank_ecpm_v1~rank_v31_ecpm-3-97781612.first_rank_v2_pc_rank_v29&utm_term=org.apache.hadoop.ipc.RemoteException%3AUser%3A+root+is+not+allowed+to+impersonate+root&spm=1018.2226.3001.4187

码农公寓

DBeaver连接Hive

相关文章