线上使用cdh4.6.0和hive0.13.1,在hive0.11之后开始支持orcfile,hive0.13.1使用的是pb2.5.0,cdh4.6.0是用的2.4.0a，线上测试orcfile，建表正常，但是插入数据时报错报错：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
java.lang.VerifyError: class org.apache.hadoop.hive.ql.io.orc.OrcProto$RowIndex overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(Unknown Source)
at java.security.SecureClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.access$100(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.apache.hadoop.hive.ql.io.orc.WriterImpl.<init>(WriterImpl.java:129)
at org.apache.hadoop.hive.ql.io.orc.OrcFile.createWriter(OrcFile.java:369)
at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.close(OrcOutputFormat.java:103)
at org.apache.hadoop.hive.ql.exec.Utilities.createEmptyFile(Utilities.java:3065)
at org.apache.hadoop.hive.ql.exec.Utilities.createDummyFileForEmptyPartition(Utilities.java:3089)
at org.apache.hadoop.hive.ql.exec.Utilities.getInputPaths(Utilities.java:3013)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:369)
at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
|
从错误可以看出是由于pb兼容问题导致，尝试使用pb2.4.0a编译hive0.13.1:
1.pom.xml文件：
更改
|
1
|
<protobuf.version>2.5.0</protobuf.version>
|
为
|
1
|
<protobuf.version>2.4.0a</protobuf.version>
|
2.运行mvn进行编译：
|
1
|
mvn clean package -DskipTests -Phadoop-2 -Pdist
|
发现报如下错误：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
[ERROR] /home/caiguangguang/hive_0.13_debug/ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java:[9828,38] cannot find symbolsymbol
: class Parserlocation: package com.google.protobuf[ERROR]
/home/caiguangguang/hive_0.13_debug/ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java:[9839,31] cannot find symbolsymbol :
class Parserlocation: package com.google.protobuf
....[ERROR] /home/caiguangguang/hive_0.13_debug/ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java:[169,9] getUnknownFields()
in org.apache.hadoop.hive.ql.io.orc.OrcProto.IntegerStatistics cannot override getUnknownFields() in com.google.protobuf.GeneratedMessage; overridden method is final[ERROR] /home/caiguangguang/hive_0.13_debug/ql/src/gen/protobuf/
gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java:[189,20]
cannot find symbolsymbol : method parseUnknownField(com.google.protobuf.CodedInputStream,com.google.protobuf.UnknownFieldSet.Builder,com.google.protobuf.ExtensionRegistryLite,int)
|
3.这里
|
1
|
hive_0.13_debug/ql/src/gen/protobuf/gen-java/org/apache/hadoop/hive/ql/io/orc/OrcProto.java |
是由protoc2.5.0生成的，但是jar包却是2.4.0a的，因此会报错
4.考虑使用2.4.0a的protoc手动生成java代码
|
1
2
|
protoc --versionlibprotoc 2.4.0a
|
这里主要涉及两个proto文件：
|
1
2
3
|
protoc ql/src/protobuf/org/apache/hadoop/hive/ql/io/orc/orc_proto.proto --java_out=ql/src/gen/protobuf/gen-java/
protoc ./hcatalog/storage-handlers/hbase/src/protobuf/org/apache/hcatalog/hbase/snapshot/RevisionManagerEndpoint.proto
--java_out=hcatalog/storage-handlers/hbase/src/gen-java/
|
5.重新编译成功，将hive-exec jar包解压后，可以看到pom.properties中pb的版本已经是2.4.0a了：
|
1
2
3
4
5
6
|
META-INF/maven/com.google.protobuf/protobuf-java/pom.properties
#Generated by Maven#Thu Mar 10 11:50:31 CST 2011version=2.4.0agroupId=com.google.protobufartifactId=protobuf-java |
更换hive-exec jar包之后，测试向orc file的表插入数据也正常。
本文转自菜菜光 51CTO博客，原文链接：http://blog.51cto.com/caiguangguang/1599128，如需转载请自行联系原作者