hbase读取快照数据-lzo压缩遇到的问题

2023-12-30 08:18:58

1.读取hbase快照数据时报UnsatisfiedLinkError: no gplcompression in java.library.path错：

2019-09-04 17:36:07,441 ERROR [testJobName,1,ReaderTask_HbaseReader_0_1] GPLNativeCodeLoader:35 | Could not load native gpl library
java.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:31)
	at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:60)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:127)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:116)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:328)
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97)
	at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452)
	at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60)
	at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209)
	at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98)
	at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73)
	at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-09-04 17:36:07,460 ERROR [testJobName,1,ReaderTask_HbaseReader_0_1] LzoCodec:70 | Cannot load native-lzo without native-hadoop

　　在使用java 命令执行时可加上 -Djava.library.path=/opt/hadoop/lib/native，就没有此报错

java  -Djava.library.path=/opt/hadoop/lib/native -cp lib/*:conf/*:stream-0.0.1.jar com.hbasereader.HbaseReader

2.解决完上述异常后，又出现native-lzo library not available异常

 org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:102)
	at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452)
	at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60)
	at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209)
	at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98)
	at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73)
	at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: native-lzo library not available
	at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:330)
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97)
	... 21 more

	at com.suning.data.integration.engine.stream.core.exception.DiException.asDiException(DiException.java:51)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:131)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.doStart(HbaseReader.java:94)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.access$000(HbaseReader.java:32)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:76)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader$1.run(HbaseReader.java:73)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.HbaseReader.start(HbaseReader.java:73)
	at com.suning.data.integration.engine.stream.plugin.reader.ReaderTask$ReaderThread.run(ReaderTask.java:122)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.RuntimeException: native-lzo library not available
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:102)
	at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:6562)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6514)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6481)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6452)
	at org.apache.hadoop.hbase.client.ClientSideRegionScanner.<init>(ClientSideRegionScanner.java:60)
	at org.apache.hadoop.hbase.mapreduce.TableSnapshotInputFormatImpl$RecordReader.initialize(TableSnapshotInputFormatImpl.java:209)
	at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat$TableSnapshotRecordReader.<init>(TableSnapshotInputFormat.java:98)
	at org.apache.hadoop.hbase.mapred.TableSnapshotInputFormat.getRecordReader(TableSnapshotInputFormat.java:152)
	at com.suning.data.integration.engine.stream.plugin.reader.hbasereader.utils.HbaseReaderUtil.read(HbaseReaderUtil.java:95)
	... 12 more
Caused by: java.lang.RuntimeException: native-lzo library not available
	at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:330)
	at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:97)
	... 21 more

 错误是从这报出来的：

hbase读取快照数据-lzo压缩遇到的问题

 

 

经过一周的排查终于找到原因了，真是坑爹，因为引用了hadoop-gpl-compression-xxx.jar，而hadoop-gpl-compression是比较老的版本，

hbase读取快照数据-lzo压缩遇到的问题

 

 将hadoop-gpl-compression注释掉，使用hadoop-lzo的jar包，问题解决

目前在使用Twitter维护的版本，在GitHub上，下载源码 git clone https://github.com/twitter/hadoop-lzo.git ，使用maven编译打包后，将hadoop-lzo-xxx.jar加入到项目中，完美解决上述native-lzo library not available 问题。

至于如何使用maven编译hadoop-lzo，可以参照我的一篇博客： Hadoop3.1.2 + Hbase2.2.0 设置lzo压缩算法 

 

注意：hadoop-gpl-compression已经过时，请使用hadoop-lzo

　　   hadoop-gpl-compression已经过时，请使用hadoop-lzo

　　   hadoop-gpl-compression已经过时，请使用hadoop-lzo

　　　重要的事情说三遍！！！

上一篇：java – Hadoop-LZO奇怪的native-lzo库没有错误


下一篇：Hadoop配置lzo和lzop