bulk write hbase 时遇到下面的错误：
19/03/20 02:16:02 ERROR LoadIncrementalHFiles: IOException during splitting
java.util.concurrent.ExecutionException: java.io.IOException: The endkey of one region for table ap:users_v2 is not equal to the startkey of the next region in hbase:meta.Please use hbck tool to fix it first.
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:558)
at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:414)
at com.hotstar.ap.ingest.batch.tool.bulkload.BulkloadHelper$.bulkloadWrite(BulkloadHelper.scala:55)
at com.hotstar.ap.ingest.batch.tool.HBaseWriterHelper$.writeToHBase(HBaseWriterHelper.scala:57)
at com.hotstar.ap.ingest.batch.ingester.v2.VirtualBatchIngester.runOffline(VirtualBatchIngester.scala:20)
at com.hotstar.ap.ingest.batch.ingester.VirtualIngester.run(VirtualIngester.scala:12)
at com.hotstar.ap.ingest.batch.Main$$anonfun$1.apply(Main.scala:52)
at com.hotstar.ap.ingest.batch.Main$$anonfun$1.apply(Main.scala:52)
at scala.collection.immutable.List.foreach(List.scala:381)
at com.hotstar.ap.ingest.batch.Main$.delayedEndpoint$com$hotstar$ap$ingest$batch$Main$1(Main.scala:52)
at com.hotstar.ap.ingest.batch.Main$delayedInit$body.apply(Main.scala:14)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at com.hotstar.ap.ingest.batch.Main$.main(Main.scala:14)
at com.hotstar.ap.ingest.batch.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:721)
Caused by: java.io.IOException: The endkey of one region for table ap:users_v2 is not equal to the startkey of the next region in hbase:meta.Please use hbck tool to fix it first.
at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:682)
at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3.call(LoadIncrementalHFiles.java:548)
at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3.call(LoadIncrementalHFiles.java:545)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
sudo -u hbase hbase hbck 检查，存在 inconsistence，其中表 ap:users_v2 有错误，但显示为
Table ap:users_v2 is okay.
Number of regions: 201
Deployed on: ip-10-10-240-127.ec2.internal,16020,1552903865915 ip-10-10-240-145.ec2.internal,16020,1552903928202 ip-10-10-240-48.ec2.internal,16020,1552903982881 ip-10-10-241-65.ec2.internal,16020,1552904038213 ip-10-10-243-103.ec2.internal,16020,1552904093359 ip-10-10-243-116.ec2.internal,16020,1552904166150
sudo -u hbase hbase hbck -repair 修复，解决了 inconsistence，结果显示为
Table ap:users_v2 is okay.
Number of regions: 202
Deployed on: ip-10-10-240-127.ec2.internal,16020,1552903865915 ip-10-10-240-145.ec2.internal,16020,1552903928202 ip-10-10-240-48.ec2.internal,16020,1552903982881 ip-10-10-241-65.ec2.internal,16020,1552904038213 ip-10-10-243-103.ec2.internal,16020,1552904093359 ip-10-10-243-116.ec2.internal,16020,1552904166150