【Druid】集群问题总结
1、druid安装目录下的var文件夹下手动创建tmp文件夹
2021-07-07 09:03:03 [task-runner-0-priority-0] ERROR SeekableStreamIndexTaskRunner:123 - Encountered exception while running task.
org.apache.druid.java.util.common.ISE: Failed to create temporary directory in [var/tmp]! Make sure the `java.io.tmpdir` property is set to an existing and writable directory with enough free space.
at org.apache.druid.segment.indexing.RealtimeTuningConfig.createNewBasePersistDirectory(RealtimeTuningConfig.java:67) ~[druid-server-0.16.0-incubating-SNAPSHOT.jar:?]
at org.apache.druid.segment.indexing.RealtimeTuningConfig.makeDefaultTuningConfig(RealtimeTuningConfig.java:79) ~[druid-server-0.16.0-incubating-SNAPSHOT.jar:?]
at org.apache.druid.segment.realtime.FireDepartment.<init>(FireDepartment.java:61) ~[druid-server-0.16.0-incubating-SNAPSHOT.jar:?]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:351) ~[druid-indexing-service-0.16.0-incubating-SNAPSHOT.jar:?]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:242) [druid-indexing-service-0.16.0-incubating-SNAPSHOT.jar:?]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:168) [druid-indexing-service-0.16.0-incubating-SNAPSHOT.jar:?]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:413) [druid-indexing-service-0.16.0-incubating-SNAPSHOT.jar:?]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:385) [druid-indexing-service-0.16.0-incubating-SNAPSHOT.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_272]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_272]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_272]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_272]
Caused by: java.lang.IllegalStateException: Failed to create directory within 10000 attempts (tried 1625648583973-0 to 1625648583973-9999)
at com.google.common.io.Files.createTempDir(Files.java:600) ~[guava-16.0.1.jar:?]
at org.apache.druid.segment.indexing.RealtimeTuningConfig.createNewBasePersistDirectory(RealtimeTuningConfig.java:61) ~[druid-server-0.16.0-incubating-SNAPSHOT.jar:?]
... 11 more
2、在Data Server上会生成OOM的文件,类似hs_err_pid2617.log
异常栈中也可以看出是Peon进程发生OOM。调整$DRUID_HOME/conf/druid/cluster/middleManager/jvm.config中-Xms1g -Xmx1g
2021-07-08 02:00:14 [main] ERROR CliPeon:123 - Error when starting up. Failing.
com.google.inject.ProvisionException: Unable to provision, see the following errors:
1) Error in custom provider, java.lang.OutOfMemoryError
at org.apache.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:117) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.DruidProcessingModule)
at org.apache.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:117) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.DruidProcessingModule)
while locating org.apache.druid.collections.NonBlockingPool<java.nio.ByteBuffer> annotated with @org.apache.druid.guice.annotations.Global()
for the 2nd parameter of org.apache.druid.query.groupby.GroupByQueryEngine.<init>(GroupByQueryEngine.java:82)
at org.apache.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:88) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.QueryRunnerFactoryModule)
while locating org.apache.druid.query.groupby.GroupByQueryEngine
for the 2nd parameter of org.apache.druid.query.groupby.strategy.GroupByStrategyV1.<init>(GroupByStrategyV1.java:76)
while locating org.apache.druid.query.groupby.strategy.GroupByStrategyV1
for the 2nd parameter of org.apache.druid.query.groupby.strategy.GroupByStrategySelector.<init>(GroupByStrategySelector.java:44)
while locating org.apache.druid.query.groupby.strategy.GroupByStrategySelector
for the 1st parameter of org.apache.druid.query.groupby.GroupByQueryQueryToolChest.<init>(GroupByQueryQueryToolChest.java:108)
at org.apache.druid.guice.QueryToolChestModule.configure(QueryToolChestModule.java:101) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.QueryRunnerFactoryModule)
while locating org.apache.druid.query.groupby.GroupByQueryQueryToolChest
while locating org.apache.druid.query.QueryToolChest annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=76, type=MAPBINDER, keyType=java.lang.Class<? extends org.apache.druid.query.Query>)
at org.apache.druid.guice.DruidBinders.queryToolChestBinder(DruidBinders.java:47) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.guice.QueryRunnerFactoryModule -> com.google.inject.multibindings.MapBinder$RealMapBinder)
while locating java.util.Map<java.lang.Class<? extends org.apache.druid.query.Query>, org.apache.druid.query.QueryToolChest>
for the 1st parameter of org.apache.druid.query.MapQueryToolChestWarehouse.<init>(MapQueryToolChestWarehouse.java:35)
while locating org.apache.druid.query.MapQueryToolChestWarehouse
while locating org.apache.druid.query.QueryToolChestWarehouse
for the 1st parameter of org.apache.druid.server.QueryLifecycleFactory.<init>(QueryLifecycleFactory.java:52)
at org.apache.druid.server.QueryLifecycleFactory.class(QueryLifecycleFactory.java:52)
while locating org.apache.druid.server.QueryLifecycleFactory
for the 1st parameter of org.apache.druid.server.QueryResource.<init>(QueryResource.java:113)
at org.apache.druid.server.QueryResource.class(QueryResource.java:78)
while locating org.apache.druid.server.QueryResource
Caused by: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at org.apache.druid.offheap.OffheapBufferGenerator.get(OffheapBufferGenerator.java:52)
at org.apache.druid.offheap.OffheapBufferGenerator.get(OffheapBufferGenerator.java:28)
at org.apache.druid.collections.StupidPool.makeObjectWithHandler(StupidPool.java:116)
at org.apache.druid.collections.StupidPool.<init>(StupidPool.java:87)
at org.apache.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:122)
at org.apache.druid.guice.DruidProcessingModule$$FastClassByGuice$$5c8d684e.invoke(<generated>)
at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401)
at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:104)
at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:104)
at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:104)
at com.google.inject.multibindings.MapBinder$RealMapBinder$ValueProvider.get(MapBinder.java:821)
at com.google.inject.multibindings.MapBinder$RealMapBinder$RealMapProvider.get(MapBinder.java:605)
at com.google.inject.multibindings.MapBinder$RealMapBinder$RealMapProvider.get(MapBinder.java:586)
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:53)
at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:45)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015)
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1050)
at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:138)
at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:106)
at org.apache.druid.cli.CliPeon.run(CliPeon.java:350)
at org.apache.druid.cli.Main.main(Main.java:112)