1.ES大量做FULL GC，日志如下：

[2016-12-15 14:53:21,496][WARN ][monitor.jvm ] [vsp4] [gc][old][94725][4389] duration [26.9s], collections [1]/[27s], total [26.9s]/[15.9h], memory [19.7gb]->[17gb]/[19.8gb], all_pools {[young] [1.1gb]->[43.1mb]/[1.1gb]}{[survivor] [130.2mb]->[0b]/[149.7mb]}{[old] [18.5gb]->[16.9gb]/[18.5gb]}
[2016-12-15 14:53:57,117][WARN ][monitor.jvm ] [vsp4] [gc][old][94731][4390] duration [29.9s], collections [1]/[30.4s], total [29.9s]/[15.9h], memory [18.6gb]->[18gb]/[19.8gb], all_pools {[young] [71.1mb]->[51.8mb]/[1.1gb]}{[survivor] [149.7mb]->[0b]/[149.7mb]}{[old] [18.4gb]->[18gb]/[18.5gb]}
[2016-12-15 14:54:31,246][WARN ][monitor.jvm ] [vsp4] [gc][old][94735][4391] duration [30.6s], collections [1]/[31.1s], total [30.6s]/[15.9h], memory [18.5gb]->[17.9gb]/[19.8gb], all_pools {[young] [14.3mb]->[1.3mb]/[1.1gb]}{[survivor] [22.1mb]->[0b]/[149.7mb]}{[old] [18.4gb]->[17.9gb]/[18.5gb]}

ES内存配置策略有2点：

1.不超过可用内存的50%

2.不超过32G

fielddata加载数据到内存是按index来的，不会只加载检索结果数据，indices.fielddata.cache.size（5gb or 20%）控制fielddata可用内存，内存不够时，淘汰老数据，ES默认不淘汰。设置该值并不好，这样内存不够时每次会从磁盘读取，引起大量磁盘I/O，但如果想要ES只缓存最近的数据到内存，需要配置。

 监控fielddata

GET /_nodes/stats/indices/fielddata?level=indices&fields=*

By setting ?fields=*, the memory usage is broken down for each field.

fielddata circuit breaker可以在fielddata加载到内存前预估内存是否够用，如果内存不够用而继续读取fielddata到内存会导致内存溢出

连接：https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html

2. 检索报错

错误日志如下：

Failed to execute phase [query], all shards failed; shardFailures {[7l4w6bMqTReFs68KxMe1LA][smart_metadata-2015010100-2015010800][0]: RemoteTransportException[[vsp4][10.17.139.128:9300][indices:data/read/search[phase/query]]]; nested: EsRejectedExecutionExce        ption[rejected execution of org.elasticsearch.transport.TransportService$4@3b702529 on EsThreadPoolExecutor[search, queue capacity = 10000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6a6693a8[Running, pool size = 37, active threads = 37, queue        d tasks = 10000, completed tasks = 269010]]]; }

解决办法：

修改elasticsearch.yml

threadpool.bulk.type: fixed
Ÿthreadpool.bulk.size: 120
threadpool.bulk.queue_size: -1
Ÿthreadpool.search.queue_size: -1

EsRejectedExecutionException in elasticsearch for parallel search

Answer1:

Elasticsearch has a thread pool and a queue for search per node. A thread pool will have N number of workers ready to handle the requests. When a request comes and if a worker is free , this is handled by the worker. Now by default the number of workers is equal to the number of cores on that CPU. When the workers are full and there are more search requests , the request will go to queue. The size of queue is also limited. Its by default size is say 100 and if there happens more parallel requests than this , then those requests would be rejected as you can see in the error log.

The solution to this would be to -

1. Increase the size of queue or threadpool - The immediate solution for this would be to increase the size of the search queue. We can also increase the size of threadpool , but then that might badly effect the performance of individual queries. So increasing the queue might be a good idea. But then remember that this queue is memory residential and increasing the queue size too much can result in Out Of Memory issues. You can get more info on the samehere.
2. Increase number of nodes and replicas - Remember each node has its own search threadpool/queue. Also search can happen on primary shard OR replica.

Answer2:

Maybe it sounds strange, but you need to lower the parallel searches count. With that exception, Elasticsearch tells you that you are overloading it. There are some limits (at thread count level) that are set in Elasticsearch and, most of the times, the defaults for these limits are the best option. So, if you are testing your cluster to see how much load it can hold, this would be an indicator that some limits have been reached.

Alternatively, if you really want to change the default you can try increasing the queue size for searches to accommodate the concurrency demands, but keep in mind that the larger the queue size, the more pressure you put on your cluster that, in the end, will cause instability.

A node holds several thread pools in order to improve how threads memory consumption are managed within a node. Many of these pools also have queues associated with them, which allow pending requests to be held instead of discarded.

There are several thread pools, but the important ones include:

Changing a specific thread pool can be done by setting its type-specific parameters; for example, changing the index thread pool to have more threads:

thread_pool:

    index:

        size: 30