Hi, we’re operating an ELK cluster on a single host. The current setup includes:
3 node ( 1 master-nodata, 2 master-data) containers
2 logstash containers (for app and for container logs)
2 kibana containers(1 for administrative purposes only)
I noticed that we get process rejections on write threadpool only on node 3 container. If I understood it correctly this happens when cpu cant handle the queue. We are using 2x E5 2620 intel xeon processors on this host. Each node has 16GB heap, each logstash has 4gb heap and there is also plenty of ram left after logstash and elasticsearch heaps.
I have more shards than i normally should have so i suppose that might be causing a problem on these nodes in particular in this subject. We changed our indexing system but it will take about a month until it settles. I’ve thought of a couple solutions but not sure how they will work so i’d like your knowledge on this.
- If i add 2 more data node containers to this cluster and keep the total shard count for each index at 2 will it make some room as it begins to spread the shards evenly to all nodes?
- Is it possible to increase the thread count for write threadpool? If yes where? I dont think increasing or decreasing queue size might effect this in a positive way. So I’d like to know if i can change thread count directly.
- When the new index structure settles as i mentioned above, will it affect this situation in a positive way?
Thanks in advance.