We got a stack running in containers for some time but a new bug appear after an host reboot today.
We have 12 elastic Node on 3 Host in containers :
Each servers has 2 Data node, 1 Ingest and 1 Master eligible.
We rebooted an host today to do some test (not elastic related) and when we closed it, our other master node crash with a memory issue…
When we started it back up, nothing can be done, as soon as one elastic master node is elected as master, we see in a few second this message appear :
[2021-04-06T21:16:57,258][WARN ][o.e.i.b.in_flight_requests] [ODE-MASTER-002] [in_flight_requests] New used memory 32764433992 [30.5gb] for data of [<http_request>] would be larger than configured breaker: 17179869184 [16gb], breaking
In less than 30 sec it pass from 0 to twice the heap value.
If we stop the ingest node, everythings run smoothly. As soon as we start one, the memory issue occur and less than a minute after the master node goes OOM again.
What can an ingest node do to fill the master node heap this quickly?
How can I debug this?
We are currently running 1.13.1 and it’s been running fine for the last 2 weeks.