Hi,
we use ODFE with a hot/warm/delete policy. The policy works fine for every rolled-over index. To avoid disk space issues we’d like to have a policy where the sum of all data stored in a set of rolled-over indices can be used to trigger a deletion of the oldest index.
Is there a possibility to configure such a behaviour?
Thanks in advance for an answer,
Hey @odfemdietz ,
Is something like this what you’re looking for (read full comments)?
opened 09:04PM - 27 Jul 20 UTC
enhancement
Problem:
Currently ISM allows you to transition an index to a new state based o… n the index's age. This works generally well for managing the overall size of a logging cluster. This works best when you have a consistent amount of volume that is generated at some interval, such as 100GB per day for example. In this case you can calculate the amount of time to keep an index around before deleting it with pretty good accuracy.
If you have a varying amount of volume, or it is largely unpredictable however, this becomes a problem. You cannot calculate with much certainty whether keeping an index around for 1, 5, 15, etc. days will make you eventually run out of space.
Solution Request
It would be great if you could define transitions based upon available space. If you are evenly distributing your data across nodes (which you should) then you could instead configure a policy to delete indices when there is less than 10%, 20%, etc. available space. This would have to delete indices in descending order from oldest to newest of course.
This would be especially beneficial to an organization like ours which has an annually seasonal usage pattern. If I define an ISM policy that keeps my disk usage at bay today, it may no longer work well a month from now when my usage and volume is higher.
This leads us to either have to over-provision our cluster for our heaviest usage or constantly monitor the cluster and keep adjusting the ISM policies.
An added bonus would be to be able to do this based on the storage space of different types of nodes. For example, we run a hot/warm workflow. Our warm nodes can hold far more data than the hot nodes. Thus, it would be optimal to be able to transition indices to warm once the hot nodes are getting full and then again to transition to delete once the warm nodes are getting full.
It’s not currently available, but if that’s what you’re looking for please +1 and/or reply so we can keep track of the requests for it.
Thanks
Hi @dbbaughe ,
thanks for your response. The issue sounds exact like what I’m looking for. Have subscribed and thumbed-up the solution request from @drock .
Cheers and keep well,