this is my first post, but i keep reading this forum for some time already to know the current state about Opensearch and friends.
I’m currently trying to setup elasticsearch+kibana as well as opensearch + dashboard at home (well i already have, that was the easy part) and i’m now playing with filebeat and metricbeat to get some data in.
I’m still doing a lot of testing and i’m still not 100% set whether elasticsearch/opensearch is what i need, and i hope someone can give me some feedback on whether things makes sense.
What i’m trying to achieve is to do some Log/Metrics monitoring of my home IT. I have a lot of smaller orange-pis, some debian machines + VMs + docker for a lot of services. Keeping all of them uptodate and analyse why certain things go wrong is a big task and right now i’m using icinga for the monitoring part, but i would like to change this, as i want to store the data somewhere and also store the logs in a central location where i can also play with some anomaly detection and centrally managed alarm system.
I know that i can also keep icinga and just forward output, but that feels making the system even more complicated than what it should be. I want to get things simpler (in terms of components involved).
Reading the blogs at logz.io (excellent blog posts btw) i know that using opensearch for logs and prometheus + grafana for metrics is a proposed way of doing things, but to be honest that involves too many components especially as i also need beats or logstash for also storing logs…
So right now i’m planning to also store metrics in opensearch and are currently testing this (still doing this with elasticsearch, but eager to switch).
The big question right now is, does all this make sense to you ? Am i on the right track ?
Specific topics i’m currently thinking about which is currently limiting me are the following:
Storing system metrics just from 2 machines for some days is already eating up quite some disk space. So i would like to do some retention on the metrics. I’ve setup ILM and using Rollup Jobs, but here is the culprit. When rolling up the data to 1d statistics and using a combination of raw_index + rollup_index for the Dashboard i’m limited to only seeing the aggregated data (1d), what i would like to have is to be able to show the current raw data when available (e.g. the 2 days), but only use the aggregated ones when browsing older data. To my understanding the elasticsearch version doesn’t support this.
In addition i also would like to have multiple rollups, similar to what influxdb or graphite are doing, keep raw data for last 2 days, use 1h aggregation for 1 week, use 1day aggregation for the rest of the month etc.
Creating rollup jobs for this is not the problem (just a bit tedious, as just the time histogram changes), but in the end i can just combine a raw_index + one rollup_index in the index pattern.
I couldn’t find any of those limitations in the docs for opendistro and also not for opensearch. So the main question is, does it have the same limitation than the elasticsearch version ?
If yes, how is this problem usually solved at an enterprise level ?