Can anyone please suggest me the best way to ingest the live log stream data (which was running and generating the new logs always) to OpenSearch.… Finally want to see the Dashboard/Visualization with different unique fields in OpenSearchDashboard…
I have found the below combinations can be used but need your help to choose the best…
Logstash
2.FileBeat
FileBeat + logstash
FileBeat + Kafka +logstash
Kafka + Fluentd
If possible please refer me to some best document to choose it…
Generally, there is no one-sized-fits-all approach, so I can only comment on higher-level issues.
The beat family (filebeat, etc.) should be considered a bit of a gamble in context of OpensSearch. I’m not personally sure of the future of that software (the creators have left Elastic and there newer product that fills the same nice) and there has been actions to box OpenSearch out of using in some contexts.
Logstash, like beats, has seen some actions to prevent connection to OpenSearch. However, there is a Logstash OpenSearch Output plugin that mitigates any connection issues, so I think it’s a safer bet than Beats. Fluentd is is used by a lot of folks and fills the same niche as Logstash. It’s more independent, so I wouldn’t expect to ever see any unresolvable issues with OpenSearch.
However, both Fluentd and Logstash are written in Ruby, a fine language but one not know for performance nor efficiency and that’s been a sticking point for a lot of folks.