Logstash | Elasticsearch output requires licensed cluster as of 7.13

As you can see on latest Logstash 7.13 release notes there’s this:

Feat: always check ES license #1005

Historically the output did not do a license check on OSS Logstash.
However with the recent ES licensing change (since 7.11) there’s no OSS ES artifact anymore.
Thus the distinction makes no sense.
Potential hiccups: ES versions older than 7.11.
The plugin would need to check the version first and skip the license check (under OSS) for ES < 7.11 for backwards compatibility.
This PR needs to ensure we are sending data to licensed ES.

As a result, Logstash always checks for license key when it connects to a cluster (GET /_license). Seems like a step towards guarding out OpenSearch? Thoughts?

2 Likes

@spapadop Yep. We’re aware. We have a few things in the works to counter this issue.

4 Likes

It will be very interesting to see how OpenSearch/Amazon responds to this. I get the concept of only forking ES/Kibana as that essentially worked out for them on ODFE with the plugins. I also understand that it would be more work to try to fork the beats & logstash as well. However, the move from Elastic seems inevitable, if not perhaps sooner than expected. From Elastic’s perspective, it makes total sense to try to push changes into the elements that the OpenSearch environment still uses but has no control over to drive incompatibility (Logstash/beats/whatever else??). I guess OpenSearch could just try to put data elements or make the API elements work ‘like’ how beats/logstash expects them too, but then they are constantly chasing that forever which seems like a losing battle - or at least making it hard on users who install/use OpenSearch. I am not sure what ways other than additional forks they have at their disposal, but I will be interested to see if they come up with a more inventive/less rework solution.

Hello kyle,
first at all I’ve to say I’m pretty disgusted in how Elastic is handing all this: despite their claims to support openness and competition they are behaving like the worst evil company out there.
Locking out users from logstash and/or beats, even if they are still on the Apache 2.0 license, is a dirty trick to try to stifle competition from OpenSearch.
Could you please give us some details about the countermeasures you and your colleagues at AWS are working on ? Is it a logstash output plugin for OpenSearch? either a beats output plugin for OpenSearch? something else ?.
In the company I work for we deeply rely on both LogStash and Beats and as such we need to evaluate an exit strategy as soon as possible (we’re already ready to move from ES to OpenSearch); to be perfectly clear, I think it would be GREAT to have a way to keep the code and know-how already developed without having to rewrite all from stratch due to “being forced” to move to fluentd or similar.

Thank you
Alessandro

1 Like

New & relevant to this conversation: https://docs-beta.opensearch.org/docs/agents-and-ingestion-tools/

1 Like

So once again question for you @searchymcsearchface why are we putting vendor-specific things into the docs for an open-source project in this manner? Are we going to cover ingestion into every other commercial offering as well? I would suggest updating AWS documentation and not putting that into OpenSearch docs to keep this focused on Open Source only. Please let me know if this makes sense.

1 Like

Can you comment as to what this additional documentation means with respect to beats/logstash’s function in OpenSearch? Unless I am misreading this, the document seems to infer that beats to OpenSearch is no longer supported and forces the use of logstash? I am also unclear how the logstash opensearch plugin will be distributed and as part of what package. Is this just a short-term response with a more complete statement in the next couple of weeks, or is this a statement that beats will no longer be supported in OpenSearch like it is in ES?

It’s a fair callout. Here is the reasoning behind it living in OpenSearch docs:

  • The OpenSearch output plugin will live under the OpenSearch org on GitHub, that plugin will be able to connect to a variety of flavours of ES and OpenSearch.
  • A single URL to help folks navigate the problem

WRT the other offerings, the docs are open source - I don’t think the docs team would be steadfastly opposed other services being included on this page through a PR. The matrix is for the benefit of the entire community, so, if Logz (for example) wanted to include an offering on this page, that could makes sense.

Also, if you disagree with the inclusion of other services, an issue on the docs is an option.

1 Like

I’m sure there will be additional documentation to come on beats/logstash, but let me just clarify a few points.

  1. No, you can still use Beats directly with OpenSearch. Version 7.13 has a license check now, so you can use 7.12 with OpenSearch.
  2. OpenSearch Logstash output plugin will be distributed separately. It’s typical for output plugins to be distributed separately from Logstash itself.

Thanks for the clarification. Since it appears that beats (as distributed by Elastic), past version 7.12 will not work with OpenSearch, will beats be forked? I am unsure how to ensure software is kept up-to-date security-wise if there is a requirement to lock at a specific version or even what Elastic’s maintenance window on prior second digit releases.

FYI, I’ve made a PR to restore support for Elasticsearch OSS distributions 7.0.x-7.10.2 in logstash-output-elasticsearch.

1 Like

@justme the team is still trying to determine what’s best to do about Beats.

Let me flip this a little - I’d be curious to know how you’d feel about other shippers (say, Fluent Bit) over Beats.

That is an interesting suggestion. In a prior task, I had some experience using beats->ES, which is why I was shooting for beats->OpenSearch. I was somewhat familiar with Fluentd being a potential replacement for logstash, but I am not too familiar with how fluentd would replace a beats, to be honest. I am trying to get another instance setup on a new project using opensearch from scratch (currently stuck Using PKI certificate for kibanaserver user: 'Tenant global_tenant is not allowed for this user' here, but hopefully someone has a suggestion on that one soon - it is a low-burn background project at work here right now). The initial plan was going to be beats->OpenSearch since they are all yum/dnf automatically updated (manually installed things are a royal pain / expensive to monitor for so we do everything we can to stick with a yum/dnf supported stack if we can at all). I will have to search for how fluentd can replace beats and get back with you, unless you have some pointers/links you can share.

To be clear, I was suggesting Fluentbit not Fluentd. They’re related but different. Beats is to Logstash what Fluent Bit is to Fluentd (roughly, but not exactly).

Sorry - I must have sounded pretty dumb there :man_facepalming:. I’ll definitely take a look at it. Looks like it supports pki authentication into es and has a yum/dnf repo. I appreciate the suggestion.