Convergence of opensearch & opendistro plugins

I’d like to just add that the concept of uninstalling plugins rather than providing a plugin-free artifact is not forwards compatible unless we add an uninstall-all-plugins command. Otherwise, for each new OpenSearch version we would have to account for the additional new plugins to uninstall in our scripts.

1 Like

Just throwing thoughts. Maybe the approach of disabling plugins vs uninstalling may work. So if there is no need for specific plugin, we can add a configuration option to disable the plugin. Like for example there was with x-pack related plugins.

1 Like

Super interesting thoughts and valid perspectives here.

Just a quick note that ES already ships w/ a loaded plugins directory (e.g., analysis-nori, analysis-stempel, repository-azure, store-smb). So I’m curious what percentage of folks are actually using all of those plugins, what percentage of plugins are being used, if there are folks that just ignore them or uninstall them, how you view them (required core plugins even though they’re actually optional), etc… The monolithic approach doesn’t scale well, the separate repository approach suffers from “bundle, install, remove” concerns, offering multiple separate archive downloads can be very confusing. So I don’t think there’s a silver bullet here and I think any step away from a monolithic bundle is a step in the right direction; so much so that I would really like to see the entire codebase move to a VERY microservice like approach - all the way down to extracting the aggregation framework (and their ValueSource support classes) out of server into a separate module.

1 Like

We use a few of the plugins from the plugins directory in ES (such as the repository-s3 plugin). The others don’t need to be uninstalled because they’re not installed yet (they are just idle in the plugins directory).

+1 for this

1 Like

Otherwise, for each new OpenSearch version we would have to account for the additional new plugins to uninstall in our scripts.

Exactly. And yes, it is important here to note the difference between “install” and “remove”. This is what we’re discussing here and the number one difference between plugins and modules. Modules are required (installed), plugins are not. Here we’re only talking about plugins (not modules) and how folks would like to handle them. OpenSearch has inherited a monolithic repository approach where all predecessor OSS plugin source lives in the plugins directory and distributed as part of all “flavors”. For ES, X-Pack lived in a separate monolithic directory and was included in all distribution flavors except oss. Now OpenSerach has the added ODFE plugins, which follow the “microservice” approach living in separate repositories. The idea for OpenSearch was to only provide a single “tgz/zip” artifact on the download page that includes ALL plugins in the plugin directory, but those plugins are still optional and only installed explicitly. In that regard, is there a reason to even provide a -minimal artifact? Just delete the source if you don’t want it on the filesystem? This minimizes the clutter (and possible confusion) on the download page, provides all users/customers with everything with the ability to pick and choose what they want to install instead of having to go to another location and download multiple artifacts.

2 Likes

There is a separate user/customer case: the third party plugin developer. DistributionDownloader is used here; buried under the covers of the opensearchplugin gradle extension. Today, the third party plugin developer extends opensearchplugin which has the burden of downloading the “tgz/zip” artifact (based on desired “flavor”). In this use case, the artifacts mattered. The suggested change here is to publish the released jars on maven central, and nightly build jars (unstable) in a separate opensearch provided snapshot repository. This way third-party plugin developers can write the plugin and only include the desired dependencies in the defined dependencies block of build.gradle. In this use case, there would also be no need to provide a “minimal” download on the opensearch website.

2 Likes

I agree 100% as this maintains the current behavior + as you said, minimizes confusion when faced with several artifacts to use.

Here’s a link to the yaml file with the elasticsearch / Kibana build details. We also have some documentation on how the ELK stack is used in Antrea. Keep in mind that this is just one example of our many products that incorporate these technologies in various ways.