Upgrade path to OpenSearch

I realize we don’t have a thread on the upgrade path to OpenSearch. For context, I am looking into various ODFE plugins and their potential upgrade path. What I’d like to hear about on this thread is what’s important to you for migration/upgrade?

We have ElasticSearch + Kibana (aka ES), Open Distro for Elasticsearch (OSS Elasticsearch+Kibana+OSS plugins, aka ODFE) migrating to OpenSearch + OpenSearch Dashboards + OpenSeach plugins (aka OpenSearch).

Some questions I can think of:

  • Should OpenSearch be 100% backwards compatible with ES?
  • Is it important to you that OpenSearch be a drop-in replacement for ES? For ODFE?
  • For completeness, would you be ok if there was no upgrade path at all, because maybe you only plan to do new deployments with OpenSearch?

Your thoughts and additional questions would be much appreciated.

Thanks,
-dB.

4 Likes

We use only Elasticsearch itself, no plugins (other than discovery-ec2 plugin) or dashboards, but we run tens of ES clusters so it’s quite important for us to be able to upgrade seamlessly to OpenSearch. We are currently running ES 6.8 and are testing ES 7.9 but we most likely skip directly to OpenSearch if the upgrade is reasonably cheap - ideally for free.

So to your questions:

  • 100% backward compatibility? - Yes please.
  • Drop in replacement? - Yes, we need it to be able to read ES 6 or 7 data structures. Pointing it to ES 6/7 data directory should just work.
  • No upgrade path at all? - Not in our case. We would have to reindex TBs of data during upgrade which is just not feasible.

But I think it’s OK if OpenSearch diverges in later versions and the upgrade path would be only ES 7 → OpenSearch 1.0 → OpenSearch 2.0 → OpenSearch X.0. The important thing is to allow current ES users to have painless upgrade process and once they are on OpenSearch you are in much better position to control upgrades to future versions.

4 Likes

Same as what Katulus said would be true for us at Sematext, too.

Otis

1 Like

At VMware, we’ve embedded Elasticsearch and Kibana into our products, and are considering replacing them with ES, so in answer to your questions … we need backwards compatibility, a drop-in replacement, and upgrade path. Our interest is mostly around ES, rather than the plugins.

1 Like

we have the same use-case, i.e. we (plan to) ship Elasticsearch and want to use OpenSearch as a drop-in replacement. we do need a security plugin and for a few use-cases maybe 1-2 additional plugins where we haven’t checked yet what their plans are regarding OpenSearch.
since we’re not running it ourselves in most cases it needs to be a smooth upgrade/replacement (we are running on kubernetes/openshift). obviously the nicest thing would be if i could do an online-upgrade of an Elasticsearch 7.10.x cluster to OpenSearch 1.0.0, but i’d also be ok with having a short down-time if the cluster wouldn’t be happy about that). picking up the existing data-files would be very helpful as it’d allow an easy transition without any index migrations / re-indexing.

I’d like to add a little to what @dblock is asking - for those of you needing seamless upgrades: what other tooling do you use with your current ES clusters or embedded environment?

Logstash? Beats? Fluentbit? Custom software?

custom software (ours & 3rd party) in our case (both for read & write).
note that most of them will be using spring-data-elasticsearch, so this must stay compatible. see also Spring Data for ElasticSearch support / collaboration · Issue #542 · opensearch-project/OpenSearch · GitHub for this.

1 Like

We use:

  • ODFE embedded into our products
  • ODFE “standalone” (usually installed in docker / Kubernetes)
  • Amazon Elasticsearch Service

We really want to migrate to OpenSearch once ready, but we cannot reindex all the data, so the index migration must be seamless. We can accept a minimum downtime.
We also need that OpenSearch/ODFE and Amazon Elasticsearch Service remain aligned, because we have the same products running on different clouds (using AES on AWS and ODFE elsewhere)

About the tools that we use:

1 Like

In Aiven we support:
OS ES + Kibana and additional open source plugins
Besides that we use our own plugins which support ZSTD compression and client-side encryption for S3, GCS and Azure

Thanks,
Andrey

1 Like

I’ve opened OpenSearch#638 to help us work backwards from the assumption of maximum and most seamless drop-in replacement. We have a dozen ODFE plugins to test out various strategies and I know different developers have different ideas about how we could accomplish this technically - let’s start putting them into there. Would love to hear from some plugin authors that are considering supporting OpenSearch, outside of ODFE, too.

Our most important requirement for upgrade is no outage. There should be an upgrade path allowing for upgrade without any outage, preferably while keeping the cluster in read/write mode. Read-only upgrade would be considered to be a last resort, but still feasible.

2 Likes

Thanks everyone for your feedback. It’s pretty clear that customers want a rolling upgrade without downtime. I opened Ensure OpenSearch is a drop-in replacement for ODFE 1.13 and ES 7.10.x for v1.0.0 · Issue #671 · opensearch-project/OpenSearch · GitHub and Ensure OpenSearch Dashboards v1.0.0 is a drop-in replacement for Kibana v7.10.x · Issue #334 · opensearch-project/OpenSearch-Dashboards · GitHub to track related work-items.