Spring data integration

I use Spring Data Elasticsearch with an ELastic search instance. If I want to use OpenSearch, I don’t know if i can continue to use Spring Data Elasticsearch. Is there other solution to use OpenSearch in Spring in a higher level?
Thank you.

Welcome @giu85 - I moved your post to the OpenSearch category.

You should be able to use Spring Data with Opensearch. OpenSearch behaves like Elasticsearch 7.10.2, so if that is supported by Spring Data (which I think it is) then you’re fine. Let us know how it goes or if you run into any issues!

see also:

1 Like

A note before my comment: I am the project lead and main maintainer of Spring Data Elasticsearch at the moment.

Spring Data Elasticsearch (SDE) uses the client libraries from Elasticsearch (ES) to connect to an ES (or different) cluster. The last version of SDE that uses an ES client up to version 7.10.2 is SDE version 4.1.x that uses ES lib 7.9.3. SDE 4.2 already is on 7.12.1, the current main branch which will become SDE 4.3 uses ES 7.13.1.

It should be possible to use the SDE versions that use an 7.12 or 7.13 client with Opensearch (OS), if Opensearch is compatible with ES and as long as no functionality is used that was introduced after ES 7.10.

I had some short tests and for example OS does not support runtime fields in the index mapping, that was added in ES 7.11 and will be part of SDE 4.3. So although an application might be running without problems, but errors will come up when a feature is used in SDE that is not supported in OS.

As for the compatibility: One thing that SDE does is to retrieve the version of the cluster it is running against. To do this, we use the org.elasticsearch.client.core.MainResponse org.elasticsearch.client.RestHighLevelClient.info() call, that’s basically what a GET / returns.

On ES this will return

{
    "cluster_name": "docker-cluster",
    "cluster_uuid": "HoISicdkQxyWbYBwAPtE8g",
    "name": "33463b03a667",
    "tagline": "You Know, for Search",
    "version": {
        "build_date": "2021-04-20T20:56:39.040728659Z",
        "build_flavor": "default",
        "build_hash": "3186837139b9c6b6d23c3200870651f10d3343b7",
        "build_snapshot": false,
        "build_type": "docker",
        "lucene_version": "8.8.0",
        "minimum_index_compatibility_version": "6.0.0-beta1",
        "minimum_wire_compatibility_version": "6.8.0",
        "number": "7.12.1"
    }
}

On OS we get

{
  "name" : "b156c817d389",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "03q29RgHQwicLXU6pLeJvQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.0.0-rc1",
    "build_type" : "tar",
    "build_hash" : "26d579287f50bb33e17c8fe1f05ea208d5c64d1f",
    "build_date" : "2021-05-28T18:18:49.848386Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  }
}

Please note that OS does not send the tagline element. The tagline has always been a required field in the ES code that parses the answer into the MainResponse class. This parsing now fails, so that this call fails when accessing an OS instance.

I did not take the time to look for more incompatibilities, I could try to setup our tests to use an OS instance for the integration tests instead of an ES instance, but I can’t tell when I might find time for that.

So as we are implementing features that are missing from SDE it might be that these features are not available in OS.

As for future development, Elastic is currently working on a new client that will replace the RestHighLevelClient and that will be licensed with the Apache 2 license (GitHub - elastic/elasticsearch-java: Official Elasticsearch Java Client). When this is released, we will switch to use this client which is built on the Elasticsearch specification (GitHub - elastic/elasticsearch-specification: Elasticsearch full specification).

So as a summary: If OS is compatible with ES 7.10.2 it should not be a problem to use SDE (probably even with the next ES client) as long as no functions in SDE are used that use ES functionality added after 7.10.

I hope this clarifies the situation a bit.

2 Likes

@sothawo Wow - thanks for the very complete write up.

FYI - In 1.0 GA there has been a change merged regarding the version number (#708), so OpenSearch can report as 7.10.2.

With regards to the tagline field - it was removed as part of the debranding. Do you happen to know if the presence of the field is the only relevant bit or does the value also need to remain as it was?

Also - would you be open to contributions to help with OpenSearch compatibility?

As for what value is returned in the version: That is only used to create a warning log entry if the client library used and the cluster differ in at least the minor part, mostly to provide a hint if problems arise.

The tagline: I can only look in the code and there both in the actual client and the one for the next client I can only see that tagline is a required field, no reference to the value.

As for Opensearch compatibility: We will have to change the way of how SDE accesses the cluster anyway for the next version of the ElasticsearchClient since we cannot use the same classes to build requests and get responses as we do now. What comes to my mind would be to introduce something like a SearchEngineDriver interface and there could then be implementations for Elasticsearch and OpenSearch. SDE then would use a provided driver (that might be even be provided by SPI), if there is functionality that is not supported by the driver it could throw a corresponding exception - that’s just a first idea. Who would then write this and contribute - well SDE is a community driven project, so any contributions are welcome. As for me contributing on this part: As currently almost all work on SDE is done by me in my spare time, I don’t know how much I could do on the programming part besides setting up basic the architecture and providing this interface. I’ll check this with the Spring Data project lead what he thinks about such an approach.

2 Likes

Hi – we’ve opened [BUG] Elasticsearch RestHighLevelClient can not connect to OpenSearch server due to missing "tagline" field in Main Response · Issue #901 · opensearch-project/OpenSearch · GitHub to track the issue.

Thanks!
/C

1 Like

We’ve restored a tagline (Add 'tagline' back to MainResponse in server that was removed in PR #427 by tlfeng · Pull Request #913 · opensearch-project/OpenSearch · GitHub) so you should see it in the response for 1.0.0

Thanks,
/C

1 Like

:tada: Glad to see this!

[Although it’s functional yet super un-fun - I need to open an issue to “Add whimsy”]

Yeah, I was definitely at war with myself on this one :wink: But hey, we can always change it later :slight_smile: