"flattened" type with OpenDistro

Hi,
we tried to install OpenDistro while preserving the data of our non-OSS Elasticsearch single-node system.

Unfortunately roughly half of our indices are in a “red” state, because Elasticsearch-OSS does not understand the “flattened” Type:
org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [local_metadata]

The affected indices are created by Kibana and Filebeat, but we would like to keep the Kibana configuration and filebeat logs.

Is there any way to preserve existing data from a non-OSS Elasticsearch (with indices with “flattened” type) and still use OpenDistro?
Or do we have to convert/delete those fields?
If we have to convert, what is best-practice do achieve this?

Thanks a lot and best regards

hey ssc!

The flattened field type is a proprietary / x-pack feature, so it’s not supported by open distro. WRT Filebeat, there are two distinct versions - one is open source and compatible with Open Distro - IIRC, it won’t have any incompatible data types.

@ssc the fast solution to resolve your cluster problem is to delete the indices that have the flattened field (I guess that are .kibana with version 7.10.x)

If you don’t want delete the indices, but start your cluster, you can copy the mapper-flattened module from a standard Elasticsearch with XPack Distro to your plugin distro in OpenDistro. I checked the code mapper-flattened and the flattened plugin is completely independent from other XPack modules. It can live alone.
You cannot disable flattened mapping in XPack, because it’s internally used by Kibana.

I don’t want think bad, but it seems a trick to prevent the migration to OD from Elastic.

If I have some time this weekend, I’ll create a flattened mapper plugin that simply doesn’t index the records o replace XPack one so that people can easy upgrade to OpenDistro.

2 Likes

FYI Feature Request - Provide alternative to datatype flattened · Issue #523 · opendistro-for-elasticsearch/opendistro-build · GitHub

In my repo there is working implementation: https://github.com/aparo/opendistro-flattened-mapper-plugin.
@searchymcsearchface if you want I can release also a binary version.

That is awesome! I’m going to circulate this with some folks who might be interested. Expect some feedback and questions!

I updated te code to use opensearch GitHub - aparo/opensearch-flattened-mapper-plugin: OpenSearch Flattened Mapper Plugin to be able to load Kibana indices for ES 7.10.x or above in OpenSearch
Today I jave no time to test against a opensearch version, I’ll do tomorrow. If the OS code is the same of ES there should be no issues with the new version.

Looking forward to it and knowing your experience porting a plugin to OpenSearch.

The migration was very fast using a perl script to rename the packages:

#!/bin/bash

find . -type f \( -iname \*.scala -o -iname \*.sbt \)  -exec perl -pi -e '
  undef $/;
  use Regexp::Common;
  $x= "(){"."}";
  $bp = "(?:$RE{balanced}{-parens=>$x})";
  $arg = "(?:[^()}{,]*$bp?)+";
  s/elasticsearch/opensearch/g;

' {} +

chcase -rd -x 's/elasticsearch/opensearch/g' .

1 Like