Catch 22 : downgrade to reindex : fails to start ES

Open Source Elasticsearch and Kibana
#1

Hi,
We were running 7.9.1 but after after a machine power outage, ES did not start due to
an index created at version 5.6. And recommends downgrading to 6.x and reindex.

So I replaced with dpkg -i elasticsearch-oss-6.8.13.deb and
now I get an error message in startup :
failed to read [id:116, file:/dynga/es-iou2/elasticsearch/nodes/0/_state/node-116.st

    [2020-11-04T10:41:59,031][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [iou2.uninett.no] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read [id:116, file:/dynga/es-iou2/elasticsearch/nodes/0/_state/node-116.st]]; nested: IOException[failed to read [id:116, file:/dynga/es-iou2/elasticsearch/nodes/0/_state/node-116.st]]; nested: XContentParseException[[-1:36] [node_meta_data] unknown field [node_version], parser not found];

should I also downgrade some or all of the suppor packages to do the reindexing necessary ?

    opendistro-alerting opendistro-anomaly-detection opendistro-index-management opendistro-job-scheduler opendistro-knn
  opendistro-performance-analyzer opendistro-security opendistro-sql opendistroforelasticsearch
#2

I turns out that after trying a couple of other version og elasticsearch-oss the 7.9.1 version eventually accepted the index and came up. Case dismissed !

#3

After a reboot, the problem is back. No change in sw the last weeks. So how can ES come in such a state that it suddenly don’t acet indexes that has been up and running after a restart. I noticed there is a field version created on and one for the current. Could be that the startup code makes wrong guesses ?

We have both newer and older indices. How can we get our data back ?

#4

Can you show the message where it recommends downgrading?

I am wondering if perhaps the message is incorrect and the problem really has nothing to do with downgrading. Perhaps you just have some indexes that got unrecoverably corrupted during the power failure

#5

java.lang.IllegalStateException: The index [[uninett6/9HwC3NFKT7m0Ut7SK6M7Qw]] was created with version [5.6.10] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.9.1.

#6

Did you ever actually run the version 5.6.10 like it claims?

#7

Yes that may have been the case we sstartet at about 4 a few years ago.

#8

(I cooperate with okvittem on this sometime “grumpy” ES-cluster.)

The ES cluster in question is a single node cluster. It seems to have a folder in its data-path for both a node 0 and a node 1. Folders and files for all operational indices are located under node 0. Some other legacy/zombie indices exists under node 1. Occasionally, after a reboot and restart, ES seems to discover these legacy indices and starts complaining about too old index versions. Removing node 1 files made ES start. However ES then creates a new node 1 folder hierarchy (with no indices).

… so it may seem that the question is: Why does our system insist on having a node 1 data folder in addition to the operational node 0 ?