Upgrade failed and unable to start Elasticsearch

Hi,

We ran an elasticsearch pod with kubernetes on version 0.7 of OpenDistro and tried to upgrade to 1.2.0 yet somehow the process failed and i tried to revert back simply by using version 0.7 again. However we’re seeing this in the logs of Elasticsearch and are unable to start ES:

[    2019-09-23T06:21:18,137][ERROR][o.e.g.GatewayMetaState   ] [test-es01] failed to read or upgrade local state, exiting...
    java.io.IOException: failed to find global metadata [generation: 114]
    	at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:148) [elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:102) [elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.node.Node.<init>(Node.java:473) [elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.node.Node.<init>(Node.java:251) [elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) [elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) [elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.2.0.jar:7.2.0]
    	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.2.0.jar:7.2.0]
    [2019-09-23T06:21:18,146][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [brasve-es01] uncaught exception in thread [main]
    org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: IOException[failed to find global metadata [generation: 114]];
    	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.2.0.jar:7.2.0]
    	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.2.0.jar:7.2.0]
    Caused by: org.elasticsearch.ElasticsearchException: failed to bind service
    	at org.elasticsearch.node.Node.<init>(Node.java:580) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]
    	... 6 more
    Caused by: java.io.IOException: failed to find global metadata [generation: 114]
    	at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:86) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:148) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:102) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.node.Node.<init>(Node.java:473) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.node.Node.<init>(Node.java:251) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.2.0.jar:7.2.0]
    	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.2.0.jar:7.2.0]
    	... 6 more

Is the only way to start over again using a new Data directory? Unfortunately we have no snapshots.

Apologies for the delay. Please download the latest version of Open Distro for Elasticsearch. In case you are still facing this issue, please raise a Github Issue.