Unable to migrate from 0.9.0 to OpenDistro 1.2.0

Hi,

I am running Opendistro 0.9.0 in Docker, and now I want to upgrade to 1.2.0.

In the container is mounted:

  • xxx:/usr/share/elasticsearch/data"
  • xxx:/usr/share/elasticsearch/config/root-ca.pem"
  • xxx:/usr/share/elasticsearch/config/node.pem"
  • xxx:/usr/share/elasticsearch/config/node-key.pem"
  • xxx:/usr/share/elasticsearch/config/admin.pem"
  • xxx:/usr/share/elasticsearch/config/admin-key.pem"
  • xxx:/usr/share/elasticsearch/config/elasticsearch.yml"
  • xxx:/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml"
  • xxx:/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/config.yml"

I am following the steps for migrating the security index detailed here:

  1. I run a backup in the 0.9.0 container:

    plugins/opendistro_security/tools/securityadmin.sh -r -cd /backup_elastic6 -icl -nhnv -cacert config/root-ca.pem -cert config/admin.pem -key config/admin-key.pem
    
  2. I copy the directory created outside the container with docker cp

  3. I change the data inside elasticsearch.yml, internal_users.yml and config.yml to match the 1.2.0 version

  4. I start the image with the tag 1.2.0 with the new files

  5. Copy the directory created in 1 inside the container with docker cp

  6. After Elasticsearch and Kibana are ready I migrate the data with securityadmin.sh, and I get the following error:

plugins/opendistro_security/tools/securityadmin.sh -migrate /backup_elastic6 -icl -nhnv -cacert config/root-ca.pem -cert config/admin.pem -key config/admin-key.pem
Open Distro Security Admin v7
[...]
.opendistro_security index already exists, so we do not need to create one.
Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!
== Migration started ==
=======================
-> Backup current configuration to /backup-elastic6
Will retrieve 'security/config' into /backup-elastic6/config.yml (legacy mode)
   SUCC: Configuration for 'config' stored in /backup-elastic6/config.yml
Will retrieve 'security/roles' into /backup-elastic6/roles.yml (legacy mode)
   SUCC: Configuration for 'roles' stored in /backup-elastic6/roles.yml
Will retrieve 'security/rolesmapping' into /backup-elastic6/roles_mapping.yml (legacy mode)
   SUCC: Configuration for 'rolesmapping' stored in /backup-elastic6/roles_mapping.yml
Will retrieve 'security/internalusers' into /backup-elastic6/internal_users.yml (legacy mode)
   SUCC: Configuration for 'internalusers' stored in /backup-elastic6/internal_users.yml
Will retrieve 'security/actiongroups' into /backup-elastic6/action_groups.yml (legacy mode)
   SUCC: Configuration for 'actiongroups' stored in /backup-elastic6/action_groups.yml
  done
-> Migrate configuration to new format and store it here: /backup-elastic6/v7
  done
-> Delete old .opendistro_security index
Deleted index '.opendistro_security'  done
-> Upload new configuration into Elasticsearch cluster
Will update '_doc/config' with /backup-elastic6/v7/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '_doc/roles' with /backup-elastic6/v7/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '_doc/rolesmapping' with /backup-elastic6/v7/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '_doc/internalusers' with /backup-elastic6/v7/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '_doc/actiongroups' with /backup-elastic6/v7/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '_doc/tenants' with /backup-elastic6/v7/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
FAIL: 1 nodes reported failures. First failure is FailedNodeException[Failed node [IwyAdshWRDWJesl345z-bA]]; nested: RemoteTransportException[[elastic_service][172.17.0.4:9300][cluster:admin/opendistro_security/config/update[n]]]; nested: NotSerializableExceptionWrapper[static_resource_exception: Cannot override static roles];
FAIL: Expected 1 nodes to return response, but got 0
Done with failures
  ERR: unable to upload

I have found this issue reported here: https://github.com/opendistro-for-elasticsearch/security/issues/87.

I have tried to remove the standard roles in backup_elastic6/roles_2019_....yml and start again, but when I run the migrate command, it creates a file inside backup_elastic6/v7 with the standard roles, so the same error is raised.

If I ignore the error and try to restart the container, I get this error continuously:

[2019-11-26T16:35:07,371][ERROR][c.a.o.s.a.BackendRegistry] [elastic_service] Not yet initialized (you may need to run securityadmin)
[2019-11-26T16:35:08,017][ERROR][c.a.o.s.c.ConfigurationRepository] [elastic_service] com.amazon.opendistroforelasticsearch.security.securityconf.DynamicConfigFactory@2aa1f69d listener errored: StaticResourceException[Cannot override static roles]
com.amazon.opendistroforelasticsearch.security.configuration.StaticResourceException: Cannot override static roles
	at com.amazon.opendistroforelasticsearch.security.securityconf.DynamicConfigFactory.onChange(DynamicConfigFactory.java:130) ~[opendistro_security-1.2.0.0.jar:1.2.0.0]
	at com.amazon.opendistroforelasticsearch.security.configuration.ConfigurationRepository.notifyAboutChanges(ConfigurationRepository.java:308) [opendistro_security-1.2.0.0.jar:1.2.0.0]
	at com.amazon.opendistroforelasticsearch.security.configuration.ConfigurationRepository.reloadConfiguration0(ConfigurationRepository.java:297) [opendistro_security-1.2.0.0.jar:1.2.0.0]
	at com.amazon.opendistroforelasticsearch.security.configuration.ConfigurationRepository.reloadConfiguration(ConfigurationRepository.java:280) [opendistro_security-1.2.0.0.jar:1.2.0.0]
	at com.amazon.opendistroforelasticsearch.security.configuration.ConfigurationRepository$1.run(ConfigurationRepository.java:197) [opendistro_security-1.2.0.0.jar:1.2.0.0]
	at java.lang.Thread.run(Thread.java:834) [?:?]

I think I am missing something but I am lost.

If I run Opendistro 1.2 without migrations everything run correctly

If you have migrated the data before, can you tell me what are the differences? or how to proceed?

Thank you