Unable to initialize .opendistro_security index

Hi,

I have two nodes, one for master and one for data.
My data node can join the master.

Some logs from my master node.

[2019-06-12T11:02:19,099][INFO ][o.e.c.s.ClusterApplierService] [10.49.112.19-hdata-node-0] detected_master {10.49.113.9-master-node-0}{B2z4lEzURoSNl0Qv2Hez7Q}{UZs5s6iHTYe4i3UpcwtIpw}{10.49.113.9}{10.49.113.9:9300}, added {{10.49.113.9-master-node-0}{B2z4lEzURoSNl0Qv2Hez7Q}{UZs5s6iHTYe4i3UpcwtIpw}{10.49.113.9}{10.49.113.9:9300},}, reason: apply cluster state (from master [master {10.49.113.9-master-node-0}{B2z4lEzURoSNl0Qv2Hez7Q}{UZs5s6iHTYe4i3UpcwtIpw}{10.49.113.9}{10.49.113.9:9300} committed version [17]])
[2019-06-12T11:02:19,189][INFO ][c.a.o.s.c.IndexBaseConfigurationRepository] [10.49.112.19-hdata-node-0] .opendistro_security index does not exist yet, so no need to load config on node startup. Use securityadmin to initialize cluster
[2019-06-12T11:02:19,204][INFO ][o.e.h.n.Netty4HttpServerTransport] [10.49.112.19-hdata-node-0] publish_address {10.49.112.19:9200}, bound_addresses {127.0.0.1:9200}, {10.49.112.19:9200}
[2019-06-12T11:02:19,205][INFO ][o.e.n.Node ] [10.49.112.19-hdata-node-0] started
[2019-06-12T11:02:19,205][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [10.49.112.19-hdata-node-0] 4 Open Distro Security modules loaded so far: [Module [type=MULTITENANCY, implementing class=com.amazon.opendistroforelasticsearch.security.configuration.PrivilegesInterceptorImpl], Module [type=DLSFLS, implementing class=com.amazon.opendistroforelasticsearch.security.configuration.OpenDistroSecurityFlsDlsIndexSearcherWrapper], Module [type=AUDITLOG, implementing class=com.amazon.opendistroforelasticsearch.security.auditlog.impl.AuditLogImpl], Module [type=REST_MANAGEMENT_API, implementing class=com.amazon.opendistroforelasticsearch.security.dlic.rest.api.OpenDistroSecurityRestApiActions]]

Then, I run securityadmin.sh to initialize the index. Mostly, it hangs on the waiting yellow cluster state step.

Open Distro Security Admin v6
Will connect to localhost:9300 … done
Elasticsearch Version: 6.7.1
Open Distro Security Version: 0.9.0.0
Connected as CN=…
Contacting elasticsearch cluster ‘compass-elasticsearch-preprod’ and wait for YELLOW clusterstate …

So, I tried to use -arc option.

Open Distro Security Admin v6
Will connect to localhost:9300 … done
Elasticsearch Version: 6.7.1
Open Distro Security Version: 0.9.0.0
Connected as …
Contacting elasticsearch cluster ‘compass-elasticsearch-preprod’ …
Clustername: compass-elasticsearch-preprod
Clusterstate: RED
Number of nodes: 2
Number of data nodes: 1
.opendistro_security index does not exists, attempt to create it … done (0-all replicas)
Populate config from /usr/share/elasticsearch/plugins/opendistro_security/securityconfig
Will update ‘security/config’ with ./securityconfig/config.yml
SUCC: Configuration for ‘config’ created or updated
Will update ‘security/roles’ with ./securityconfig/roles.yml
SUCC: Configuration for ‘roles’ created or updated
Will update ‘security/rolesmapping’ with ./securityconfig/roles_mapping.yml
SUCC: Configuration for ‘rolesmapping’ created or updated
Will update ‘security/internalusers’ with ./securityconfig/internal_users.yml
SUCC: Configuration for ‘internalusers’ created or updated
Will update ‘security/actiongroups’ with ./securityconfig/action_groups.yml
SUCC: Configuration for ‘actiongroups’ created or updated
Done with success

Logs in master node

[2019-06-12T11:07:55,993][INFO ][o.e.c.r.a.AllocationService] [10.49.113.9-master-node-0] updating number_of_replicas to [0] for indices [.opendistro_security]

[2019-06-12T11:07:56,094][INFO ][o.e.c.m.MetaDataMappingService] [10.49.113.9-master-node-0] [.opendistro_security/HlJdUu5vREe8Bn_HftLV_A] create_mapping [security]

[2019-06-12T11:07:56,148][INFO ][o.e.c.m.MetaDataMappingService] [10.49.113.9-master-node-0] [.opendistro_security/HlJdUu5vREe8Bn_HftLV_A] update_mapping [security]

[2019-06-12T11:07:56,190][INFO ][o.e.c.m.MetaDataMappingService] [10.49.113.9-master-node-0] [.opendistro_security/HlJdUu5vREe8Bn_HftLV_A] update_mapping [security]

[2019-06-12T11:07:56,224][INFO ][o.e.c.m.MetaDataMappingService] [10.49.113.9-master-node-0] [.opendistro_security/HlJdUu5vREe8Bn_HftLV_A] update_mapping [security]

[2019-06-12T11:07:56,258][INFO ][o.e.c.m.MetaDataMappingService] [10.49.113.9-master-node-0] [.opendistro_security/HlJdUu5vREe8Bn_HftLV_A] update_mapping [security]

[2019-06-12T11:07:57,709][ERROR][c.a.o.s.a.BackendRegistry] [10.49.113.9-master-node-0] Not yet initialized (you may need to run securityadmin)

It still staid Not yet initialized.

How do I fixed this issue?

I can make my cluster to be GREEN.

securityadmin.sh executed successfully.

Open Distro Security Admin v6

Will connect to localhost:9300 … done

Elasticsearch Version: 6.7.1

Open Distro Security Version: 0.9.0.0

Connected as …

Contacting elasticsearch cluster ‘compass-elasticsearch-preprod’ and wait for YELLOW clusterstate …

Clustername: compass-elasticsearch-preprod

Clusterstate: GREEN

Number of nodes: 2

Number of data nodes: 1

.opendistro_security index does not exists, attempt to create it … done (0-all replicas)

Populate config from /usr/share/elasticsearch/plugins/opendistro_security/securityconfig

Will update ‘security/config’ with ./securityconfig/config.yml

SUCC: Configuration for ‘config’ created or updated

Will update ‘security/roles’ with ./securityconfig/roles.yml

SUCC: Configuration for ‘roles’ created or updated

Will update ‘security/rolesmapping’ with ./securityconfig/roles_mapping.yml

SUCC: Configuration for ‘rolesmapping’ created or updated

Will update ‘security/internalusers’ with ./securityconfig/internal_users.yml

SUCC: Configuration for ‘internalusers’ created or updated

Will update ‘security/actiongroups’ with ./securityconfig/action_groups.yml

SUCC: Configuration for ‘actiongroups’ created or updated

Done with success

The logs still are Not yet initialized.

ES API always return Open Distro Security not initialized.

I found the logs in the data nodes.

[2019-06-12T11:22:19,542][WARN ][c.a.o.s.c.ConfigurationLoader] [10.49.112.19-hdata-node-0] No data for config while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups] (index=.opendistro_security)
[2019-06-12T11:22:19,542][WARN ][c.a.o.s.c.ConfigurationLoader] [10.49.112.19-hdata-node-0] No data for roles while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups] (index=.opendistro_security)
[2019-06-12T11:22:19,542][WARN ][c.a.o.s.c.ConfigurationLoader] [10.49.112.19-hdata-node-0] No data for rolesmapping while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups] (index=.opendistro_security)
[2019-06-12T11:22:19,542][WARN ][c.a.o.s.c.ConfigurationLoader] [10.49.112.19-hdata-node-0] No data for internalusers while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups] (index=.opendistro_security)
[2019-06-12T11:22:19,542][WARN ][c.a.o.s.c.ConfigurationLoader] [10.49.112.19-hdata-node-0] No data for actiongroups while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups] (index=.opendistro_security)

Is this relate to the issue?

It’s fixed. My bad :frowning:

I have not changed searchguard to opendistro_security in config.yml.