Restart of elasticsearch takes a long time shows - "Not yet initialized (you may need to run securityadmin)"

Security
1

I see about 20 mins of these log messages before it successfully starts up… until then elasticsearch is not reachable by logstash

I have a single node, 12GB jvm with about 1TB of data in about 400 indexes. With a 4GB JVM it used to take about 4 mins or of these error messages before it starts working.

Is there a systemic issue in my setup that is causing this issue at startup ?

[2021-02-11T15:51:26,177][ERROR][c.a.o.s.a.BackendRegistry] [wamapnap01] Not yet initialized (you may need to run securityadmin)
[2021-02-11T15:51:26,177][ERROR][c.a.o.s.a.BackendRegistry] [wamapnap01] Not yet initialized (you may need to run securityadmin)

[2021-02-11T15:51:26,661][INFO ][c.a.o.j.JobSchedulerPlugin] [wamapnap01] JobSweeper started listening to operations on index .opendistro-ism-config
[2021-02-11T15:51:26,871][ERROR][c.a.o.s.a.BackendRegistry] [wamapnap01] Not yet initialized (you may need to run securityadmin)
[2021-02-11T15:51:26,875][ERROR][c.a.o.s.a.BackendRegistry] [wamapnap01] Not yet initialized (you may need to run securityadmin)
[2021-02-11T15:51:26,986][INFO ][c.a.o.j.s.JobScheduler ] [wamapnap01] Scheduling job id -qcmmccbTiOTbdHGLKt4uQ for index .opendistro-ism-config .
[2021-02-11T15:51:26,988][INFO ][c.a.o.j.s.JobScheduler ] [wamapnap01] Scheduling job id 0WVfCBHzSXWDOmVydV1XHw for index .opendistro-ism-config .
[2021-02-11T15:51:26,988][INFO ][c.a.o.j.s.JobScheduler ] [wamapnap01] Scheduling job id 0xBhFD-NSVSclODPkrBdWg for index .opendistro-ism-config .

2

If you check /_cluster/health after the service restarts what does it say? It might just be taking time to recover the indexes into a green state.

3

Good point. i have few indexes in yellow due to replica count set to 1 and I only have one node. i will fix that replica count and try again.
Thanks!

{
“cluster_name” : “elasticsearch”,
“status” : “yellow”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“active_primary_shards” : 606,
“active_shards” : 606,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 206,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 74.63054187192118
}