Elasticsearch cluster status in yellow due to unassigned shards for .opendistro_security index

Security
1

Hi there,

Recently, our AWS-managed Elasticsearch 7.4 went into Yellow status. After investigating with

GET _cluster/allocation/explain?pretty

We get

{
  "index" : ".opendistro_security",
  "shard" : 0,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "REPLICA_ADDED",
    "at" : "2021-12-11T00:47:42.406Z",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions" : [ ... ]
}

I tried the AWS proposed suggestions but seems that .opendistro_security index is protected and I am unable to follow their instructions. Is there any other way to resolve this unassigned shard issue?

Also, some node_allocation_decisions includes

"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[.opendistro_security][0], node[xxxx], [R], s[STARTED], a[id=xxxx]]"

and

"explanation" : "there are too many copies of the shard allocated to nodes with attribute [zone], there are [18] total configured shard copies for this shard id and [2] total attribute values, expected the allocated shard count per attribute [10] to be less than or equal to the upper bound of the required number of shards per attribute [9]"

Thanks.

2

Moved to the security category.

3

@ahmadalsajid as this is AWS managed service, I’m not sure if you have the necessary access to perform the below, however the steps would be to first disable the auto expand using admin certificate (in this case default kirk.pem and kirk-key.pem) with below command:

./securityadmin.sh -dra -icl -nhnv -cacert /usr/share/opensearch/config/root-ca.pem -cert /usr/share/opensearch/config/kirk.pem -key /usr/share/opensearch/config/kirk-key.pem

Then set the necessary replicas using command:

./securityadmin.sh -us {number of replicas needed} -icl -nhnv -cacert /usr/share/opensearch/config/root-ca.pem -cert /usr/share/opensearch/config/kirk.pem -key /usr/share/opensearch/config/kirk-key.pem

If you don’t have access to the script, you might need to raise a support ticket with AWS

4

Hi @Anthony,
Thanks for your quick suggestions. You are right. I had to contact AWS support and they solved it.