Read_only action not working

Plugins Index Management
1

I am using AWS ES and created a policy to transit any indices older than 2 days to read_only. The indices are transited to “cold” however, the cold indices are NOT read_only. Is it an known issue? or something wrong with my setting?

Many thanks for any comments

Policy

{
    "policy_id": "2days_hot_90days_cold_delete",
    "description": "2 days hot, delete after 90 days",
    "last_updated_time": 1610582253323,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
        {
            "name": "hot",
            "actions": [],
            "transitions": [
                {
                    "state_name": "cold",
                    "conditions": {
                        "min_index_age": "2d"
                    }
                }
            ]
        },
        {
            "name": "cold",
            "actions": [
                {
                    "read_only": {}
                }
            ],
            "transitions": [
                {
                    "state_name": "delete",
                    "conditions": {
                        "min_index_age": "90d"
                    }
                }
            ]
        },
        {
            "name": "delete",
            "actions": [
                {
                    "delete": {}
                }
            ],
            "transitions": []
        }
    ]
}

Index setting after transition to “cold”, and I did successfully POST a new doc to this read_only index.

{
  "metrics-2021-01-12" : {
    "settings" : {
      "index" : {
        "opendistro" : {
          "index_state_management" : {
            "policy_id" : "2days_hot_90days_cold_delete"
          }
        },
        "number_of_shards" : "5",
        "blocks" : {
          "read_only_allow_delete" : "false",
          "write" : "false"
        },
        "provided_name" : "metrics-2021-01-12",
        "creation_date" : "1610469113420",
        "number_of_replicas" : "1",
        "uuid" : "qNtIuQtMRI-Ust_Zk9ZbzQ",
        "version" : {
          "created" : "7090199"
        }
      }
    }
  }
}

opendistro ISM history log

{
        "_index" : ".opendistro-ism-managed-index-history-2021.01.14-000002",
        "_type" : "_doc",
        "_id" : "73viAXcBAAPyT6nCrxyd",
        "_score" : 1.0,
        "_source" : {
          "managed_index_meta_data" : {
            "index" : "metrics-2021-01-12",
            "index_uuid" : "qNtIuQtMRI-Ust_Zk9ZbzQ",
            "policy_id" : "2days_hot_90days_cold_delete",
            "policy_seq_no" : 3,
            "policy_primary_term" : 1,
            "state" : {
              "name" : "cold",
              "start_time" : 1610644368947
            },
            "action" : {
              "name" : "read_only",
              "start_time" : 1610644368947,
              "index" : 0,
              "failed" : false,
              "consumed_retries" : 0,
              "last_retry_time" : 0
            },
            "retry_info" : {
              "failed" : false,
              "consumed_retries" : 0
            },
            "info" : {
              "message" : "Successfully set index to read-only [index=metrics-2021-01-12]"
            },
            "history_timestamp" : 1610644369309
          }
        }
      }
2

Hi @portswigger,

Could you open a support case? We’ll be able to look into the issue on your domain there.

3

Do you mean an AWS support case? or …

Thanks

4

Update from AWS support, so it looks like they have issue indeed

Dear Customer,

Thank you for contacting AWS Premium Support. This is Apurwa from the Amazon Elasticsearch team. I’ll be assisting you on this case today.

From the case correspondence, I understand that you implemented ISM policy to transition to read_only state for your index, but you were able to write to read_only index, even after the transition was completed.

To verify this at my end, I tried to reproduce this in my lab. I also observed the same behavior with similar ISM policy on my index. To further troubleshoot this behavior, I have reached out to my internal team. I would request you to allow me some time while I get back to you with a response from the internal team. Meanwhile, I’ll keep the case in “Pending Amazon Action” state.

Please accept my apologies for the inconvenience caused due to this issue.

Should you have further questions/concerns, let me know. I shall be glad to assist.

Thank you!

5

any updates from AWS?

6

Hey @portswigger and @jathin12,

Was able to confirm with support this is an issue. ISM is being impacted by another component on the service so there isn’t a fix we can add to ISM as of now. I am working with the other team to address the issue.

Thanks

8

Any news on this issue @dbbaughe ?

I am asking because this topic is 1 year old and we noticed today that we are also being affected by the same problem: in OpenSearch 1.1.0, the read_only stage sets the property index.blocks.write to false instead of true, so it’s not actually read_only.

edit: I just tried to replicate the issue in a docker image running openSearch 1.1.0 and there’s no problem there. So the issue is definitely with opensearch 1.1.0 service from aws