Invalid field: [ism_template] found in Policy

Attempting to follow resource Index State Management in Amazon Elasticsearch Service - Amazon Elasticsearch Service to add an ISM to transition my data from Hot to UltraWarm.

Policy can be created but struggling to add this policy to an index.

  • Attempt one was to create a template
_template/hot_warm_delete
{
  "index_patterns": [
    "log-collector*",
    "aws-log-collector*"
  ],
  "template": {
    "settings": {
      "opendistro.index_state_management.policy_id": "hot_cold_workflow"
    }
  }
}

This did not add any indices to be managed by the policy.

  • Attempt two was to create an ism_template as per policy below
{
    "policy": {
        "policy_id": "hot_cold_workflow",
        "description": "Demonstrate a hot-warm-delete workflow.",
        "last_updated_time": 1620283047106,
        "schema_version": 1,
        "error_notification": null,
        "default_state": "hot",
        "ism_template": {
            "index_patterns": [
                "log-collector*",
                "aws-log-collector*"
            ]
        },
        "states": [
            {
                "name": "hot",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "warm",
                        "conditions": {
                            "min_index_age": "10d"
                        }
                    }
                ]
            },
            {
                "name": "warm",
                "actions": [
                    {
                        "timeout": "24h",
                        "retry": {
                            "count": 5,
                            "backoff": "exponential",
                            "delay": "1h"
                        },
                        "warm_migration": {}
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "20d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ]
    }
}

which throws error:

[illegal_argument_exception] Invalid field: [ism_template] found in Policy.

This is being attempted on an AWS Managed ES Cluster in Ireland.
Service software release - R20210331
Elasticsearch version - 7.10

Please advise as I cannot currently add indices automatically which is harming the cluster stability

Hey @thameezbo,

The ism_template support is currently being rolled out and you should start seeing support of it on the service in the next 1-2 weeks. In the mean time in your index template example it seems like you have an extra “template” key wrapped around the settings?

I think AWS ES uses ODFE prior to 1.13.0 so you need to use policy_id instead of ism_template as per Index State Management - Open Distro for Elasticsearch Documentation

extract: “For ODFE versions prior to 1.13.0, include the policy_id in an index template so when an index is created that matches the index template pattern…”

Ah great catch.
Note that the template block was copied from the AWS Docs
Index State Management in Amazon Elasticsearch Service - Amazon Elasticsearch Service

I don’t suppose this is the place to submit a PR for their docs?

After removing the extra, template, polices are being added as expected.
Thank you

Copy copy, thank you.

Do you perhaps know how I can check the version of ODFE running?

Have submitted a feedback request to AWS to update.

Thanks again for the super swift response

Unfortunately not. No matter how hard I tried I couldn’t find anywhere a mapping between AWS ES releases and ODFE versions. The best I could do is to cross-check the features available here Document history for Amazon Elasticsearch Service - Amazon Elasticsearch Service against the opendistro releases features.

Pretty much the same conclusion I came to.

Thank you again for the assistance

1 Like

Fixed the AWS docs, sorry for the confusion.

As an update, AWS ES Service Software R20210426 enables ism_templates to be used in the managed policy.

Does not mention it anywhere in the release notes lol, but it works.

I added it to the release notes: Document history for Amazon Elasticsearch Service - Amazon Elasticsearch Service