ISM Template Fields no longer supported

Plugins Index Management
1

Prior to Version 1.13.0, we were using the below template pattern and map it to the policy.

{
  "order" : 1,
  "version" : 60001,
  "index_patterns" : ["test-*"],
  "settings" : {
"opendistro.index_state_management.policy_id": "test-policy",
"index" : {
  "number_of_shards" : "1",
  "refresh_interval" : "30s"
}
  },
  "aliases" : {
"logs-cloud": {}
  },
  "mappings" : {
"_field_names": {
  "enabled": false
},
"dynamic_templates" : [
  {
    "message_field" : {
      "path_match" : "message",
      "mapping" : {
        "norms" : false,
        "type" : "text"
      },
      "match_mapping_type" : "string"
    }
  },
  {
    "string_fields" : {
      "mapping" : {
        "norms" : false,
        "type" : "text",
        "fields" : {
          "keyword" : {
            "ignore_above" : 256,
            "type" : "keyword"
          }
        }
      },
      "match_mapping_type" : "string",
      "match" : "*"
    }
  }
],
"properties" : {
  "@timestamp" : {
    "type" : "date_nanos"
  }
}
  }
}

So now that the templates are part of policy itself and do not reference it the older way, it does not support all these above mentioned fields which it used to do earlier.

We are getting errors like

{“error”:{“root_cause”:[{“type”:“illegal_argument_exception”,“reason”:“Invalid field: [order] found in ISMTemplate.”}],“type”:“illegal_argument_exception”,“reason”:“Invalid field: [order] found in ISMTemplate.”},“status”:400}

{“error”:{“root_cause”:[{“type”:“illegal_argument_exception”,“reason”:“Invalid field: [mappings] found in ISMTemplate.”}],“type”:“illegal_argument_exception”,“reason”:“Invalid field: [mappings] found in ISMTemplate.”},“status”:400}

2

Hi @Abb,

We only moved the usage of policy_id out of the index templates and into the ISM templates. You still use the index templates for all of the other functionality (such as the settings mentioned in the exception messages).

3

Hello @dbbaughe ,

So, this is how I tried binding the existing template under ism_template attribute

{
  "policy": {
    "description": "A policy that changes the states of test indexes",
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [...],
    "ism_template": {
      "order": 1,
      "version": 60001,
      "index_patterns": [
        "test-*"
      ],
      "settings": {
        "index": {
          "number_of_shards": "1",
          "refresh_interval": "30s"
        }
      },
      "aliases": {
        "logs-cloud": {}
      },
      "mappings": {
        "_field_names": {
          "enabled": false
        },
        "dynamic_templates": [
          {
            "message_field": {
              "path_match": "message",
              "mapping": {
                "norms": false,
                "type": "text"
              },
              "match_mapping_type": "string"
            }
          },
          {
            "string_fields": {
              "mapping": {
                "norms": false,
                "type": "text",
                "fields": {
                  "keyword": {
                    "ignore_above": 256,
                    "type": "keyword"
                  }
                }
              },
              "match_mapping_type": "string",
              "match": "*"
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date_nanos"
          }
        }
      }
    }
  }
}

And later tried to assign the policy via API

curl -XPUT --insecure "https://0.0.0.0:9200/_opendistro/_ism/policies/test-policy" -H 'Content-Type:application/json' --data-binary @"kubernetes/elasticsearch/config/policy/test-policy.json" -u username:"password"

then it throws below error as such

{“error”:{“root_cause”:[{“type”:“illegal_argument_exception”,“reason”:“Invalid field: [order] found in ISMTemplate.”}],“type”:“illegal_argument_exception”,“reason”:“Invalid field: [order] found in ISMTemplate.”},“status”:400}

it throws the same error for all fields (order, version, mappings, settings, aliases ) except for index_patterns.

The policy get applied successfully if and only if index_patterns field is present under ism_template object, Below is the reponse.

> {"_id":"test-policy","_version":1,"_primary_term":69,"_seq_no":1176,"policy":{"policy":{"policy_id":"test-policy","description":"A policy that changes the states of test indexes","last_updated_time":1618468680710,"schema_version":1,"error_notification":null,"default_state":"hot","states":[{"name":"hot","actions":[],"transitions":[{"state_name":"warm","conditions":{"min_index_age":"3m"}}]},{"name":"warm","actions":[{"read_only":{}}],"transitions":[{"state_name":"delete","conditions":{"min_index_age":"3m"}}]},{"name":"delete","actions":[{"delete":{}}],"transitions":[]}],"ism_template":{"index_patterns":["test-*"],"priority":0,"last_updated_time":1618468680709}}}}

`

4

Hey @Abb,

My previous message perhaps was confusing. I meant to say we removed the functionality/usage of policy_id from the index templates. The new way to use the functionality is to use the ISM template.

What that means is let’s say you originally had an index template that looked like this:

{
    "order" : 1,
    "version" : 60001,
    "index_patterns" : ["test-*"],
    "settings" : {
        "opendistro.index_state_management.policy_id": "test-policy",
        "index" : {
            "number_of_shards" : "1",
            "refresh_interval" : "30s"
        }
    },
    "aliases" : {
        ...
    },
    "mappings" : {
        ...
    }
}

So the new way to do this is simply remove the policy_id listed in the template so your new index template looks like:

{
    "order" : 1,
    "version" : 60001,
    "index_patterns" : ["test-*"],
    "settings" : {
        "index" : {
            "number_of_shards" : "1",
            "refresh_interval" : "30s"
        }
    },
    "aliases" : {
        ...
    },
    "mappings" : {
        ...
    }
}

And to regain the functionality of having that policy automatically applied to the test-* indices you instead update the test-policy policy to include this ISM template:

{
    "ism_template": {
        "index_patterns": ["test-*"]
    }
}

So now whenever a new index is created that matches the test-* pattern the index templates will take care of applying all your mappings, aliases, settings, etc. and the ism_template will take care of applying the policy to it.

5

@dbbaughe Okay, I tried it and it worked. Thanks for the info. Thanks to you