Index_management_exception when setting ism_template

Plugins Index Management
1

Prior to version 1.13.0, we were attaching ISM policies to indices via the template api like below.

PUT /_template/some-index
{
      "index_patterns": "example*",
      "order": order,
      "settings": {
          "index": {
            "number_of_shards": shards,
            "number_of_replicas": replicas
           "opendistro.index_state_management.policy_id": ism_policy
          }
        }

After version 1.13.0, opendistro.index_state_management.policy_id is deprecated, and we had to use ism_template like in the code snippet to achieve the same behaviour.

PUT _opendistro/_ism/policies/ism_policy
{
  "policy": {
    "description": "Delete index after 14d",
    "default_state": "search",
    "states": [
      {
        "name": "search",
        "actions": [],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "14d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": {
      "index_patterns": ["example*"]
    }
  }
}

However, we are faced with this error when running the above command in the kibana dev tools. We tried using different priority numbers and it randomly works and errors out. Not sure why this is happening.

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_management_exception",
        "reason" : "New policy short has an ISM template with index pattern [example*] matching existing policy templates, please use a different priority than 44"
      }
    ],
    "type" : "index_management_exception",
    "reason" : "New policy short has an ISM template with index pattern  [example*]  matching existing policy templates, please use a different priority than 44",
    "caused_by" : {
      "type" : "exception",
      "reason" : "java.lang.IllegalArgumentException: New policy short has an ISM template with index pattern  [example*]  matching existing policy templates, please use a different priority than 44"
    }
  },
  "status" : 400
}
2

[Welcome @lajansa - I moved your question to a better category]

3

Hey @lajansa,

Since we have moved the index matching logic onto the policies themselves in the ism_template field we need a way to know which policy to choose if there are conflicting templates. That’s why there is a priority field; we use this to deal with conflicts. We also explicitly reject new policies that match another policies index pattern and has the same priority because in this case we won’t know which policy you meant for us to apply to the index. Based on that error I am assuming you have another policy that overlaps with that index pattern and has the same priority of 44.

1 Like
4

Hi @dbbaughe thank you for your reply. Hmm, as far as I can tell, we don’t have overlapping index patterns. However, we were updating the existing policies with the same index patterns that used to be updated using the index template before. This is because we have a script that does this periodically. Do you think this could have caused the problem?

5

@lajansa

Could you share the policies you currently have stored (can trim them down to just policy id and ism templates) and the one you are trying to create or update?

6

Hi @dbbaughe sorry for the late reply. You were right before that there were overlapping index patterns. We thought the index patterns were unique enough previously (e.g. cat** and dog) but the wildcard does cause it to be overlapping. Thanks for your help.

1 Like
7

Hey @lajansa can you also provide the exact solution you did to overcome this? And how that was applied to existings indices. That was easy via index.opendistro.index_state_management.policy_id as you push that to existing indices via:

PUT example*/_settings
{
    "settings": {
        "index.opendistro.index_state_management.policy_id": "ism_policy"
    }
}

but I wasn’t able to really get how the new (via ism_template) is applied to existing indices?

Can anyone also provide an example of how that is done :(?

8

Hey @GezimSejdiu,

The ism_template is applied to indices that are created in the future, i.e. it’s listening for index creation events and checking if they match the pattern specified in the ism_template.

If you’d like to apply a policy to an existing index, you would use the Add Policy API

1 Like