Can you automatically manage indices?

There is only an option to enable management manually for specific indices. Is there a way to have new indices automatically join a policy or to automatically add them with a wildcard? If I do it from the management page it just tells me my indices are not managed.

So I believe I figured this out. I applied an index template to fluentd* so that the settings are auto applied that you see in the docs.

PUT _template/template_1
{
  "index_patterns": ["fluentd*"],
  "settings": {
    "opendistro.index_state_management.policy_id": "delete_old"
  }
}
2 Likes

hi @sjs344, I had the same question once Index Management was released.
I’ve tried your trick, but I can’t see my indices which is related to my index template in Managed Indices List. Does it really works, I mean it proceeds with policies?

This only works on newly created indices. Your old ones will have to be added manually. It does work though.

You are right!
But if you update setting in such way, all my other settings will be overriden.

PUT _template/template_1
{
  "index_patterns": ["fluentd*"],
  "settings": {
    "opendistro.index_state_management.policy_id": "delete_old"
  }
}

I’ve tryed to get all template setting first GET _template/template_1 and then add "opendistro.index_state_management.policy_id": "delete_old"
Now it works.
Thanks Sam.

Could you please share your policy configuration "delete_old".

Attached below

{
    "policy": {
        "policy_id": "delete_old",
        "description": "A simple default policy that deletes old indicies.",
        "default_state": "open",
        "states": [
            {
                "name": "open",
                "actions": [
                    {
                        "open": {}
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "5d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ]
    }
}
1 Like

Hi @ogulman,

It looks like you were able to manually add the policy_id to the existing indices and workaround updating the template by getting the settings first and modifying the whole template in the update.
So is the current issue that you want to update the template itself specifically in metricbeat.yml?

Hi Drew, @dbbaughe
Yes, for an example this is my template for logstash where I specify ISM policy and works well.

# cat logstash/template.json

    {
    "template": "logstash-*",
    "settings": {
        "number_of_shards": 1,
        "number_of_replicas" : 0,
        "opendistro.index_state_management.policy_id": "delete_old"
    }
}

But in case of metricbeat, it is ignoring those settings.
image

I forgot to put index. before opendistro.index_state_management.policy_id: "delete_old", so now works! Sorry about that.
image

Awesome! Good to know :slight_smile:

Hi @sjs344,

Thank you for sharing how you solved this setup. I see you shared everything, but one final piece, how does your Fluentd conf look. Could you post how you configured Fluent to call the template on match?

is this still the way to do automatic index management? Because i cant get it to work.

I created a policy, created a policy for index pattern “logstash-*” and i can see new index have the policy in settings but the managed index page still lists newly created indexes as not managed.

Looking into the GET /_opendistro/_ism/explain/logstash-2021.03.99 shows that the policy_id is null.

Hi @nkk,

Assuming you are on one of the newer versions, we moved away from settings and now you define the index patterns directly on the policy. You can see an example in the documentation:

Thanks I knew i was missing something. Have a good weekend