Help needed for creating a template for merging & deleting old indices

[lehner.angelica]

Hi all,
I’m using a windows based Opendistro Kibana installation with a regular elasticsearch installation.

Been trying to figure out things on “Opendistro” as I’m not an experienced programmer, on some parts it was easy, other parts were not since I’m missing some basic programming skills (working on it, big thanks to several people in this community that really helped me).

The last mile I’m trying to figure is “Indexing Management” (I followed the documentation but honestly did not understand everything).

I have several indexes:
check*
ops*
dc*
mc*
(each one has a timestamp)

I want to create a template, that will automatically add newly created indices and delete indices older than 30 days.

I searched here and found this code:

"policy": {
    "policy_id": "delete_older_than_30d",
    "description": "Policy that deletes indicies older than 30 days",
    "last_updated_time": 1598550179368,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "open",
    "states": [
        {
            "name": "open",
            "actions": [],
            "transitions": [
                {
                    "state_name": "delete",
                    "conditions": {
                        "min_index_age": "30d"
                    }
                }
            ]
        },
        {
            "name": "delete",
            "actions": [
                {
                    "delete": {}
                }
            ],
            "transitions": []
        }
    ]
}

}

It would be much appreciated for a short explanation to a layman such as I on how and where I create a template? since I’m not seeing any “template” option under “Index Management”.
Do I need to create it under “Dev Tools”? If so, I will be grateful for any help on how to edit the code above and create a template that will automatically add newly created indices and delete older than 30 days.

Would also like to know how to get the correct “last_updated_time” for this code and how it effects the code \ indexes?

Would you recommend using actions such as roll overs, aliases or merging for indexes? for dealing with indexes In aspects of performance and saving disk space.

Thanks in advance and apologies for the newbie questions

[stmx38]

Hello @lehner.angelica,
At the moment we still use Elasticsearch Curator for index cleanup and started to consider to move to the Index Management. This post can be as a short guide for us.

Per documentation, in order to just cleanup old indices by wildcard we should:

1. Define a Policy which should delete old indices: Kibana --> Index management Kibana --> Create policy
   Policy ID: vpn-log-test-cleanup
   Define policy:

   {
     "policy": {
       "description": "Policy that deletes indices 'vpn-log-test' older than 30 days",
       "default_state": "open",
       "schema_version": 1,
       "states": [
         {
           "name": "open",
           "actions": [],
           "transitions": [
             {
               "state_name": "delete",
               "conditions": {
                 "min_index_age": "30d"
               }
             }
           ]
         },
         {
           "name": "delete",
           "actions": [
             {
               "delete": {}
             }
           ]
         }
       ]
     }
   }
   

2. Create an Index template, to be able to attach policy to multiple indices by wildcard and attach created policy to it. It can be done via Dev Tools/Console or cURL:
   Dev Tools
   Create Index template

   PUT _template/vpn-log-test
   {
     "index_patterns": [
       "vpn-log-test-*"
     ],
     "settings": {
       "opendistro.index_state_management.policy_id": "vpn-log-test-cleanup"
     }
   }
   

   Check the result

   GET _template/vpn-log-test
   

   cURL
   Create Index template

   curl -X PUT http://localhost:9200/_template/vpn-log-test -H 'Content-Type: application/json' -d'
   {
     "index_patterns": [
       "vpn-log-test-*"
     ],
     "settings": {
       "opendistro.index_state_management.policy_id": "vpn-log-test-cleanup"
     }
   }'
   

   Get created template

   curl http://localhost:9200/_template/vpn-log-test?pretty
   

Testing

Note: Index management policy will be attached to the indice in the moment of its creation because attachment is described in the Index template. It means that the policy will be attached to the newly index only.

1. Create a new Index

   # Variables
   elasticsearch_url=http://localhost:9200
   date=$(date +%Y-%m-%d)
   index_name=vpn-log-test-$date
   index_type=default

   users="Alice Bob"
   error="VPN connection failed"

   # Log to the Elasticsearch
   for user in $users; do
     time=$(date +%Y-%m-%d'T'%H:%M:%S.%3N)

     curl -H "Content-Type: application/json" \
        -XPOST "$elasticsearch_url/$index_name/$index_type" \
        -d "{\"Time\":\"$time\", \"User\":\"$user\", \"Error\":\"$error\"}"
        sleep 2
   done

2. Check if policy was attached

We see that only the indice created today (righ now) is ‘Managed by Policy’, as it was described in the note above.

3. For testing purposed we changed conditions to the 1m

        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "1m"
            }
          }
        ]

4. We see that policy status was changed to the Initializing

5. After 5 minutes of waiting (Index management scheduler running period) we see that policy change to the Running

6. Finaly, we see that indice dissapeared from the list as it was deleted by Index management

[lehner.angelica]

Hey,

Thanks!
If i understand correctly, I have to create a template for each of my indexes but the policy will only work on newly created indexes?

I already have 2 weeks of data for each index, can I delete them manually without fear of damaging? (using Cerebro or curl)

Regards

[stmx38]

Index template is an object which describes properties for the indices specified in the index_patterns, at creation. Theoretically, in index_patterns you can specify a wildcard * (it is possible, will it also applied to the system indices, it is a good idea?). Each indice can contain very specific configuration.

Accordingly to the short information above we should create index template for every indices set you have - check*, ops*, dc*, mc*.

You can delete indices using Dev Tools or curl. I don’t know how to use Cerebro.

Dev Tools

Get sorted list of indices
GET _cat/indices?s=index

Delete indice

DELETE vpn-log-test-2020-08-27
DELETE vpn-log-test-2020-08-2*
DELETE *test-2020-08-2*

curl

Get sorted list of indices

curl http://localhost:9200/_cat/indices?s=index

Delete indices

curl -XDELETE http://localhost:9200/vpn-log-test-2020-08-27
curl -XDELETE http://localhost:9200/vpn-log-test-2020-08-2*
curl -XDELETE http://localhost:9200/*test-2020-08-2*

Also, you can apply manually created policy to the required indices. Index template is just an automation for policy assignment for the indices with will be created in the future.