ISM status stuck in "Initializing"

I’ve been experimenting with applying policies to my indices. I tried applying the example hot/cold policy to and index, but when I assign the policy to the index the status gets stuck in “Initializing”. Ive tried creating other policies and applying those, but they get stuck as well. I followed the instructions in the OpenDistro docs for ISM. Do you have any ideas what to do?

Thanks very much

Hi @jmccombs,

Are you using ODFE or are you on Amazon Elasticsearch Service?

There is a job_interval setting which controls how fast jobs are executed (only configurable on ODFE). The default for ODFE is 5 minutes and the default for the service is 30-48 minutes. So it should show Initializing for that amount of time. Did your indices get stuck on initializing for longer than that?

I am using Amazon Elasticsearch Service, and the policy did finally get applied. Thank you for the insight.

I did test this with an index and an alias and the alias was rolled over to a new index; however, the policy was not applied to the new index. I’m trying to use the policy to create a rolling window of documents that get aged out and deleted after 30 days. I’d like to have an old index for older data, and a current index for new data to be added to, but to get this to work I need the same policy applied to the new index created by the rollover. Do I have to do this manually each time an index is rolled over? It may be I’m going about this all wrong, but I haven’t seen a suggested way of doing this in the documentation or elsewhere on line.

Hi @jmccombs,

You should be able to use an index template to do this. You would create an index template that applies to your group of indices with the policy_id and rollover_alias settings. And then whenever a new index is created from the rollover action it will have the settings applied where it will bootstrap the policy and use the same rollover_alias for rolling over.

Thanks Drew. I’ve looked through the OpenDistro and documentation on index templates at https://opendistro.github.io/for-elasticsearch-docs/docs/elasticsearch/index-data/ , but I don’t see a way to specify a policy inside the template. I also looked through the definitions for mappings and settings in the ElasticSearch documentation and I didn’t find anything there either. Also, my Kibana Index Management panel doesn’t have support for specifying templates. What am I missing?

Hey @jmccombs,

You would use something like this:

And here is a very quick/rough example:

PUT _template/example
{
  "index_patterns": [
    "some-index-pattern*"
  ],
  "settings": {
    "opendistro.index_state_management.policy_id": "my_policy",
    "opendistro.index_state_management.rollover_alias": "my_alias"
  }
}

This would create an index template that matches all indices with the pattern some-index-pattern*. Whenever an index that matches this pattern is created it will automatically apply these settings, policy_id and rollover_alias.

So in your rollover case, you have a policy that is rolling over the index. When it executes rollover it will create a new index and roll over the alias. This new index will match this template pattern and automatically apply the policy_id and rollover_alias. It will then be managed by ISM and keep repeating for newly rolled over indices.

The Kibana Index Management plugin does not have a place to manage Index Templates yet although it is something we want to add. For now you would have to create them through the Elasticsearch APIs or in the Kibana DevTools.

@dbbaughe I am having the same problem. I am using Amazon Elasticsearch Service and created an index policy and applied it to some indicies. Over 3 days later, the job status for all managed indicies is still “Initializing”. Any ideas how I can get these jobs to actually run?

Hey @jeffhunterx, does the master pending task queue look backed up?

If you’re on the managed service it would be best to cut a support ticket so support team can take a look. This type of information (why isn’t the internal job running) isn’t easily/readily available, but we are adding some APIs to job scheduler to expose that information (last time it executed, next time it will execute, etc.).

@dbbaughe When I issue GET /_cluster/pending_tasks there don’t seem to be any pending tasks:

{
  "tasks" : [ ]
}

I opened an AWS support ticket, case 7621564351, to ask for more help. Thanks!

Solved mine by enabling the ISM on the Elasticsearch domain/cluster by it ES API
Use
PUT /_cluster/settings
{
“persistent”: {
“opendistro.index_state_management.enabled” : true
}
}

1 Like

@jmccombs Have you managed to fix this issue. I am facing the same problem and in my case all the indexes are green.

Please let me know the steps to resolve?

Thanks!

You may have to wait a considerable amount of time for the transition. This was the response I got back from the developers, and they were correct:

Are you using ODFE or are you on Amazon Elasticsearch Service?

There is a job_interval setting which controls how fast jobs are executed (only configurable on ODFE). The default for ODFE is 5 minutes and the default for the service is 30-48 minutes. So it should show Initializing for that amount of time. Did your indices get stuck on initializing for longer than that?

@jmccombs I have deployed ODFE on the latest version. I have waited for more than 1 hour and still the indices snapshot policy is not progressing.