@dbbaughe We’re seeing ERROR messages in the ODFE ES log like this:
Failed to get IndexMetaData from master cluster state for index=kubernetes_cluster-kube-system-2020-06-21
followed by a stack trace. The messages appear every 2 hours which corresponds to how frequently we’ve set the ISM job to run.
The odd thing is this index doesn’t exist. It did exist but was assigned a policy that deleted it after 3 days and that appears to have worked as desired. I’ve dumped the records in the .opendistro-ism-managed-index-history-* indexes and there are only two records: one indicates the ISM policy was successfully initialized and the 2nd indicates it was transitioning to the “doomed” state (which is when it is deleted).
So, while the ISM policy appears to have worked as expected, why is the ISM plug-in attempting to get IndexMetaData for it every two hours? Our deployment creates a number of indexes each day, so it is odd that this behavior only occurs for this particular index. I’ve also checked the other ISM related index (.opendistro-ism-config) and it only contains records for the current set of indexes (i.e. only those that currently exist and are not older than 3 days).