Error while rollover action. How to create rollover alias?

Hi,

I have integrated metricbeat and I created a template before sending index to the system. I am getting indices like metricbeat-7.9.1-2020.11.26. I have attached a ISM Policy, so that my expectation is, to create a new index metricbeat-7.9.1-2020.11.26-00001 when the current index cross 100mb size. Metricbeat indices will get created daily, like metricbeat-7.9.1-2020.11.26, metricbeat-7.9.1-2020.11.27, metricbeat-7.9.1-2020.11.28 etc. SO, I want to split up every daily index, based on size. I created the policy like below;

{
    "policy_id": "metricbeat",
    "description": "metricbeat test policy",
    "last_updated_time": 1606385808719,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
        {
            "name": "hot",
            "actions": [
                {
                    "rollover": {
                        "min_size": "100mb"
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "delete",
                    "conditions": {
                        "min_index_age": "5d"
                    }
                }
            ]
        },
        {
            "name": "delete",
            "actions": [
                {
                    "delete": {}
                }
            ],
            "transitions": []
        }
    ]
}

and I received an error while trying to attempt rollover, like the alias does not exist. Then, I updated thhe metricbeat template by adding the template setting like below;

{
    "metricbeat-7.9.1" : {
        "aliases": {
            "metricbeat": {}
        }
    }
}

Now, I am getting error on ISM retry;

{
    "cause": "rollover target [metricbeat] does not exist",
    "message": "Failed to rollover index [index=metricbeat-7.9.1-2020.11.26]"
}

Why is this happening? How can I fix this?

Thanks in advance.

Your rollover target (you called it ‘metricbeat’) is not associated with any index. In fact, it’s just an alias and you must “kickstart” it by assigning it to an index.
Run API
POST /_aliases
{
“actions”: [
{
“add”: {
“index”: “<index name to which the alias should be pointed to”,
“alias”: “your alias name (which is also your rollover target)”,
“is_write_index”: true
}
}
]
}

Please also note that your index name pattern must end with -d+ (see https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html)
And I did not find a way to make it automatically (by setting the respective definitions in the beat’s configuration file), so once I reset indexes for some reason (no metricbeat indexes in ES cluster at all), this must be done manually again.
Tell me if it works for you because for me it rolls over and creates a new index but keeps on writing to the old index (e.g. if I had filebeat-000001, it creates a new filebeat-000002 but no data is in the new index and all keeps on going to the old filebeat-000001. Once filebeat-000001 is deleted according to its policy, it gets recreated and being written to again)

Hi @igorid70

For me also, new index got created (rolled over) but still, data is getting published to the old index and the new index has 0 documents. Why is this happening?

I am breaking my head for more than one week and still no answer. I migrate a “regular” Elastic stack to Open Distro due to license issue (Apache 2.0 vs. Elastic license) and I must admit that it is a complete disappointment. For example, the part of Index management took for me around 20 minutes to learn and configure in Elastic stack with its ILM (Index Lifecycle maintenance). How people use Open distro if you can’t find any documentation beside the general explanations? Isn’t the basic feature of rolling indexes over implies also writing to the new index which was just rolled over? Someone?

I agree. ILM was somewhat easier to learn than ISM I believe, also, the documentation of OD is “hard to digest”. What is missing here is step-by-step guides. For example, its not so hard for them to publish a guide / blog to achieve ISM in Filebeat / Metricbeat, which might be helpful for so many people.

I was just wondering, is it possible to create daily indices based on template and alias. FOr example, if I have an alias myalias pointing to myindex-000001, if I publish messages to myalias, finally the data will be published to myindex-000001 for sure, and if ISM is successful, it will create myindex-000002, myindex-000003 and so on. What I am trying to achieve is, is it possible to append datestamp before the index number and after index name, like, myindex-2020.12.01-00001, myindex-2020.12.01-00002, myindex-2020.12.02-00001, myindex-2020.12.03-00001, etc… Daily indices will get created, along with numbering. Is this possible?

Yes, it is possible. You need to specify something like this in your metricbeat.yml
output.elasticsearch.index: "index: “metricbeat-%{+yyyy.MM.dd}-000001”
You can play with the name as you wish (based on ES restrictions about the special characters, etc), but if you want the rollover feature, your index name must end with a digit (and during rollover, the last digit will be increased by one and the number will become 6 digits). But of course as per our common issue, once rolled over, the new index will stay “empty” meaning it will keep on writing to the old one. I will try to post a new question about the proper way (if there is such a way) to setup the automatic rollover (when the write index is the one we just rolled over to)

Hello @igorid70 ,

Were you able to make any positive progress on this issue? I seem to be having similar issues so was curious before I reached out to support. We basically want to rollover the indexes once per day OR increment up multiple times per day if the index is over 10GB which-ever comes first.

For Example:
<index_name> — <index_size>
myindex-2020.12.01-000001 — 10GB
myindex-2020.12.01-000002 — 8GB
myindex-2020.12.02-000003 — 3GB
myindex-2020.12.03-000004 — 10GB
myindex-2020.12.03-000005 — 10GB
myindex-2020.12.03-000006 — 3GB
myindex-2020.12.04-000007 — 5GB

This is mostly how ILM would setup the indexes and how we want to replicate it, other suggestions are welcome. We mainly are looking for the best option to store 30 days of data, with a somewhat fixed Index size for performance.

Thanks!
-Chris

Hi @ChrisCarter
Unfortunately, nobody answered in this forum and also in official elastic forum where I published the same question. I really don’t understand how people implement Opendistro without such a basic thing. What do they do with the indices which keep on getting accumulated? For me it was a definite ‘no go’ for Opendistro but our legal department did not allow to use Elasticsearch in its basic subscription due to the limitation in Elastic license (BTW, I am very pleased with all the rest in Opendistro).
Anyway, I ended up by writing a Perl script using the official Elasticsearch Perl client which reads the configuration file where I define the cleanup rules (based on size or age of an index/index pattern) and it deletes the indices if the conditions are met. For the indices themselves I left the default configuration, so beats produce daily indices, i.e. filebeat-7.9.1-2020.12.18, metricbeat-7.9.1-2020.12.18, heartbeat-7.9.1-2020.12.18 which of course can’t be rotated because they don’t match the pattern required for rotation.