Monitors created through api doesn't reflect proper schedule

Plugins Alerting
1

Hello,

I am working on migrating monitors from domain A to domain B. Monitors were manually created in Domain A, and we cannot do that now, as there are over 100 monitors we need to migrate.

I was able to use the Open Distro api
API - Open Distro Documentation - “_opendistro/_alerting/monitors/” to fetch and create monitors from domain A into domain B respectively.

Although I can fetch and create the monitors successfully, the monitor schedule doesn’t seem to get updated properly.

Below is the response I received through monitors GET call -
‘https://my-domain-url/_opendistro/_alerting/monitors/_search/’

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1.0,
    "hits": [
      {
        "_index": ".opendistro-alerting-config",
        "_type": "_doc",
        "_id": "80inDHsB0LoZ6kRToVXD",
        "_version": 7,
        "_seq_no": 13,
        "_primary_term": 1,
        "_score": 1.0,
        "_source": {
          "type": "monitor",
          "schema_version": 3,
          "name": "test-monitor-1",
          "user": {
            "name": "",
            "backend_roles": [],
            "roles": [],
            "custom_attribute_names": [],
            "user_requested_tenant": null
          },
          "enabled": true,
          "enabled_time": 1628004917677,
          "schedule": {
            "cron": {
              "expression": "0 7 * * *",
              "timezone": "US/Central"
            }
          },
          "inputs": [
            {
              "search": {
                "indices": [
                  "sample_data"
                ],
                "query": {
                  "size": 0,
                  "query": {
                    "match_all": {
                      "boost": 1.0
                    }
                  }
                }
              }
            }
          ],
          "triggers": [
            {
              "id": "90iqDHsB0LoZ6kRTeFVn",
              "name": "test-trigger-1",
              "severity": "1",
              "condition": {
                "script": {
                  "source": "ctx.results[0].hits.total.value > 0",
                  "lang": "painless"
                }
              },
              "actions": [
                {
                  "id": "-EiqDHsB0LoZ6kRTeFVn",
                  "name": "test-alert-1",
                  "destination_id": "8kiiDHsB0LoZ6kRTXlW9",
                  "message_template": {
                    "source": "Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n- Trigger: {{ctx.trigger.name}}\n- Severity: {{ctx.trigger.severity}}\n- Period start: {{ctx.periodStart}}\n- Period end: {{ctx.periodEnd}}",
                    "lang": "mustache"
                  },
                  "throttle_enabled": false,
                  "subject_template": {
                    "source": "Test Alert 1",
                    "lang": "mustache"
                  }
                }
              ]
            }
          ],
          "last_update_time": 1628022569481
        }
      }
    ]
  }
}

Domain A:

If you look at the schedule object in the above response, the value is returned as a cron expression where as I set it at ‘Daily’ frequency.

METHOD 1:
Anyhow, I went ahead and used the same config to create the monitors -

"schedule": {
  "cron": {
    "expression": "0 7 * * *",
    "timezone": "US/Central"
  }
}

Now, in Domain B, in the monitor overview section I see ‘Schedule’ showing a cron expression, but if I hit ‘Edit’ and scroll to the Schedule section, I see default values -

image


METHOD 2:
I also tried to set the schedule using interval, while I am creating this monitor in the new Domain, something like below.

"schedule":{
  "period": {
    "interval": "1",
    "unit": "DAYS",
    "timezone": "US/Central"
  }
}

Now the timezone is omitted in the monitor overview page, and the schedule doesn’t show update either -

I’d like to set the monitors to run every day at 7am central time.
Thanks for reading through, I’d appreciate any help to understand how to fix this


Best,
Prashanth Sripathi

2

Hi @psripathi,

Just tested this locally. I’d say this is a bug in frontend behavior. What’s happening is that the cron schedule for the Monitor you created using the backend REST API is being saved into the Monitor, however it’s displaying the default form value for the Monitor which is "By interval". If you actually select "Custom cron expression", you can see the cron values match what you saved the Monitor with.

This incorrect default is still dangerous though because if the user does not notice it and changes something else when editing the Monitor, it overrides their schedule with the default “1 minute interval”.

I think the correct behavior here might be to display the custom cron by default if the frequency is not an interval to ensure it can be displayed.

I went ahead and created a GitHub issue in the OpenSearch project repo to track this since all new issues will be handled there.

2 Likes
3

Hello @qreshi, thank you for your response and the issue created to fix this.

I do not want to use cron expression actually to create these monitors, its just the GET request returns such cron schedule, which is fine, as I can change the payload, when making POST request.

I actually couldn’t find in the API docs on how to set the monitor schedule using “Daily” interval, something similar to below -
image

So, how should that payload look like?

...
"schedule":{
  "period": {
    "interval": "1",
    "unit": "DAYS",
    "timezone": "US/Central"
  }
}
...

TIA

4

Hey @psripathi,

The “daily frequency at a given time” is actually saved as a cron schedule. So the cron you are using is the correct configuration to get the Monitor to execute daily at 7AM in your timezone.

The period interval you’ve listed would be executing the Monitor around every 24 hours with no respect to a specific time of day.