Alert is Not triggering

I have created monitor and trigger. When i am triggering manually , it is getting trigger. But automatic not triggering. “schedule_status” : showing red

{
“size”: 1,
“query”: {
“match”: {
“event_id”: {
“query”: “4720”,
“operator”: “OR”,
“prefix_length”: 0,
“max_expansions”: 50,
“fuzzy_transpositions”: true,
“lenient”: false,
“zero_terms_query”: “NONE”,
“auto_generate_synonyms_phrase_query”: true,
“boost”: 1
}
}
}
}

“cluster_name” : “elasticsearch”,
“opendistro.scheduled_jobs.enabled” : true,
“scheduled_job_index_exists” : true,
“scheduled_job_index_status” : “green”,
“nodes_on_schedule” : 0,
“nodes_not_on_schedule” : 1,
“nodes” : {
“4jPUFB8aQuuTWcgpHyhOxw” : {
“name” : “4jPUFB8”,
“schedule_status” : “red”,
“roles” : [
“MASTER”,
“DATA”,
“INGEST”

Hi,

Can you please update with step by step on how to reproduce the issue? That is the best way for us to dive deep into the issue.

If it can’t be reproduced easily please post the answers to following questions.

  1. All the information about the Monitor.
    • curl localhost:9200/_opendistro/_alerting/monitors/<MonitorJobId>
  2. Response of alerts with regards to the Monitor.
    • curl localhost:9200/.opendistro-alerting-alerts/_search?pretty -H 'Content-Type: application/json' -d '{"query": {"terms": {"_routing": ["<MonitorJobId>"]}}}'
  3. Opendistro Thread pool details
    • curl localhost:9200/_cat/thread_pool/opendistro_monitor_runner?v\&h=id,name,active,rejected,completed
  4. Full information about the Alerting stats
    • curl localhost:9200/_opendistro/_alerting/stats
  5. Any errors that is shown in the Elasticsearch log.

Hi it seems alerts is just triggering one time and after that alerts is not triggering for any new events.
Mean , i acknowledge one alerts , then i created same event again for the same alerts , but alerts didn’t trigger.

1.)

"_id" : "BmO3j2kBBSkdQvmwHoOt",
  "_version" : 6,
  "monitor" : {
    "type" : "monitor",
    "name" : "User Created",
    "enabled" : true,
    "enabled_time" : 1552894335947,
    "schedule" : {
      "period" : {
        "interval" : 1,
        "unit" : "MINUTES"
      }
    },
    "inputs" : [
      {
        "search" : {
          "indices" : [
            "winlogbeat-2019.03.19"
          ],
          "query" : {
            "size" : 100,
            "query" : {
              "match" : {
                "event_id" : {
                  "query" : "4720",
                  "operator" : "OR",
                  "prefix_length" : 0,
                  "max_expansions" : 50,
                  "fuzzy_transpositions" : true,
                  "lenient" : false,
                  "zero_terms_query" : "NONE",
                  "auto_generate_synonyms_phrase_query" : true,
                  "boost" : 1.0
                }
              }
            }
          }
        }
      }
    ],
    "triggers" : [
      {
        "id" : "C2O3j2kBBSkdQvmwjINP",
        "name" : "User Created",
        "severity" : "2",
        "condition" : {
          "script" : {
            "source" : "ctx.results[0].hits.total > 0",
            "lang" : "painless"
          }
        },
        "actions" : [ ]
      }
    ],
    "last_update_time" : 1552979590597
  }
}

2.)
{
  "took" : 18,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".opendistro-alerting-alerts",
        "_type" : "_doc",
        "_id" : "EWO4j2kBBSkdQvmwBoNM",
        "_score" : 1.0,
        "_routing" : "BmO3j2kBBSkdQvmwHoOt",
        "_source" : {
          "monitor_id" : "BmO3j2kBBSkdQvmwHoOt",
          "monitor_version" : 2,
          "monitor_name" : "User Created",
          "trigger_id" : "C2O3j2kBBSkdQvmwjINP",
          "trigger_name" : "User Created",
          "state" : "ACKNOWLEDGED",
          "error_message" : null,
          "alert_history" : [ ],
          "severity" : "2",
          "start_time" : 1552894395778,
          "last_notification_time" : 1552894456006,
          "end_time" : null,
          "acknowledged_time" : 1552894473452
        }
      }
    ]
  }
}

3.) 
id                     name                      active rejected completed
4jPUFB8aQuuTWcgpHyhOxw opendistro_monitor_runner      0        0        66

4.){
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "elasticsearch",
  "opendistro.scheduled_jobs.enabled" : true,
  "scheduled_job_index_exists" : true,
  "scheduled_job_index_status" : "green",
  "nodes_on_schedule" : 1,
  "nodes_not_on_schedule" : 0,
  "nodes" : {
    "4jPUFB8aQuuTWcgpHyhOxw" : {
      "name" : "4jPUFB8",
      "schedule_status" : "green",
      "roles" : [
        "MASTER",
        "DATA",
        "INGEST"
      ],
      "job_scheduling_metrics" : {
        "last_full_sweep_time_millis" : 189817,
        "full_sweep_on_time" : true
      },
      "jobs_info" : {
        "BmO3j2kBBSkdQvmwHoOt" : {
          "last_execution_time" : 1552980675947,
          "running_on_time" : true
        },
        "BS_OlGkB8nQEAav636lN" : {
          "last_execution_time" : 1552980679375,
          "running_on_time" : true
        }
      }
    }
  }
}

Did you test monitor without any action creation?
Is it triggering after first trigger.

Hi Learner,

From your example you have a monitor running every 1 minute and a trigger that alerts whenever it finds at least one document matching the query set up in your monitor. You can only ever have one active alert for a monitor-trigger pair, it will just update the current active alert if it already exists. When you acknowledge an alert it still remains in an “active” state, acknowledging is a way to reduce noise so that you aren’t notified constantly. When the alert is acknowledged, any “action” on the Trigger will be skipped.
Once the existing “acknowledged” alert is completed, the monitor will be back to “normal” state and can create new active alert when trigger condition is met.

How to complete alerts. Once i acknowledge alerts , i am not able to select that alerts again.

The alert will be in complete state when the “corresponding trigger evaluates to false.” You can also view the details about different Alerting State in our Opendistro Alerting Document.

Hi, Jinsoor

I am also looking for this.how I can evaluate trigger to false positive so that alerts will be completed.i gone through documents but I didn’t get much stuff about it.
From which tab I can perform trigger evaluation, from dashboard or monitor. In dashboard , once I acknowledge alerts I am not able to select those alerts again to make it complete.

Hi Yogesh,

We do not have the notion of a “false alert” in alerting. Acknowledging an alert means that you have acknowledged there is an active alert, are taking steps to fix it and do not need to be notified again (if you set up any actions for it). Fixing it can be by fixing the underlying reason that caused it to trigger or modifying the trigger itself so on the next execution it evaluates to is false. We do not support manually completing an alert, since you would just have a brand new alert and be notified the next time the monitor runs if it still evaluates to true.

That being said, the notion of a “false alert” and how we could handle such scenarios is definitely interesting, will look into it more :slight_smile:

Thanks , as i understood , as of now alerts function will not work like SIEM alerts or real time alerts. My expectation is alerts should show on dashboard or notify on each and every condition match irrespective of whether it is active or acknowledge or completed. If condition matching two time then two alerts should be on dashboard , not only one.

Hi Learner - The Alerting project is providing real-time alerts it is just not re-notifying on every occurrence of an Active alert once you have acknowledged it - which is the equivalent of saying you are working on it. Our thinking was not to spam folks (probably because we hate it when we get spammed) but I can certainly understand the counter-point. Looks like someone created an issue for the same on GitHub: https://github.com/opendistro-for-elasticsearch/alerting/issues/10. Feel free to plus one it. It may be a good candidate to make this an option as I can see how folks could like it both ways.

How I can complete alerts , so that alert will get trigger on next match.
Once alert acknowledged , I am not able to select that alerts to complete and so alert is not getting trigger or notify on second match.
Is there any way. I can complete alert. I gone through documents but no luck.

Hi Yogesh, if you would like to be notified every time the alert is triggered then you can choose to not acknowledge the alert and you will be notified every time. Acknowledging is simply a way to stop notifications for that ongoing alert, which seems to be the opposite of what you want to do.

My simple requirement
Alert Name :virus detected , I have created simple query to match virus detected events.
Now first events virus detected from system A after some time virus detected from system b , so expecting two separate alerts on dashboard.

Another example: user created alerts
If two different user is getting created then expecting two separate alerts on dashboard.

How I can achieve this.

.

Hi Yogesh,

An example of how you could work around your two requirements:

  1. Create a monitor that runs every 5 minute
  2. Write a query using the range filter with the provided period_start period_end parameters we provide to only look at documents from the last 5 minutes and matching your document types.
  3. Use a terms aggregation to create unique buckets per user_id or per system (for virus use case).
  4. Write a trigger that will notify you whenever the aggregation buckets list is > 0
  5. And in the action message, you can notify yourself with how many users were created in the last 5 minutes or how many systems have a virus in the last 5 minutes along with their user_ids/system name respectively.

You are correct in that we do not support an alert per document like you’re looking for, but there are ways to work around it like above. Let me know if I missed anything, thanks :slight_smile:

Hi Thanks,

I have managed to configured it. And now i am near to my expectation.

Below is my query. Now alerts are getting completed automatically at 5 minute and new alerts are triggering after 5mn on next event match. Now my concern is , two events matched (two different user/system) within 5 mn but still alert is one only. I want alerts on each match irrespective of time span . How i can do it.

{
    "size": 2,
    "query": {
        "bool": {
            "must": [
                {
                    "match": {
                        "event_id": {
                            "query": "4726",
                            "operator": "OR",
                            "prefix_length": 0,
                            "max_expansions": 50,
                            "fuzzy_transpositions": true,
                            "lenient": false,
                            "zero_terms_query": "NONE",
                            "auto_generate_synonyms_phrase_query": true,
                            "boost": 1
                        }
                    }
                }
            ],
            "filter": [
                {
                    "range": {
                        "@timestamp": {
                            "from": "now-5m",
                            "to": null,
                            "include_lower": true,
                            "include_upper": true,
                            "boost": 1
                        }
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    },
    "aggregations": {
        "Group_By_TargetUserName": {
            "terms": {
                "field": "event_data.TargetUserName.keyword",
                "size": 10,
                "min_doc_count": 1,
                "shard_min_doc_count": 0,
                "show_term_doc_count_error": false,
                "order": [
                    {
                        "_count": "desc"
                    },
                    {
                        "_key": "asc"
                    }
                ]
            },
            "aggregations": {
                "get_latest": {
                    "terms": {
                        "field": "@timestamp",
                        "size": 1,
                        "min_doc_count": 1,
                        "shard_min_doc_count": 0,
                        "show_term_doc_count_error": false,
                        "order": {
                            "_key": "desc"
                        }
                    }
                }
            }
        }
    }
}

Hi Yogesh,

We do not currently support alerting on a per document/event level. The example I gave is what you can do to work around that for now (it supports as low as minute intervals).

You can also make a feature request in the issues section of the GitHub repository:

I think the problem is that the retriggering of alerts is missing; as outlined in this feature-request:

We have the same issue; we use aggregations to separate the entities, but since there are no retriggering of alarms we are left in the dark. You dont need to be able to create a fully result-aware retrigger-mechanism, a simple “retrigger” with possibly a delay would go a long way of solving the problem.

1 Like

Hi, I think you have to add a time range inside a bool query. For example:

          "bool": {
            "must": {
                "match" : {
                                your query here
                           }
                "range": {
                    "@timestamp": {
                        "from": "now-1m",
                        "to": null,
                        "include_lower": true,
                        "include_upper": true,
                        "boost": 1
                    }
                  }
                }
              }

This will query your match within only the last one minute. This is how I solved this problem. I hope I can help you

3 Likes

+1 It would be nice to add retriggering of alerts. We have the same issue for daily triggered events. We use these alerts to let sales follow up with orders, the alerts can be triggered everyday for different orders.