Trigger return true but no alerts received 1.7.0

Context :

  • Open Distro alerting plugin version 1.7.0
  • Elastic stack version 7.6.1
  • 3 nodes : 1 master-only and Kibana node + 2 data nodes
  • Type : Extraction Query
  • Trigger purpose : Vertical Scan Security Detection
    Query :
{

  "size": 0,

  "query": {

    "bool": {

      "must": [

        {

          "match": {

            "tags": "firewall"

          }

        },

        {

          "range": {

            "@timestamp": {

              "gte": "now-30s"

            }

          }

        }

      ]

    }

  },

  "aggs": {

    "by_src_ip": {

      "terms": {

        "field": "source.ip"

      },

      "aggs": {

        "by_target_ip": {

          "terms": {

            "field": "destination.ip",

            "order": {

              "unique_port_count": "desc"

            }

          },

          "aggs": {

            "unique_port_count": {

              "cardinality": {

                "field": "destination.port"

              }

            }

          }

        }

      }

    }

  }

}

Trigger code :

int incident = 0;
for (int i = 0; i < ctx.results[0].aggregations.by_src_ip.buckets.size(); i++) {
    for (int j = 0; j < ctx.results[0].aggregations.by_src_ip.buckets[i].by_target_ip.buckets.size(); j++) {
        if (ctx.results[0].aggregations.by_src_ip.buckets[i].by_target_ip.buckets[j].unique_port_count.value > 1) {
            incident += 1;
        }
    }
}
if (incident !== 0) {
  return true;
} else {
  return false;
}

Rule is executed every 1 minute.

Alerting Stats :

> {
>   "_nodes" : {
>     "total" : 3,
>     "successful" : 3,
>     "failed" : 0
>   },
>   "cluster_name" : "hidden",
>   "opendistro.scheduled_jobs.enabled" : true,
>   "scheduled_job_index_exists" : true,
>   "scheduled_job_index_status" : "green",
>   "nodes_on_schedule" : 3,
>   "nodes_not_on_schedule" : 0,
>   "nodes" : {
>     "GTPXL8OsRTyxf69sYiPtfQ" : {
>       "name" : "elk-master",
>       "schedule_status" : "green",
>       "roles" : [
>         "MASTER"
>       ],
>       "job_scheduling_metrics" : {
>         "last_full_sweep_time_millis" : 268316,
>         "full_sweep_on_time" : true
>       },
>       "jobs_info" : { }
>     },
>     "U9y1CQ9ISY6qKKeUyiXuEg" : {
>       "name" : "elk-data-hot",
>       "schedule_status" : "green",
>       "roles" : [
>         "INGEST",
>         "MASTER",
>         "DATA",
>         "ML"
>       ],
>       "job_scheduling_metrics" : {
>         "last_full_sweep_time_millis" : 178600,
>         "full_sweep_on_time" : true
>       },
>       "jobs_info" : {
>         "Bi17YHIBaJTCGwydrJUA" : {
>           "last_execution_time" : 1590778817201,
>           "running_on_time" : true
>         },
>         "Jy7UYHIBaJTCGwyd_ciH" : {
>           "last_execution_time" : 1590779150786,
>           "running_on_time" : true
>         },
>         "oi-TYXIBW13vr6nbOo7x" : {
>           "last_execution_time" : 1590779138317,
>           "running_on_time" : true
>         }
>       }
>     },
>     "THcNACa9QwCli9SUfpgcRA" : {
>       "name" : "elk-data-warm",
>       "schedule_status" : "green",
>       "roles" : [
>         "INGEST",
>         "MASTER",
>         "DATA",
>         "ML"
>       ],
>       "job_scheduling_metrics" : {
>         "last_full_sweep_time_millis" : 59349,
>         "full_sweep_on_time" : true
>       },
>       "jobs_info" : {
>         "Ihf1JXIB1bbtyrEkZ-kL" : {
>           "last_execution_time" : 1590779199206,
>           "running_on_time" : true
>         }
>       }
>     }
>   }
> }

Result when i run it locally :

[ALERT ] Network Enumeration/Reconnaissance : Internal Vertical Scan

  • Message : A vertical scan was detected
  • Trigger: Port Scan
  • Severity: 3
  • Source :
    - Source IP: hidden, Event Count: 120
    - Source IP: hidden, Event Count: 64
    - Source IP: hidden, Event Count: 9
    - Source IP: hidden, Event Count: 9
    - Source IP: hidden, Event Count: 9
    - Source IP: hidden, Event Count: 8
    - Source IP: hidden, Event Count: 7
    - Source IP: hidden, Event Count: 5
    - Source IP: hidden, Event Count: 4
    - Source IP: hidden, Event Count: 4
  • Period start: 2020-05-29T18:44:11Z

My slak destination is working also my alerting installation since I still receive alerts by other monitros.

When i go to trigger INFO I see this at the end :

“triggers”: {
“Port Scan”: {
“value”: 10000,
“enum”: “ABOVE”
}
}

What does this mean ?
My trigger is apparently working but why i don’t receive any alerts. I tried to lower the unique number of ports to make sure that an alert should match but still nothing

Hello,

Do I need Anomaly Detector for this type of queries ? I mean is the extraction query focuses around hits but no aggregations ?

Thanks

Can someone give me some thoughts around this please

Hi @hilo21 can you open an issue on https://github.com/opendistro-for-elasticsearch/anomaly-detection - Thanks.

Hello @alolitas,

I did open an issue and they closed it because it had nothing to do anomaly detection but alerting.
I did open an issue in the alerting repo before this one but nothing yet !

Can someone provide some feedback about the different paths to take?

Thank you

Hi,
I am having same issue. Trigger works fine, but no alerting via SNS or email. Even when I perform manual Test message button.

Any ideas on this issue?

Freddie2020

Hi,

I’ve got the same issue, when I run manually the trigger condition response returns true but doesn’t trigger an alert automatically.

The history says “No alerts”.

Using Elastic Stack: 7.6.1