I am using the anomaly detection plugin (v7.8.0). I am able to create a detector and can preview anomalies from within the Feature creation section.
However, when I start the detector, it remains stuck in the Initializing state.
In the .opendistro-anomaly-results-history index I keep seeing this error:
Threshold model node is not available.
Can someone help with this?
Thank you.
Elasticsearch versions: 7.8.0
Kibana version: 7.8.0
Could you run
GET /_cat/nodes
and share the output here?
Yeah, here it is:
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
100.96.3.136 10 95 39 0.40 0.36 0.41 lr - elasticsearch-logstash-clients-3
100.96.12.162 62 93 100 10.86 12.51 12.26 dilrt - elasticsearch-data-2a-1
100.96.1.16 49 96 0 0.17 0.09 0.03 mr - elasticsearch-master-1
100.96.11.193 65 84 100 11.26 12.98 12.54 dilrt - elasticsearch-data-2a-5
100.118.37.177 24 96 4 0.21 0.13 0.11 mr * elasticsearch-master-2
100.96.12.233 27 99 100 12.15 10.28 10.76 dilrt - elasticsearch-data-2a-3
100.96.20.3 19 92 100 9.82 10.95 10.79 dilrt - elasticsearch-data-2a-0
100.96.1.32 60 95 41 0.38 0.68 0.69 lr - elasticsearch-logstash-clients-2
100.96.1.132 27 96 2 0.10 0.10 0.13 lr - elasticsearch-client-0
100.96.0.5 47 96 100 10.93 11.08 11.02 dilrt - elasticsearch-data-2a-4
100.96.11.192 57 95 50 1.47 1.24 1.03 lr - elasticsearch-logstash-clients-0
100.96.38.96 53 96 100 15.39 13.46 11.83 dilrt - elasticsearch-data-2a-2
100.96.0.16 36 95 56 0.24 0.47 0.59 lr - elasticsearch-logstash-clients-1
100.96.3.144 48 96 0 0.08 0.03 0.02 mr - elasticsearch-master-0
Can you also run the following command?
GET /_cat/nodeattrs?v
Does your node attributes contain “box_type”? Are you using hot or warm nodes?
Sure, here is the output:
node host ip attr value
elasticsearch-master-2 100.118.37.177 100.118.37.177 xpack.installed true
elasticsearch-master-2 100.118.37.177 100.118.37.177 transform.node false
elasticsearch-master-1 100.96.1.16 100.96.1.16 xpack.installed true
elasticsearch-master-1 100.96.1.16 100.96.1.16 transform.node false
elasticsearch-master-0 100.96.3.144 100.96.3.144 xpack.installed true
elasticsearch-master-0 100.96.3.144 100.96.3.144 transform.node false
elasticsearch-logstash-clients-3 100.96.3.136 100.96.3.136 ml.machine_memory 13958643712
elasticsearch-logstash-clients-3 100.96.3.136 100.96.3.136 ml.max_open_jobs 20
elasticsearch-logstash-clients-3 100.96.3.136 100.96.3.136 xpack.installed true
elasticsearch-logstash-clients-3 100.96.3.136 100.96.3.136 transform.node false
elasticsearch-logstash-clients-2 100.96.1.32 100.96.1.32 ml.machine_memory 13958643712
elasticsearch-logstash-clients-2 100.96.1.32 100.96.1.32 ml.max_open_jobs 20
elasticsearch-logstash-clients-2 100.96.1.32 100.96.1.32 xpack.installed true
elasticsearch-logstash-clients-2 100.96.1.32 100.96.1.32 transform.node false
elasticsearch-logstash-clients-1 100.96.0.16 100.96.0.16 ml.machine_memory 13958643712
elasticsearch-logstash-clients-1 100.96.0.16 100.96.0.16 ml.max_open_jobs 20
elasticsearch-logstash-clients-1 100.96.0.16 100.96.0.16 xpack.installed true
elasticsearch-logstash-clients-1 100.96.0.16 100.96.0.16 transform.node false
elasticsearch-logstash-clients-0 100.96.11.192 100.96.11.192 ml.machine_memory 13958643712
elasticsearch-logstash-clients-0 100.96.11.192 100.96.11.192 ml.max_open_jobs 20
elasticsearch-logstash-clients-0 100.96.11.192 100.96.11.192 xpack.installed true
elasticsearch-logstash-clients-0 100.96.11.192 100.96.11.192 transform.node false
elasticsearch-data-2a-5 100.96.11.193 100.96.11.193 ml.machine_memory 122406567936
elasticsearch-data-2a-5 100.96.11.193 100.96.11.193 ml.max_open_jobs 20
elasticsearch-data-2a-5 100.96.11.193 100.96.11.193 xpack.installed true
elasticsearch-data-2a-5 100.96.11.193 100.96.11.193 box_type data
elasticsearch-data-2a-5 100.96.11.193 100.96.11.193 zone 2a
elasticsearch-data-2a-5 100.96.11.193 100.96.11.193 transform.node true
elasticsearch-data-2a-4 100.96.0.5 100.96.0.5 ml.machine_memory 122406567936
elasticsearch-data-2a-4 100.96.0.5 100.96.0.5 ml.max_open_jobs 20
elasticsearch-data-2a-4 100.96.0.5 100.96.0.5 xpack.installed true
elasticsearch-data-2a-4 100.96.0.5 100.96.0.5 box_type data
elasticsearch-data-2a-4 100.96.0.5 100.96.0.5 zone 2a
elasticsearch-data-2a-4 100.96.0.5 100.96.0.5 transform.node true
elasticsearch-data-2a-3 100.96.12.233 100.96.12.233 ml.machine_memory 122406567936
elasticsearch-data-2a-3 100.96.12.233 100.96.12.233 ml.max_open_jobs 20
elasticsearch-data-2a-3 100.96.12.233 100.96.12.233 xpack.installed true
elasticsearch-data-2a-3 100.96.12.233 100.96.12.233 box_type data
elasticsearch-data-2a-3 100.96.12.233 100.96.12.233 zone 2a
elasticsearch-data-2a-3 100.96.12.233 100.96.12.233 transform.node true
elasticsearch-data-2a-2 100.96.38.96 100.96.38.96 ml.machine_memory 122406567936
elasticsearch-data-2a-2 100.96.38.96 100.96.38.96 ml.max_open_jobs 20
elasticsearch-data-2a-2 100.96.38.96 100.96.38.96 xpack.installed true
elasticsearch-data-2a-2 100.96.38.96 100.96.38.96 box_type data
elasticsearch-data-2a-2 100.96.38.96 100.96.38.96 zone 2a
elasticsearch-data-2a-2 100.96.38.96 100.96.38.96 transform.node true
elasticsearch-data-2a-1 100.96.12.162 100.96.12.162 ml.machine_memory 122406567936
elasticsearch-data-2a-1 100.96.12.162 100.96.12.162 ml.max_open_jobs 20
elasticsearch-data-2a-1 100.96.12.162 100.96.12.162 xpack.installed true
elasticsearch-data-2a-1 100.96.12.162 100.96.12.162 box_type data
elasticsearch-data-2a-1 100.96.12.162 100.96.12.162 zone 2a
elasticsearch-data-2a-1 100.96.12.162 100.96.12.162 transform.node true
elasticsearch-data-2a-0 100.96.20.3 100.96.20.3 ml.machine_memory 122406567936
elasticsearch-data-2a-0 100.96.20.3 100.96.20.3 ml.max_open_jobs 20
elasticsearch-data-2a-0 100.96.20.3 100.96.20.3 xpack.installed true
elasticsearch-data-2a-0 100.96.20.3 100.96.20.3 box_type data
elasticsearch-data-2a-0 100.96.20.3 100.96.20.3 zone 2a
elasticsearch-data-2a-0 100.96.20.3 100.96.20.3 transform.node true
elasticsearch-client-0 100.96.1.132 100.96.1.132 ml.machine_memory 13958643712
elasticsearch-client-0 100.96.1.132 100.96.1.132 ml.max_open_jobs 20
elasticsearch-client-0 100.96.1.132 100.96.1.132 xpack.installed true
elasticsearch-client-0 100.96.1.132 100.96.1.132 transform.node false
Also yes, I have box_type attribute. I am using hot nodes only.
Can you change the box_type value from data to hot?
Okay, I changed that, and I see a different error now:
No full shingle in current detection window
Update: after keeping the detector in init state for a while, I got this error:
Aug 7, 2020 @ 14:49:43.799 Cannot get training data
Aug 7, 2020 @ 14:44:43.799 No RCF models are available either because RCF models are not ready or all nodes are unresponsive or the system might have bugs.
Aug 7, 2020 @ 14:39:43.799 No full shingle in current detection window
Do you continuous training data in your history? Without that, you will get “Cannot get training data”.