Errors opening anomaly detection plugin for dashboards after creation via API

I’m migrating my project from Open Distro to OpenSearch and am having an issue I didn’t have before.

This is using opensearch in a docker image based on opensearchproject/opensearch:1.1.0, the main difference being that I’m removing the opensearch-security plugin.

I’m using the API to create some detectors. As far as I can tell from the output, there are no errors in creating the detectors.

I can see the .opendistro-anomaly-detectors index with the _cat API:

bash-5.1# curl -XGET http://opensearch:9200/_cat/indices?expand_wildcards=all | sort
green  open .kibana_1                                          bl6RECCfQ0WzvuvTneASmw 1 0   1865   265 655.9kb 655.9kb
green  open arkime_dstats_v30                                  DRpMlzt5ThCx2RaUudtObw 2 0     16     4 186.9kb 186.9kb
green  open arkime_fields_v30                                  ZlhsGU0aSRO6s0UOZ4VAAQ 1 0   1163    37 272.2kb 272.2kb
green  open arkime_files_v30                                   vlwmFJrWRPyzc7khYStDTw 2 0      6     0  29.1kb  29.1kb
green  open arkime_history_v1-21w46                            QfS_sJcURhG57fVNBC6wBw 1 0      1     0   5.2kb   5.2kb
green  open arkime_hunts_v30                                   RljBCva3SSGtq4abwEaG1A 1 0      0     0    208b    208b
green  open arkime_lookups_v30                                 OivEyO0YSgq43hpkGMu3Mg 1 0      0     0    208b    208b
green  open arkime_queries_v30                                 o55HBsmzQg-9l8aykqCiMw 1 0      0     0    208b    208b
green  open arkime_sequence_v30                                -OjhMqaQSkSW4V2-mfm7YA 1 0      1     2  13.3kb  13.3kb
green  open arkime_sessions3-200428                            OtwfYG8TTUWbV7_cm6zMlw 1 0 165416 30418  93.1mb  93.1mb
green  open arkime_sessions3-211122                            O_ILfY3QS4i1iyx9yNqKdg 1 0  14872  5490    17mb    17mb
green  open arkime_stats_v30                                   EJHssTzYQZOwBwvBiV-kUg 1 0      1    11  98.5kb  98.5kb
green  open arkime_users_v30                                   fveZyV8CSUu6HHc4ehXh4Q 1 0      2     5 204.5kb 204.5kb
yellow open .opendistro-anomaly-detectors                      rEgXGb6uSxaAQGIjVgo7nA 1 1      7     0  32.7kb  32.7kb
yellow open .opendistro-ism-config                             CgqYlqd0Tqi_uvDJTWERKA 1 1      7     2  89.3kb  89.3kb
yellow open .opendistro-ism-managed-index-history-2021.11.22-1 pyUOj7g4Rpef_ySflc-JDg 1 1      6     0  13.7kb  13.7kb
yellow open .opendistro-job-scheduler-lock                     T3SBe837TY-yGAOYkLuORg 1 1      2     3  24.7kb  24.7kb

And I can also do a _search operation and see them:

bash-5.1# curl -XPOST "http://opensearch:9200/_plugins/_anomaly_detection/detectors/_search" -H 'osd-xsrf:true' -H 'Content-type:application/json' -d "@/tmp/arkimesearch.json" | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3130  100  3030  100   100   200k   6779 --:--:-- --:--:-- --:--:--  218k
{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".opendistro-anomaly-detectors",
        "_type": "_doc",
        "_id": "QdZ3SX0BVwteEm6PYOmp",
        "_version": 1,
        "_seq_no": 0,
        "_primary_term": 1,
        "_score": 1,
        "_source": {
          "name": "total_bytes_AD",
          "description": "Detect anomalies based on total bytes, summed",
          "time_field": "firstPacket",
          "indices": [
            "arkime_sessions3-*"
          ],
          "filter_query": {
            "bool": {
              "filter": [
                {
                  "exists": {
                    "field": "network.bytes",
                    "boost": 1
                  }
                }
              ],
              "adjust_pure_negative": true,
              "boost": 1
            }
          },
          "detection_interval": {
            "period": {
              "interval": 10,
              "unit": "Minutes"
            }
          },
          "window_delay": {
            "period": {
              "interval": 1,
              "unit": "Minutes"
            }
          },
          "shingle_size": 8,
          "schema_version": 0,
          "feature_attributes": [
            {
              "feature_id": "QNZ3SX0BVwteEm6PYOkL",
              "feature_name": "total_bytes",
              "feature_enabled": true,
              "aggregation_query": {
                "total_bytes": {
                  "sum": {
                    "field": "network.bytes"
                  }
                }
              }
            }
          ],
          "last_update_time": 1637615100072,
          "detector_type": "SINGLE_ENTITY"
        }
      },
      {
        "_index": ".opendistro-anomaly-detectors",
        "_type": "_doc",
        "_id": "Q9Z3SX0BVwteEm6PYOnK",
        "_version": 1,
        "_seq_no": 1,
        "_primary_term": 1,
        "_score": 1,
        "_source": {
          "name": "file_mime_type_AD",
          "description": "Detect anomalies based on transferred file type",
          "time_field": "firstPacket",
          "indices": [
            "arkime_sessions3-*"
          ],
          "filter_query": {
            "bool": {
              "filter": [
                {
                  "exists": {
                    "field": "file.mime_type",
                    "boost": 1
                  }
                }
              ],
              "adjust_pure_negative": true,
              "boost": 1
            }
          },
          "detection_interval": {
            "period": {
              "interval": 10,
              "unit": "Minutes"
            }
          },
          "window_delay": {
            "period": {
              "interval": 1,
              "unit": "Minutes"
            }
          },
          "shingle_size": 8,
          "schema_version": 0,
          "feature_attributes": [
            {
              "feature_id": "QtZ3SX0BVwteEm6PYOm8",
              "feature_name": "file_mime_type",
              "feature_enabled": true,
              "aggregation_query": {
                "file_mime_type": {
                  "value_count": {
                    "field": "file.mime_type"
                  }
                }
              }
            }
          ],
          "last_update_time": 1637615100106,
          "detector_type": "SINGLE_ENTITY"
        }
      },
      {
        "_index": ".opendistro-anomaly-detectors",
        "_type": "_doc",
        "_id": "RtZ3SX0BVwteEm6PYOnl",
        "_version": 1,
        "_seq_no": 2,
        "_primary_term": 1,
        "_score": 1,
        "_source": {
          "name": "zeek_service_AD",
          "description": "Detect anomalies based application protocol, aggregated by action and by result",
          "time_field": "firstPacket",
          "indices": [
            "arkime_sessions3-*"
          ],
          "filter_query": {
            "bool": {
              "filter": [
                {
                  "exists": {
                    "field": "network.protocol",
                    "boost": 1
                  }
                },
                {
                  "exists": {
                    "field": "event.action",
                    "boost": 1
                  }
                },
                {
                  "exists": {
                    "field": "event.result",
                    "boost": 1
                  }
                }
              ],
              "adjust_pure_negative": true,
              "boost": 1
            }
          },
          "detection_interval": {
            "period": {
              "interval": 10,
              "unit": "Minutes"
            }
          },
          "window_delay": {
            "period": {
              "interval": 1,
              "unit": "Minutes"
            }
          },
          "shingle_size": 8,
          "schema_version": 0,
          "feature_attributes": [
            {
              "feature_id": "RNZ3SX0BVwteEm6PYOnb",
              "feature_name": "event_action",
              "feature_enabled": true,
              "aggregation_query": {
                "event_action": {
                  "value_count": {
                    "field": "event.action"
                  }
                }
              }
            },
            {
              "feature_id": "RdZ3SX0BVwteEm6PYOnb",
              "feature_name": "event_result",
              "feature_enabled": true,
              "aggregation_query": {
                "event_result": {
                  "value_count": {
                    "field": "event.result"
                  }
                }
              }
            }
          ],
          "last_update_time": 1637615100133,
          "category_field": [
            "network.protocol"
          ],
          "detector_type": "MULTI_ENTITY"
        }
      }
    ]
  }
}

I’m also running OpenSearch dashboards in docker, based on opensearchproject/opensearch-dashboards:1.1.0, the only modification of any interest being that the securityDashboards has been removed.

When I navigate to the Anomaly Detectors plugin UI in dashboards, I get the following in the console:

org.opensearch.index.IndexNotFoundException: no such index [.opendistro-anomaly-detection-state]
 at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:959) ~[opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:896) ~[opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:852) ~[opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:231) ~[opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:207) ~[opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.action.search.TransportSearchAction.resolveLocalIndices(TransportSearchAction.java:634) ~[opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:662) ~[opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:541) ~[opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$3(TransportSearchAction.java:333) ~[opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:76) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:127) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:100) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:376) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:259) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:120) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:141) [opensearch-index-management-1.1.0.0.jar:1.1.0.0]
 at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.1.0.0.jar:1.1.0.0]
 at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.action.support.TransportAction.execute(TransportAction.java:168) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.action.support.TransportAction.execute(TransportAction.java:96) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:99) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:88) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:428) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.client.support.AbstractClient.search(AbstractClient.java:561) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.ad.transport.handler.ADSearchHandler.validateRole(ADSearchHandler.java:84) [opensearch-anomaly-detection-1.1.0.0.jar:1.1.0.0]
 at org.opensearch.ad.transport.handler.ADSearchHandler.search(ADSearchHandler.java:72) [opensearch-anomaly-detection-1.1.0.0.jar:1.1.0.0]
 at org.opensearch.ad.transport.SearchADTasksTransportAction.doExecute(SearchADTasksTransportAction.java:50) [opensearch-anomaly-detection-1.1.0.0.jar:1.1.0.0]
 at org.opensearch.ad.transport.SearchADTasksTransportAction.doExecute(SearchADTasksTransportAction.java:39) [opensearch-anomaly-detection-1.1.0.0.jar:1.1.0.0]
 at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:141) [opensearch-index-management-1.1.0.0.jar:1.1.0.0]
 at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.1.0.jar:1.1.0]
Anomaly detector - Unable to search detectors { Error: [index_not_found_exception] no such index [.opendistro-anomaly-detection-state], with { index=".opendistro-anomaly-detection-state" & resource.id=".opendistro-anomaly-detection-state" & resource.type="index_or_alias" & index_uuid="_na_" }
    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  status: 404,
  displayName: 'NotFound',
  message:
   '[index_not_found_exception] no such index [.opendistro-anomaly-detection-state], with { index=".opendistro-anomaly-detection-state" & resource.id=".opendistro-anomaly-detection-state" & resource.type="index_or_alias" & index_uuid="_na_" }',
  path: '/_plugins/_anomaly_detection/detectors/tasks/_search',
  query: {},
  body:
   { error:
      { root_cause: [Array],
        type: 'index_not_found_exception',
        reason: 'no such index [.opendistro-anomaly-detection-state]',
        index: '.opendistro-anomaly-detection-state',
 at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.1.0.0.jar:1.1.0.0]
 at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:190) [opensearch-1.1.0.jar:1.1.0]
        'resource.id': '.opendistro-anomaly-detection-state',
        'resource.type': 'index_or_alias',
        index_uuid: '_na_' },
     status: 404 },
  statusCode: 404,
  response:
   '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [.opendistro-anomaly-detection-state]","index":".opendistro-anomaly-detection-state","resource.id":".opendistro-anomaly-detection-state","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [.opendistro-anomaly-detection-state]","index":".opendistro-anomaly-detection-state","resource.id":".opendistro-anomaly-detection-state","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}',
  toString: [Function],
  toJSON: [Function] }
 at org.opensearch.action.support.TransportAction.execute(TransportAction.java:168) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.action.support.TransportAction.execute(TransportAction.java:96) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:99) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:88) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:428) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.ad.rest.AbstractSearchAction.lambda$prepareRequest$0(AbstractSearchAction.java:101) [opensearch-anomaly-detection-1.1.0.0.jar:1.1.0.0]
 at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:123) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.rest.RestController.dispatchRequest(RestController.java:271) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:353) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.rest.RestController.dispatchRequest(RestController.java:204) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:332) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:397) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:322) [opensearch-1.1.0.jar:1.1.0]
 at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-1.1.0.jar:1.1.0]
 at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-1.1.0.jar:1.1.0]
 at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-1.1.0.jar:1.1.0]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
172.22.8.1 - tlacuache [22/Nov/2021:21:25:51 +0000] "GET /dashboards/api/anomaly_detectors/_indices?index=*sample-* HTTP/1.1" 200 37 "https://localhost/dashboards/app/anomaly-detection-dashboards" "Mozilla/5.0 (X11; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0"
172.22.8.1 - tlacuache [22/Nov/2021:21:25:51 +0000] "GET /dashboards/api/anomaly_detectors/detectors?from=0&search=sample&indices=&size=1000&sortDirection=asc&sortField=name HTTP/1.1" 200 61 "https://localhost/dashboards/app/anomaly-detection-dashboards" "Mozilla/5.0 (X11; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0"
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.59.Final.jar:4.1.59.Final]
 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.59.Final.jar:4.1.59.Final]
 at java.lang.Thread.run(Thread.java:832) [?:?]

Seems to be complaining about .opendistro-anomaly-detection-state not existing.

Back last year I had a somewhat similar problem with OpenDistro, which may have been addressed in this bug. Does this look like the same thing? Is this a new bug or a regression, or am I just doing something wrong?

I did set logging in the opensearch container to INFO and got this on creation of my detectors:

[2021-11-23T14:30:54,382][INFO ][o.o.a.r.RestIndexAnomalyDetectorAction] [opensearch] AnomalyDetector POST action for detectorId
[2021-11-23T14:30:54,407][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Check [.opendistro-anomaly-results]'s mapping
[2021-11-23T14:30:54,408][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] We don't need to update [.opendistro-anomaly-results]'s mapping
[2021-11-23T14:30:54,409][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Mark [.opendistro-anomaly-results]'s mapping up-to-date
[2021-11-23T14:30:54,409][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Check [.opendistro-anomaly-detectors]'s mapping
[2021-11-23T14:30:54,409][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] We don't need to update [.opendistro-anomaly-detectors]'s mapping
[2021-11-23T14:30:54,410][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Mark [.opendistro-anomaly-detectors]'s mapping up-to-date
[2021-11-23T14:30:54,410][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Check [.opendistro-anomaly-detector-jobs]'s mapping
[2021-11-23T14:30:54,410][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] We don't need to update [.opendistro-anomaly-detector-jobs]'s mapping
[2021-11-23T14:30:54,410][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Mark [.opendistro-anomaly-detector-jobs]'s mapping up-to-date
[2021-11-23T14:30:54,410][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Check [.opendistro-anomaly-checkpoints]'s mapping
[2021-11-23T14:30:54,410][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] We don't need to update [.opendistro-anomaly-checkpoints]'s mapping
[2021-11-23T14:30:54,410][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Mark [.opendistro-anomaly-checkpoints]'s mapping up-to-date
[2021-11-23T14:30:54,411][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Check [.opendistro-anomaly-detection-state]'s mapping
[2021-11-23T14:30:54,411][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] We don't need to update [.opendistro-anomaly-detection-state]'s mapping
[2021-11-23T14:30:54,411][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Mark [.opendistro-anomaly-detection-state]'s mapping up-to-date
[2021-11-23T14:30:54,412][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Check [.opendistro-anomaly-results]'s setting
[2021-11-23T14:30:54,413][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Mark [.opendistro-anomaly-results]'s setting up-to-date
[2021-11-23T14:30:54,413][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Check [.opendistro-anomaly-detectors]'s setting
[2021-11-23T14:30:54,414][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Mark [.opendistro-anomaly-detectors]'s setting up-to-date
[2021-11-23T14:30:54,414][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Check [.opendistro-anomaly-detector-jobs]'s setting
[2021-11-23T14:30:54,415][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Mark [.opendistro-anomaly-detector-jobs]'s mapping up-to-date
[2021-11-23T14:30:54,415][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Check [.opendistro-anomaly-checkpoints]'s setting
[2021-11-23T14:30:54,415][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Mark [.opendistro-anomaly-checkpoints]'s setting up-to-date
[2021-11-23T14:30:54,415][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Check [.opendistro-anomaly-detection-state]'s setting
[2021-11-23T14:30:54,416][INFO ][o.o.a.i.AnomalyDetectionIndices] [opensearch] Mark [.opendistro-anomaly-detection-state]'s setting up-to-date
[2021-11-23T14:30:54,423][INFO ][o.o.a.r.h.IndexAnomalyDetectorActionHandler] [opensearch] AnomalyDetector Indices do not exist
[2021-11-23T14:30:54,430][INFO ][o.o.c.m.MetadataCreateIndexService] [opensearch] [.opendistro-anomaly-detectors] creating index, cause [api], templates [], shards [1]/[1]
[2021-11-23T14:30:54,530][INFO ][o.o.a.r.h.IndexAnomalyDetectorActionHandler] [opensearch] Created .opendistro-anomaly-detectors with mappings.
[2021-11-23T14:30:54,530][INFO ][o.o.a.r.h.IndexAnomalyDetectorActionHandler] [opensearch] prepareAnomalyDetectorIndexing called after creating indices
[2021-11-23T14:30:54,554][INFO ][o.o.a.r.RestIndexAnomalyDetectorAction] [opensearch] AnomalyDetector POST action for detectorId
[2021-11-23T14:30:54,556][INFO ][o.o.a.r.h.IndexAnomalyDetectorActionHandler] [opensearch] AnomalyDetector Indices do exist, calling prepareAnomalyDetectorIndexing
[2021-11-23T14:30:54,556][INFO ][o.o.a.r.h.IndexAnomalyDetectorActionHandler] [opensearch] prepareAnomalyDetectorIndexing called after creating indices
[2021-11-23T14:30:54,582][INFO ][o.o.a.r.RestIndexAnomalyDetectorAction] [opensearch] AnomalyDetector POST action for detectorId
[2021-11-23T14:30:54,584][INFO ][o.o.a.r.h.IndexAnomalyDetectorActionHandler] [opensearch] AnomalyDetector Indices do exist, calling prepareAnomalyDetectorIndexing
[2021-11-23T14:30:54,584][INFO ][o.o.a.r.h.IndexAnomalyDetectorActionHandler] [opensearch] prepareAnomalyDetectorIndexing called after creating indices

But I don’t see anything concerning there.

So, after more experimentation it turns out if I issue a start detector job it causes the appropriate indices to be created:

[2021-11-23T17:45:32,188][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch] JobSweeper started listening to operations on index .opendistro-anomaly-detector-jobs
[2021-11-23T17:45:32,194][INFO ][o.o.c.m.MetadataCreateIndexService] [opensearch] [.opendistro-anomaly-detector-jobs] creating index, cause [api], templates [], shards [1]/[1]
[2021-11-23T17:45:32,228][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch] JobSweeper started listening to operations on index .opendistro-anomaly-detector-jobs
[2021-11-23T17:45:32,317][INFO ][o.o.a.r.h.IndexAnomalyDetectorJobActionHandler] [opensearch] Created .opendistro-anomaly-detectors with mappings.
[2021-11-23T17:45:32,334][INFO ][o.o.c.m.MetadataCreateIndexService] [opensearch] [.opendistro-anomaly-detection-state] creating index, cause [api], templates [], shards [1]/[1]
[2021-11-23T17:45:32,434][INFO ][o.o.a.t.ADTaskManager    ] [opensearch] Created .opendistro-anomaly-detection-state with mappings.
[2021-11-23T17:45:32,484][INFO ][o.o.j.s.JobScheduler     ] [opensearch] Scheduling job id x-PkTX0Bj6KasTa5qLAc for index .opendistro-anomaly-detector-jobs .
[2021-11-23T17:45:49,106][INFO ][o.o.j.s.JobScheduler     ] [opensearch] Descheduling jobId: x-PkTX0Bj6KasTa5qLAc
[2021-11-23T17:45:49,130][INFO ][o.o.a.t.DeleteModelTransportAction] [opensearch] Delete model for x-PkTX0Bj6KasTa5qLAc
[2021-11-23T17:45:49,131][INFO ][o.o.a.t.DeleteModelTransportAction] [opensearch] Deleted model for [x-PkTX0Bj6KasTa5qLAc] with response [null] 
[2021-11-23T17:45:49,131][INFO ][o.o.a.m.CheckpointDao    ] [opensearch] Delete checkpoints of detector x-PkTX0Bj6KasTa5qLAc
[2021-11-23T17:45:49,133][INFO ][o.o.a.t.DeleteModelTransportAction] [opensearch] Finished deleting x-PkTX0Bj6KasTa5qLAc
[2021-11-23T17:45:49,134][INFO ][o.o.a.t.StopDetectorTransportAction] [opensearch] models of detector x-PkTX0Bj6KasTa5qLAc get deleted
[2021-11-23T17:45:49,134][INFO ][o.o.a.r.h.IndexAnomalyDetectorJobActionHandler] [opensearch] AD model deleted successfully for detector x-PkTX0Bj6KasTa5qLAc
[2021-11-23T17:45:49,133][INFO ][o.o.a.m.CheckpointDao    ] [opensearch] 0 checkpoints docs get deleted
[2021-11-23T17:46:18,603][ERROR][o.o.a.u.AlertingException] [opensearch] Alerting error: [.opendistro-alerting-config] IndexNotFoundException[no such index [.opendistro-alerting-config]]
[2021-11-23T17:46:44,629][DEPRECATION][o.o.d.c.m.MetadataCreateIndexService] [opensearch] index name [.opendistro-job-scheduler-lock] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices
[2021-11-23T17:46:44,633][INFO ][o.o.c.m.MetadataCreateIndexService] [opensearch] [.opendistro-job-scheduler-lock] creating index, cause [api], templates [], shards [1]/[1]
[2021-11-23T17:46:45,398][INFO ][o.o.c.m.MetadataCreateIndexService] [opensearch] [.opendistro-ism-managed-index-history-2021.11.23-1] creating index, cause [api], templates [], shards [1]/[1]
$ curl -sSL --silent -XGET http://opensearch:9200/_cat/indices?expand_wildcards=all|grep -i opendistro
yellow open .opendistro-ism-config                             -ZWqIRg1TmWu7Ef5a8PBsg 1 1    5  16  29.3kb  29.3kb
yellow open .opendistro-anomaly-detectors                      4BjM4KwRSzujOiMYjTba7Q 1 1    3   0  11.6kb  11.6kb
yellow open .opendistro-anomaly-detection-state                RO9F6EZBSiKNzsnZwI42iQ 1 1    3   0  34.2kb  34.2kb
yellow open .opendistro-anomaly-detector-jobs                  OWANynn-TPq3UWJ2HjnauA 1 1    1   0  14.9kb  14.9kb
yellow open .opendistro-job-scheduler-lock                     bIH03FKwT4KsDcIrkn-sxw 1 1    1   1    11kb    11kb
yellow open .opendistro-ism-managed-index-history-2021.11.23-1 pUs-59dlT4iT2pZS8mgtWw 1 1    1   0   7.8kb   7.8kb

This is doable as a workaround, but seems like a pain to have to start and stop a detector to have the indices set up correctly to be able to even load the anomaly detectors dashboard plugin. I think I’ll log a bug on github.

Logged on GitHub: opensearch-project/anomaly-detection-dashboards-plugin#155

Turns out this was already logged and fixed a week or so ago:

Sigh, actually it seems to be broken even worse in v1.2.0.

opensearch-project/anomaly-detection-dashboards-plugin#156

@tlacuache Have updates on this issue "indices not found"-style errors when detectors were created via API · Issue #156 · opensearch-project/anomaly-detection-dashboards-plugin · GitHub, let’s track on that issue.
Also paste updates here

hi, @mmguero, thanks for reporting this issue. The detector list page will search both detectors and anomaly results. When no detector started, then no AD result indices created on new cluster. So the backend code will throw “No indices found” exception when search anomaly results.

One workaround is you can create a test detector with custom result index. A custom result index will be created when create detector. Then the search AD result API won’t throw error and detector list can show up. This is an example of detector using custom result index opensearch-ad-plugin-result-test . Read more details on Anomaly detection - OpenSearch documentation, check this step “Specify custom result index.”

POST _plugins/_anomaly_detection/detectors
{
  "name": "test",
  "description": "Test detector",
  "time_field": "timestamp",
  "indices": [
    "server_log"
  ],
  "filter_query": {
    "bool": {
      "filter": [
        {
          "range": {
            "value": {
              "gte": 1
            }
          }
        }
      ]
    }
  },
  "detection_interval": {
    "period": {
      "interval": 1,
      "unit": "Minutes"
    }
  },
  "window_delay": {
    "period": {
      "interval": 1,
      "unit": "Minutes"
    }
  },
  "feature_attributes": [
    {
      "feature_name": "test",
      "feature_enabled": true,
      "aggregation_query": {
        "test": {
          "sum": {
            "field": "value"
          }
        }
      }
    }
  ],
  "result_index": "opensearch-ad-plugin-result-test"
}

2 Likes