Not able to get Performance Analyzer metrics

Hi Team,

We are getting Performance Analyzer metrics like below.

Can any one please help us to get all metrics in dashboard.

Thank you.

Can you tell us a little more about what product/version you’re running?

Hi @searchymcsearchface ,

We are using “opendistroforelasticsearch-1.13.0”.

Can you please confirm any config need to be change form our end then we will do.

Please help us to get all metrics in the dashboard.

Can you confirm if this is the version used for both performance-analyzer and perftop?

Also can you specify if you are using perftop artifact for macos or linux? Additionally perftop displays metrics available in /tmp/metrics* files. Can you confirm if the unavailable metrics from the above graph is present in these files.

1 Like

@sruti1312

Yes we are using both same versions(opendistro-perf-top-1.13.0.0-linux-x64.zip & opendistroforelasticsearch-1.13.0-linux-x64.tar.gz ).

we could see the below files under /tmp path.

-rw-rw-r-- 1 elastic elastic 286720 Sep 28 10:50 metricsdb_1632806420000
-rw-rw-r-- 1 elastic elastic 286720 Sep 28 10:50 metricsdb_1632806425000
-rw-rw-r-- 1 elastic elastic 2451551835 Sep 28 10:50 PerformanceAnalyzer.log
-rw-rw-r-- 1 elastic elastic 62883818 Sep 28 10:50 performance_analyzer_agent_stats.log

But we are not able to see any data in metricsdb_* files showing as empty files.

We are getting the below error in PerformanceAnalyzer.log file.
2021-09-28 10:48:31.931 [PA:Reader] [UFEJTZ3uQHCY6WXh0hH1uQ-task-1-] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.rca.framework.api.persist.SQLParsingUtil - Eden fails to match any row in field MemType.
2021-09-28 10:48:31.931 [PA:Reader] [UFEJTZ3uQHCY6WXh0hH1uQ-task-1-] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.rca.framework.api.persist.SQLParsingUtil - Survivor fails to match any row in field MemType.
2021-09-28 10:48:31.931 [PA:Reader] [UFEJTZ3uQHCY6WXh0hH1uQ-task-1-] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.rca.store.collector.NodeConfigCollector - Metric value is NaN for resource:resource_enum: YOUNG_GEN
metric_enum: HEAP_MAX

Please let me know if any additional info required from my end.

Thanks,
Anil

The metricsdb files are deleted every 10 secs. From your output, the metricsdb files are not empty. One possibility is that the files are deleted when you open the file. Can you copy these files to another location and inspect them?

Also can you try the following commands for executing perftop? Also I would suggest having this dashboard for atleast a minute so that all the graphs start populating the metrics.

./opendistro-perf-top-linux --dashboard ClusterOverview --endpoint localhost:9600
./opendistro-perf-top-linux --dashboard NodeAnalysis --endpoint localhost:9600

Can you also add additional information about the installation used for spinning up the cluster and the type of system?

Hi @sruti1312

Thanks for updating.
We have deployed the cluster as tar ball in below link.

Even though we did follow the document which you shared got the same dashboard.

By running the below commands and opened the dashboard more than a minute, but same issues.

./opendistro-perf-top-linux --dashboard ClusterOverview --endpoint localhost:9600
./opendistro-perf-top-linux --dashboard NodeAnalysis --endpoint localhost:9600

Please help us to get all metrics in dashboard.

Thanks
Anil

Hi @sruti1312

I am also facing the similar in the performance analyzer and I tried deploying OpenSearch helm chart in my k8s cluster and below is the pod status,

[root@k8master1 robin]# kubectl get pods -n es
NAME                             READY   STATUS    RESTARTS   AGE
opensearch-cluster-master-0      1/1     Running   4          21d
opensearch-cluster-master-1      1/1     Running   4          21d
opensearch-cluster-master-2      1/1     Running   0          21d

and I tried executing the below commands inside the pod to enable performance analyzer

curl -XPOST https://opensearch-cluster-master:9200/_plugins/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k
curl -XPOST https://opensearch-cluster-master:9200/_plugins/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k

curl -k -u admin:admin -XPOST https://opensearch-cluster-master:9200/_plugins/_performanceanalyzer/batch/config -H 'Content-Type: application/json' -d '{"enabled": true}'
curl -k -u admin:admin -XPOST https://opensearch-cluster-master:9200/_plugins/_performanceanalyzer/batch/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}'
[opensearch@opensearch-cluster-master-0 ~]$ curl -k -u admin:admin -XGET https://opensearch-cluster-master:9200/_plugins/_performanceanalyzer/config
{"performanceAnalyzerEnabled":true,"rcaEnabled":true,"loggingEnabled":false,"shardsPerCollection":0,"batchMetricsEnabled":true,"batchMetricsRetentionPeriodMinutes":7}

[opensearch@opensearch-cluster-master-0 ~]$ curl -k -u admin:admin -XGET https://opensearch-cluster-master:9200/_plugins/_performanceanalyzer/cluster/config
{"currentPerformanceAnalyzerClusterState":11,"shardsPerCollection":0,"batchMetricsRetentionPeriodMinutes":7}

and then I have tried downloading the opensearch perf top linux and copied the file to inside the pod,

https://github.com/opensearch-project/perftop/releases/download/1.0.0.0/opensearch-perf-top-1.0.0.0-linux-x64.zip

I executed the below command inside the pod to see the performance of cluster,

[opensearch@opensearch-cluster-master-0 opensearch-perf-top-linux]$ ./opensearch-perf-top-linux --dashboard ClusterOverview --endpoint localhost:9600

below is the response I got in the cmd line,

image

[opensearch@opensearch-cluster-master-0 opensearch-perf-top-linux]$ ./opensearch-perf-top-linux --dashboard NodeAnalysis --endpoint localhost:9600

below is the response I got in the cmd line,

image

[opensearch@opensearch-cluster-master-0 opensearch-perf-top-linux]$ ./opensearch-perf-top-linux --dashboard ClusterThreadAnalysis --endpoint localhost:9600

below is the response I got in the cmd line,

image

Can you please help me to understand why the special characters are showing in the dashboard.

Please correct me If I am doing anything wrong in the setup.

Thanks,
Ganeshbabu R

Hi @anilmca01 and @ganeshbabu Can you add more info/details about OS you’re running perftop and opensearch on?

Hi @sruti1312

Below is the server details where k8s cluster is running.

[root@k8master1 ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

K8s node details,

[root@k8master1 ~]# kubectl get nodes -o wide
NAME          STATUS     ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
10.69.2.150   Ready      master   154d   v1.17.9   10.69.2.150   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.151   Ready      master   154d   v1.17.9   10.69.2.151   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.152   Ready      master   154d   v1.17.9   10.69.2.152   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.156   Ready      worker   154d   v1.17.9   10.69.2.156   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.157   Ready      worker   154d   v1.17.9   10.69.2.157   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.158   Ready      worker   154d   v1.17.9   10.69.2.158   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.159   NotReady   worker   154d   v1.17.9   10.69.2.159   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://Unknown
10.69.2.160   NotReady   worker   154d   v1.17.9   10.69.2.160   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.161   Ready      worker   154d   v1.17.9   10.69.2.161   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.162   Ready      worker   154d   v1.17.9   10.69.2.162   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.163   Ready      worker   154d   v1.17.9   10.69.2.163   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11
10.69.2.164   Ready      worker   154d   v1.17.9   10.69.2.164   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://19.3.11

I am running opensearch version 1.0.0 in my k8s cluster as helm charts,

[root@k8master1 ~]# helm list -n es
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                  APP VERSION
elasticsearch   es              1               2021-09-28 10:15:22.425822891 +0000 UTC deployed        opensearch-1.0.2       1.0.0
[root@k8master1 ~]# kubectl get pods -n es
NAME                             READY   STATUS    RESTARTS   AGE
opensearch-cluster-master-0      1/1     Running   4          23d
opensearch-cluster-master-1      1/1     Running   4          23d
opensearch-cluster-master-2      1/1     Running   1          23d

Opensearch version

[root@k8master1 ~]# kubectl exec -it opensearch-cluster-master-0 bash -n kafka
[opensearch@opensearch-cluster-master-0 ~]$ curl -k -u admin:admin https://opensearch-cluster-master:9200
{
  "name" : "opensearch-cluster-master-1",
  "cluster_name" : "opensearch-cluster",
  "cluster_uuid" : "npD1SyJBSymiHrKgI9KSwg",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.0.0",
    "build_type" : "tar",
    "build_hash" : "34550c5b17124ddc59458ef774f6b43a086522e3",
    "build_date" : "2021-07-02T23:22:21.383695Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

As I mentioned above I have downloaded the opensearch perf top linux from perfop releases copied inside the pod and executed the command to see the metrics.

Let me know if I doing anything wrong.

Attached the performance analyzer properties for reference,

[opensearch@opensearch-cluster-master-0 pa_config]$ cat performance-analyzer.properties
# ======================== OpenSearch performance analyzer plugin config =========================

# NOTE: this is an example for Linux. Please modify the config accordingly if you are using it under other OS.

# Metrics data location
metrics-location = /dev/shm/performanceanalyzer/

# Metrics deletion interval (minutes) for metrics data.
# Interval should be between 1 to 60.
metrics-deletion-interval = 1

# If set to true, the system cleans up the files behind it. So at any point, we should expect only 2
# metrics-db-file-prefix-path files. If set to false, no files are cleaned up. This can be useful, if you are archiving
# the files and wouldn't like for them to be cleaned up.
cleanup-metrics-db-files = true

# WebService exposed by App's port
webservice-listener-port = 9600

# Port for RPC Communication
rpc-port = 9650

# Metric DB File Prefix Path location
metrics-db-file-prefix-path = /tmp/metricsdb_

https-enabled = false

# Setup the correct path for server certificates
certificate-file-path = specify_path
private-key-file-path = specify_path
#trusted-cas-file-path = specify_path

# Setup the correct path for client certificates (by default, the client will just use the server certificates)
#client-certificate-file-path = specify_path
#client-private-key-file-path = specify_path
#client-trusted-cas-file-path = specify_path

# WebService bind host; default only to local interface
webservice-bind-host = 0.0.0.0

# Plugin Stats Metadata file name, expected to be in the same location
plugin-stats-metadata = plugin-stats-metadata

# Agent Stats Metadata file name, expected to be in the same location
agent-stats-metadata = agent-stats-metadata

Thanks,
Ganeshbabu R