Hi Team,
We are getting Performance Analyzer metrics like below.
Can any one please help us to get all metrics in dashboard.
Thank you.
Can you tell us a little more about what product/version you’re running?
Hi @searchymcsearchface ,
We are using “opendistroforelasticsearch-1.13.0”.
Can you please confirm any config need to be change form our end then we will do.
Please help us to get all metrics in the dashboard.
Can you confirm if this is the version used for both performance-analyzer and perftop?
Also can you specify if you are using perftop artifact for macos or linux? Additionally perftop displays metrics available in /tmp/metrics* files. Can you confirm if the unavailable metrics from the above graph is present in these files.
1 Like
Yes we are using both same versions(opendistro-perf-top-1.13.0.0-linux-x64.zip & opendistroforelasticsearch-1.13.0-linux-x64.tar.gz ).
we could see the below files under /tmp path.
-rw-rw-r-- 1 elastic elastic 286720 Sep 28 10:50 metricsdb_1632806420000
-rw-rw-r-- 1 elastic elastic 286720 Sep 28 10:50 metricsdb_1632806425000
-rw-rw-r-- 1 elastic elastic 2451551835 Sep 28 10:50 PerformanceAnalyzer.log
-rw-rw-r-- 1 elastic elastic 62883818 Sep 28 10:50 performance_analyzer_agent_stats.log
But we are not able to see any data in metricsdb_* files showing as empty files.
We are getting the below error in PerformanceAnalyzer.log file.
2021-09-28 10:48:31.931 [PA:Reader] [UFEJTZ3uQHCY6WXh0hH1uQ-task-1-] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.rca.framework.api.persist.SQLParsingUtil - Eden fails to match any row in field MemType.
2021-09-28 10:48:31.931 [PA:Reader] [UFEJTZ3uQHCY6WXh0hH1uQ-task-1-] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.rca.framework.api.persist.SQLParsingUtil - Survivor fails to match any row in field MemType.
2021-09-28 10:48:31.931 [PA:Reader] [UFEJTZ3uQHCY6WXh0hH1uQ-task-1-] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.rca.store.collector.NodeConfigCollector - Metric value is NaN for resource:resource_enum: YOUNG_GEN
metric_enum: HEAP_MAX
Please let me know if any additional info required from my end.
Thanks,
Anil
The metricsdb files are deleted every 10 secs. From your output, the metricsdb files are not empty. One possibility is that the files are deleted when you open the file. Can you copy these files to another location and inspect them?
Also can you try the following commands for executing perftop? Also I would suggest having this dashboard for atleast a minute so that all the graphs start populating the metrics.
./opendistro-perf-top-linux --dashboard ClusterOverview --endpoint localhost:9600
./opendistro-perf-top-linux --dashboard NodeAnalysis --endpoint localhost:9600
Can you also add additional information about the installation used for spinning up the cluster and the type of system?
Hi @sruti1312
Thanks for updating.
We have deployed the cluster as tar ball in below link.
https://opendistro.github.io/for-elasticsearch/downloads.html
Even though we did follow the document which you shared got the same dashboard.
By running the below commands and opened the dashboard more than a minute, but same issues.
./opendistro-perf-top-linux --dashboard ClusterOverview --endpoint localhost:9600
./opendistro-perf-top-linux --dashboard NodeAnalysis --endpoint localhost:9600
Please help us to get all metrics in dashboard.
Thanks
Anil
Hi @sruti1312
I am also facing the similar in the performance analyzer and I tried deploying OpenSearch helm chart in my k8s cluster and below is the pod status,
[root@k8master1 robin]# kubectl get pods -n es
NAME READY STATUS RESTARTS AGE
opensearch-cluster-master-0 1/1 Running 4 21d
opensearch-cluster-master-1 1/1 Running 4 21d
opensearch-cluster-master-2 1/1 Running 0 21d
and I tried executing the below commands inside the pod to enable performance analyzer
curl -XPOST https://opensearch-cluster-master:9200/_plugins/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k
curl -XPOST https://opensearch-cluster-master:9200/_plugins/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin' -k
curl -k -u admin:admin -XPOST https://opensearch-cluster-master:9200/_plugins/_performanceanalyzer/batch/config -H 'Content-Type: application/json' -d '{"enabled": true}'
curl -k -u admin:admin -XPOST https://opensearch-cluster-master:9200/_plugins/_performanceanalyzer/batch/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}'
[opensearch@opensearch-cluster-master-0 ~]$ curl -k -u admin:admin -XGET https://opensearch-cluster-master:9200/_plugins/_performanceanalyzer/config
{"performanceAnalyzerEnabled":true,"rcaEnabled":true,"loggingEnabled":false,"shardsPerCollection":0,"batchMetricsEnabled":true,"batchMetricsRetentionPeriodMinutes":7}
[opensearch@opensearch-cluster-master-0 ~]$ curl -k -u admin:admin -XGET https://opensearch-cluster-master:9200/_plugins/_performanceanalyzer/cluster/config
{"currentPerformanceAnalyzerClusterState":11,"shardsPerCollection":0,"batchMetricsRetentionPeriodMinutes":7}
and then I have tried downloading the opensearch perf top linux and copied the file to inside the pod,
I executed the below command inside the pod to see the performance of cluster,
[opensearch@opensearch-cluster-master-0 opensearch-perf-top-linux]$ ./opensearch-perf-top-linux --dashboard ClusterOverview --endpoint localhost:9600
below is the response I got in the cmd line,
[opensearch@opensearch-cluster-master-0 opensearch-perf-top-linux]$ ./opensearch-perf-top-linux --dashboard NodeAnalysis --endpoint localhost:9600
below is the response I got in the cmd line,
[opensearch@opensearch-cluster-master-0 opensearch-perf-top-linux]$ ./opensearch-perf-top-linux --dashboard ClusterThreadAnalysis --endpoint localhost:9600
below is the response I got in the cmd line,
Can you please help me to understand why the special characters are showing in the dashboard.
Please correct me If I am doing anything wrong in the setup.
Thanks,
Ganeshbabu R
Hi @anilmca01 and @ganeshbabu Can you add more info/details about OS you’re running perftop and opensearch on?
Hi @sruti1312
Below is the server details where k8s cluster is running.
[root@k8master1 ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
K8s node details,
[root@k8master1 ~]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
10.69.2.150 Ready master 154d v1.17.9 10.69.2.150 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://19.3.11
10.69.2.151 Ready master 154d v1.17.9 10.69.2.151 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://19.3.11
10.69.2.152 Ready master 154d v1.17.9 10.69.2.152 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://19.3.11
10.69.2.156 Ready worker 154d v1.17.9 10.69.2.156 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://19.3.11
10.69.2.157 Ready worker 154d v1.17.9 10.69.2.157 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://19.3.11
10.69.2.158 Ready worker 154d v1.17.9 10.69.2.158 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://19.3.11
10.69.2.159 NotReady worker 154d v1.17.9 10.69.2.159 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://Unknown
10.69.2.160 NotReady worker 154d v1.17.9 10.69.2.160 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://19.3.11
10.69.2.161 Ready worker 154d v1.17.9 10.69.2.161 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://19.3.11
10.69.2.162 Ready worker 154d v1.17.9 10.69.2.162 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://19.3.11
10.69.2.163 Ready worker 154d v1.17.9 10.69.2.163 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://19.3.11
10.69.2.164 Ready worker 154d v1.17.9 10.69.2.164 <none> CentOS Linux 7 (Core) 3.10.0-957.el7.x86_64 docker://19.3.11
I am running opensearch version 1.0.0 in my k8s cluster as helm charts,
[root@k8master1 ~]# helm list -n es
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
elasticsearch es 1 2021-09-28 10:15:22.425822891 +0000 UTC deployed opensearch-1.0.2 1.0.0
[root@k8master1 ~]# kubectl get pods -n es
NAME READY STATUS RESTARTS AGE
opensearch-cluster-master-0 1/1 Running 4 23d
opensearch-cluster-master-1 1/1 Running 4 23d
opensearch-cluster-master-2 1/1 Running 1 23d
Opensearch version
[root@k8master1 ~]# kubectl exec -it opensearch-cluster-master-0 bash -n kafka
[opensearch@opensearch-cluster-master-0 ~]$ curl -k -u admin:admin https://opensearch-cluster-master:9200
{
"name" : "opensearch-cluster-master-1",
"cluster_name" : "opensearch-cluster",
"cluster_uuid" : "npD1SyJBSymiHrKgI9KSwg",
"version" : {
"distribution" : "opensearch",
"number" : "1.0.0",
"build_type" : "tar",
"build_hash" : "34550c5b17124ddc59458ef774f6b43a086522e3",
"build_date" : "2021-07-02T23:22:21.383695Z",
"build_snapshot" : false,
"lucene_version" : "8.8.2",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}
As I mentioned above I have downloaded the opensearch perf top linux from perfop releases copied inside the pod and executed the command to see the metrics.
Let me know if I doing anything wrong.
Attached the performance analyzer properties for reference,
[opensearch@opensearch-cluster-master-0 pa_config]$ cat performance-analyzer.properties
# ======================== OpenSearch performance analyzer plugin config =========================
# NOTE: this is an example for Linux. Please modify the config accordingly if you are using it under other OS.
# Metrics data location
metrics-location = /dev/shm/performanceanalyzer/
# Metrics deletion interval (minutes) for metrics data.
# Interval should be between 1 to 60.
metrics-deletion-interval = 1
# If set to true, the system cleans up the files behind it. So at any point, we should expect only 2
# metrics-db-file-prefix-path files. If set to false, no files are cleaned up. This can be useful, if you are archiving
# the files and wouldn't like for them to be cleaned up.
cleanup-metrics-db-files = true
# WebService exposed by App's port
webservice-listener-port = 9600
# Port for RPC Communication
rpc-port = 9650
# Metric DB File Prefix Path location
metrics-db-file-prefix-path = /tmp/metricsdb_
https-enabled = false
# Setup the correct path for server certificates
certificate-file-path = specify_path
private-key-file-path = specify_path
#trusted-cas-file-path = specify_path
# Setup the correct path for client certificates (by default, the client will just use the server certificates)
#client-certificate-file-path = specify_path
#client-private-key-file-path = specify_path
#client-trusted-cas-file-path = specify_path
# WebService bind host; default only to local interface
webservice-bind-host = 0.0.0.0
# Plugin Stats Metadata file name, expected to be in the same location
plugin-stats-metadata = plugin-stats-metadata
# Agent Stats Metadata file name, expected to be in the same location
agent-stats-metadata = agent-stats-metadata
Thanks,
Ganeshbabu R