Not able to get Performance Analyzer metrics

Hi Team,

We are getting Performance Analyzer metrics like below.

Can any one please help us to get all metrics in dashboard.

Thank you.

Can you tell us a little more about what product/version you’re running?

Hi @searchymcsearchface ,

We are using “opendistroforelasticsearch-1.13.0”.

Can you please confirm any config need to be change form our end then we will do.

Please help us to get all metrics in the dashboard.

Can you confirm if this is the version used for both performance-analyzer and perftop?

Also can you specify if you are using perftop artifact for macos or linux? Additionally perftop displays metrics available in /tmp/metrics* files. Can you confirm if the unavailable metrics from the above graph is present in these files.

1 Like

@sruti1312

Yes we are using both same versions(opendistro-perf-top-1.13.0.0-linux-x64.zip & opendistroforelasticsearch-1.13.0-linux-x64.tar.gz ).

we could see the below files under /tmp path.

-rw-rw-r-- 1 elastic elastic 286720 Sep 28 10:50 metricsdb_1632806420000
-rw-rw-r-- 1 elastic elastic 286720 Sep 28 10:50 metricsdb_1632806425000
-rw-rw-r-- 1 elastic elastic 2451551835 Sep 28 10:50 PerformanceAnalyzer.log
-rw-rw-r-- 1 elastic elastic 62883818 Sep 28 10:50 performance_analyzer_agent_stats.log

But we are not able to see any data in metricsdb_* files showing as empty files.

We are getting the below error in PerformanceAnalyzer.log file.
2021-09-28 10:48:31.931 [PA:Reader] [UFEJTZ3uQHCY6WXh0hH1uQ-task-1-] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.rca.framework.api.persist.SQLParsingUtil - Eden fails to match any row in field MemType.
2021-09-28 10:48:31.931 [PA:Reader] [UFEJTZ3uQHCY6WXh0hH1uQ-task-1-] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.rca.framework.api.persist.SQLParsingUtil - Survivor fails to match any row in field MemType.
2021-09-28 10:48:31.931 [PA:Reader] [UFEJTZ3uQHCY6WXh0hH1uQ-task-1-] ERROR com.amazon.opendistro.elasticsearch.performanceanalyzer.rca.store.collector.NodeConfigCollector - Metric value is NaN for resource:resource_enum: YOUNG_GEN
metric_enum: HEAP_MAX

Please let me know if any additional info required from my end.

Thanks,
Anil

The metricsdb files are deleted every 10 secs. From your output, the metricsdb files are not empty. One possibility is that the files are deleted when you open the file. Can you copy these files to another location and inspect them?

Also can you try the following commands for executing perftop? Also I would suggest having this dashboard for atleast a minute so that all the graphs start populating the metrics.

./opendistro-perf-top-linux --dashboard ClusterOverview --endpoint localhost:9600
./opendistro-perf-top-linux --dashboard NodeAnalysis --endpoint localhost:9600

Can you also add additional information about the installation used for spinning up the cluster and the type of system?

Hi @sruti1312

Thanks for updating.
We have deployed the cluster as tar ball in below link.

Even though we did follow the document which you shared got the same dashboard.

By running the below commands and opened the dashboard more than a minute, but same issues.

./opendistro-perf-top-linux --dashboard ClusterOverview --endpoint localhost:9600
./opendistro-perf-top-linux --dashboard NodeAnalysis --endpoint localhost:9600

Please help us to get all metrics in dashboard.

Thanks
Anil