I was looking at the source code of https://github.com/opendistro-for-elasticsearch/performance-analyzer quickly and I would like to ask some questions about general architecture. The Perf analyzer seems to be a native ES plugin that stores metrics into a file on disk (MetricDB) on 5 sec intervals. There is also a standalone Java daemon (PerformanceAnalyzerApp.java) that is started (for each ES node?) listening on port 9600 that is able to respond to client HTTP requests (like requests from the CLI tool) and is able to serve accumulated stats from one or more nodes. Am I at least close?
One thing that is not very clear to me is why in blog posts (like the recent https://aws.amazon.com/blogs/opensource/analyze-your-open-distro-for-elasticsearch-cluster-using-performance-analyzer-and-perftop/) it is often said that:
Performance Analyzer is an agent and REST API […] independent of the Java Virtual Machine (JVM)
What exactly do you mean by the independency? Isn’t Perf Analyzer running in the same JVM as all other ES plugins and sharing it with ES node itself? What happens when ES is under heavy load can it still collect ES metrics? May be I am missing something.