Hello,
I am running into some issue of the performance analyzer plugin.
The first 1 is that every minutes it dump the metrics it has collected to stdout
(I am running open distro 1.9 in docker)
Update: I am getting these logs when I upgrade a existing conatiner from 1.3 to 1.9.
On a fresh install on a new machine (no existing database) these logs dont showup. (default logs settings, no changes to log files)
EOE
[2020-08-14T10:16:14,967][INFO ][stats_log ] [d1fc2515b3d2] ------------------------------------------------------------------------
Program=PerformanceAnalyzerPlugin
rca-version=0.0.1
Metrics=
StartTime=0.000
EndTime=Fri, 14 Aug 2020 10:16:14 UTC
Time=1597400174967 msecs
Timing=total-time:1.597400174967E12/1
Counters=
EOE
[2020-08-14T10:16:14,969][INFO ][stats_log ] [d1fc2515b3d2] ------------------------------------------------------------------------
Program=PerformanceAnalyzerPlugin
rca-version=0.0.1
Metrics=
StartTime=0.000
EndTime=Fri, 14 Aug 2020 10:16:14 UTC
Time=1597400174969 msecs
Timing=total-time:1.597400174969E12/1
Counters=
EOE
[2020-08-14T10:16:14,969][INFO ][stats_log ] [d1fc2515b3d2] ----------------------------------------
I have tried to disabled the info log of the PerformanceAnalyzerPlugin by editing
/usr/share/elasticsearch/performance-analyzer-rca/pa_config/log4j2.xml
/usr/share/elasticsearch/plugins/opendistro_performance_analyzer/pa_config/log4j2.xml
By settings the log levels to FATAL or by removing the console output references.
But so far no result.
I am also seeing a lot of metricsdb_ files in /tmp.
Currectly after a day around 16000. The files are not that large (100k each). But still previous version did not do this.
-rw-r--r-- 1 elasticsearch elasticsearch 110K Aug 14 10:19 metricsdb_1597400375000
-rw-r--r-- 1 elasticsearch elasticsearch 113K Aug 14 10:20 metricsdb_1597400380000
-rw-r--r-- 1 elasticsearch elasticsearch 106K Aug 14 10:20 metricsdb_1597400385000
-rw-r--r-- 1 elasticsearch elasticsearch 106K Aug 14 10:20 metricsdb_1597400390000
-rw-r--r-- 1 elasticsearch elasticsearch 111K Aug 14 10:20 metricsdb_1597400395000
Does anyone have a any suggestions on how to fix these 2 problems?
Edit:
Current log4j2.xml
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="FATAL">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
</Console>
<File name="PerformanceAnalyzerLog" fileName="/appl/logs/elasticsearch/nodeName/PerformanceAnalyzer.log" immediateFlush="true" append="true">
<PatternLayout pattern="%d{yyy-MM-dd HH:mm:ss.SSS} [PA:Reader] [%t] %-5level %logger{36} - %msg%n"/>
</File>
<File name="StatsLog" fileName="/appl/logs/elasticsearch/nodeName/performance_analyzer_agent_stats.log" immediateFlush="true" append="true">
</File>
</Appenders>
<Loggers>
<Logger name="stats_log" level="FATAL" additivity="false">
<AppenderRef ref="StatsLog"/>
</Logger>
<Root level="FATAL">
<AppenderRef ref="Console" />
<AppenderRef ref="PerformanceAnalyzerLog"/>
</Root>
</Loggers>
</Configuration>
And performance-analyzer.properties
however this is the default that comes with the docker container.
# ======================== Elasticsearch performance analyzer plugin config =========================
# NOTE: this is an example for Linux. Please modify the config accordingly if you are using it under other OS.
# Metrics data location
metrics-location = /dev/shm/performanceanalyzer/
# Metrics deletion interval (minutes) for metrics data.
# Interval should be between 1 to 60.
metrics-deletion-interval = 1
# If set to true, the system cleans up the files behind it. So at any point, we should expect only 2
# metrics-db-file-prefix-path files. If set to false, no files are cleaned up. This can be useful, if you are archiving
# the files and wouldn't like for them to be cleaned up.
cleanup-metrics-db-files = true
# WebService exposed by App's port
webservice-listener-port = 9600
# Metric DB File Prefix Path location
metrics-db-file-prefix-path = /tmp/metricsdb_
https-enabled = false
#Setup the correct path for certificates
certificate-file-path = specify_path
private-key-file-path = specify_path
# WebService bind host; default only to local interface
#webservice-bind-host =
# Plugin Stats Metadata file name, expected to be in the same location
plugin-stats-metadata = plugin-stats-metadata
# Agent Stats Metadata file name, expected to be in the same location
agent-stats-metadata = agent-stats-metadata
