Performance Analyzer settings problems

Hello,
I am running into some issue of the performance analyzer plugin.
The first 1 is that every minutes it dump the metrics it has collected to stdout
(I am running open distro 1.9 in docker)
Update: I am getting these logs when I upgrade a existing conatiner from 1.3 to 1.9.
On a fresh install on a new machine (no existing database) these logs dont showup. (default logs settings, no changes to log files)

EOE
[2020-08-14T10:16:14,967][INFO ][stats_log                ] [d1fc2515b3d2] ------------------------------------------------------------------------
Program=PerformanceAnalyzerPlugin
rca-version=0.0.1
Metrics=
StartTime=0.000
EndTime=Fri, 14 Aug 2020 10:16:14 UTC
Time=1597400174967 msecs
Timing=total-time:1.597400174967E12/1
Counters=
EOE
[2020-08-14T10:16:14,969][INFO ][stats_log                ] [d1fc2515b3d2] ------------------------------------------------------------------------
Program=PerformanceAnalyzerPlugin
rca-version=0.0.1
Metrics=
StartTime=0.000
EndTime=Fri, 14 Aug 2020 10:16:14 UTC
Time=1597400174969 msecs
Timing=total-time:1.597400174969E12/1
Counters=
EOE
[2020-08-14T10:16:14,969][INFO ][stats_log                ] [d1fc2515b3d2] ----------------------------------------

I have tried to disabled the info log of the PerformanceAnalyzerPlugin by editing
/usr/share/elasticsearch/performance-analyzer-rca/pa_config/log4j2.xml
/usr/share/elasticsearch/plugins/opendistro_performance_analyzer/pa_config/log4j2.xml

By settings the log levels to FATAL or by removing the console output references.
But so far no result.

I am also seeing a lot of metricsdb_ files in /tmp.
Currectly after a day around 16000. The files are not that large (100k each). But still previous version did not do this.

-rw-r--r-- 1 elasticsearch elasticsearch 110K Aug 14 10:19 metricsdb_1597400375000
-rw-r--r-- 1 elasticsearch elasticsearch 113K Aug 14 10:20 metricsdb_1597400380000
-rw-r--r-- 1 elasticsearch elasticsearch 106K Aug 14 10:20 metricsdb_1597400385000
-rw-r--r-- 1 elasticsearch elasticsearch 106K Aug 14 10:20 metricsdb_1597400390000
-rw-r--r-- 1 elasticsearch elasticsearch 111K Aug 14 10:20 metricsdb_1597400395000

Does anyone have a any suggestions on how to fix these 2 problems?

Edit:
Current log4j2.xml

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="FATAL">
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
        </Console>
        <File name="PerformanceAnalyzerLog" fileName="/appl/logs/elasticsearch/nodeName/PerformanceAnalyzer.log" immediateFlush="true" append="true">
            <PatternLayout pattern="%d{yyy-MM-dd HH:mm:ss.SSS} [PA:Reader] [%t] %-5level %logger{36} - %msg%n"/>
        </File>
        <File name="StatsLog" fileName="/appl/logs/elasticsearch/nodeName/performance_analyzer_agent_stats.log" immediateFlush="true" append="true">
        </File>
    </Appenders>
    <Loggers>
        <Logger name="stats_log" level="FATAL" additivity="false">
            <AppenderRef ref="StatsLog"/>
        </Logger>
        <Root level="FATAL">
            <AppenderRef ref="Console" />
            <AppenderRef ref="PerformanceAnalyzerLog"/>
        </Root>
    </Loggers>
</Configuration>

And performance-analyzer.properties
however this is the default that comes with the docker container.

# ======================== Elasticsearch performance analyzer plugin config =========================

# NOTE: this is an example for Linux. Please modify the config accordingly if you are using it under other OS.

# Metrics data location
metrics-location = /dev/shm/performanceanalyzer/

# Metrics deletion interval (minutes) for metrics data.
# Interval should be between 1 to 60.
metrics-deletion-interval = 1

# If set to true, the system cleans up the files behind it. So at any point, we should expect only 2
# metrics-db-file-prefix-path files. If set to false, no files are cleaned up. This can be useful, if you are archiving
# the files and wouldn't like for them to be cleaned up.
cleanup-metrics-db-files = true

# WebService exposed by App's port
webservice-listener-port = 9600

# Metric DB File Prefix Path location
metrics-db-file-prefix-path = /tmp/metricsdb_

https-enabled = false

#Setup the correct path for certificates
certificate-file-path = specify_path

private-key-file-path = specify_path

# WebService bind host; default only to local interface
#webservice-bind-host =

# Plugin Stats Metadata file name, expected to be in the same location
plugin-stats-metadata = plugin-stats-metadata

# Agent Stats Metadata file name, expected to be in the same location
agent-stats-metadata = agent-stats-metadata

I have now upgraded to 1.8.
So far everything looks good and i am not having the same issues as above.

I found the same problems in 1.9. Hosts filesystems went full because of container’s /tmp/metricsdb_* files.

Had to downgrade to 1.8 :neutral_face:

Related issue: https://github.com/opendistro-for-elasticsearch/performance-analyzer/issues/178