Enabling performance analyzer kills ES node

#1

OS: CentOS Linux release 7.6.1810 (Core)
OpenDistro: 0.8.0-1 installed from repo

When I have performance analyzer installed my ES node refuses to start, giving me the below error. Starts fine when I yum remove opendistro-performance-analyzer

[2019-04-09T20:42:22,196][ERROR][c.a.o.e.p.o.OSGlobals    ] [dev-es-b-3] Error in reading/parsing clk.tck value: java.lang.NumberFormatException: null
java.lang.NumberFormatException: null
	at java.lang.Long.parseUnsignedLong(Long.java:680) ~[?:1.8.0_152]
	at java.lang.Long.parseUnsignedLong(Long.java:746) ~[?:1.8.0_152]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.os.OSGlobals.getScClkTckFromConfig(OSGlobals.java:65) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.os.OSGlobals.<clinit>(OSGlobals.java:43) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.os.ThreadCPU.<init>(ThreadCPU.java:139) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.os.ThreadCPU.<clinit>(ThreadCPU.java:31) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.metrics_generator.linux.LinuxOSMetricsGenerator.getPagingActivityGenerator(LinuxOSMetricsGenerator.java:56) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.OSMetricsCollector.collectMetrics(OSMetricsCollector.java:50) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.lambda$run$0(PerformanceAnalyzerMetricsCollector.java:57) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.lambda$invokePrivileged$1(PerformanceAnalyzerPlugin.java:103) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_152]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.invokePrivileged(PerformanceAnalyzerPlugin.java:101) [opendistro_performance_analyzer-0.8.0.0.jar:0.8.0.0]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.run(PerformanceAnalyzerMetricsCollector.java:57) [opendistro_performance_analyzer-0.8.0.0.jar:0.8.0.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
[2019-04-09T20:42:22,271][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [dev-es-b-3] fatal error in thread [pool-2-thread-2], exiting
java.lang.NoClassDefFoundError: com/sun/tools/attach/VirtualMachine
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.jvm.ThreadList.runAttachDump(ThreadList.java:129) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.jvm.ThreadList.lambda$runThreadDump$3(ThreadList.java:201) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.lambda$invokePrivileged$1(PerformanceAnalyzerPlugin.java:103) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.invokePrivileged(PerformanceAnalyzerPlugin.java:101) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.jvm.ThreadList.runThreadDump(ThreadList.java:201) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.jvm.ThreadList.getNativeTidMap(ThreadList.java:96) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.OSMetricsCollector.collectMetrics(OSMetricsCollector.java:56) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.lambda$run$0(PerformanceAnalyzerMetricsCollector.java:57) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.lambda$invokePrivileged$1(PerformanceAnalyzerPlugin.java:103) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.invokePrivileged(PerformanceAnalyzerPlugin.java:101) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.run(PerformanceAnalyzerMetricsCollector.java:57) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: java.lang.ClassNotFoundException: com.sun.tools.attach.VirtualMachine
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_152]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_152]
	at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814) ~[?:1.8.0_152]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_152]
	... 16 more
#2

Thanks for reporting the issue. Can you please let us know, how you are starting ES with plugin: are you using opendistro RPM or docker image or you built plugin & used plugin install script?

If it is RPM or docker, we have taken care of the following.

But if you are building plugin & installed, the following things required to set:

  1. pass clk.tck= as -D option for java (for linux: can get the value by executing: /usr/bin/getconf CLK_TCK)
  2. If using JDK 8, need to bring your tools.jar (from JDK), and have it in the class path (as specified in https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/); If using >= JDK 9, need to set: jdk.attach.allowAttachSelf=true as -D option
  3. ES security policy need to set to: java.security.policy=<ES_HOME>/plugins/opendistro_performance_analyzer/pa_config/es_security.policy as -D option

Will be adding an issue (on github) for more detailed error messages if these are not set properly

2 Likes
#3

Thanks! I was installing with the RPM though I was deploying with an existing ES config template but must have missed these reqs.

#4

Thanks for more details.

  1. For JDK 8 dependencies (tools.jar), even with RPM, it is bring your own tools.jar model; as mentioned in the documentation
  2. Surprised to see the clk.tck exception, as we made sure to set it with the RPM. Which OS you are trying?
2 Likes
#5

I’m on Centos7, I’m assuming it won’t be the RPMs problem it will be my ansible playbook using its own systemd config file. Thanks!

1 Like