Replacing certs gives error it cannot read

Security
1

Hello,

I am following the ODfE guide to replace the certs and I am getting an error that it cannot read the certs after generating self signed. I followed the guide exactly so i do not know where to go from here.

Here is the error ( Unable to read /etc/elasticsearch/self_node.pem (/etc/elasticsearch/self_node.pem). Please make sure this files exists and is readable regarding to permissions. Property: opendistro_security.ssl.transport.pemcert_filepath) and below is the whole elasticsearch.log for start up, any idea what I am doing wrong?:

[2019-09-20T10:52:58,581][ERROR][o.e.b.Bootstrap ] [LAELK02L] Exception
java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.node.Node.(Node.java:308) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.1.1.jar:7.1.1]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.1.1.jar:7.1.1]
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.1.1.jar:7.1.1]
… 15 more
Caused by: org.elasticsearch.ElasticsearchException: Unable to read /etc/elasticsearch/self_node.pem (/etc/elasticsearch/self_node.pem). Please make sure this files exists and is readable regarding to permissions. Property: opendistro_security.ssl.transport.pemcert_filepath
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.checkPath(DefaultOpenDistroSecurityKeyStore.java:820) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.resolve(DefaultOpenDistroSecurityKeyStore.java:210) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initSSLConfig(DefaultOpenDistroSecurityKeyStore.java:330) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.(DefaultOpenDistroSecurityKeyStore.java:151) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.OpenDistroSecuritySSLPlugin.(OpenDistroSecuritySSLPlugin.java:194) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin.(OpenDistroSecurityPlugin.java:212) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.1.1.jar:7.1.1]
… 15 more
[2019-09-20T10:52:58,596][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [LAELK02L] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.1.1.jar:7.1.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.1.1.jar:7.1.1]
Caused by: java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.1.1.jar:7.1.1]

    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.node.Node.<init>(Node.java:308) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.1.jar:7.1.1]
    ... 6 more

Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.1.1.jar:7.1.1]

    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.node.Node.<init>(Node.java:308) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.1.jar:7.1.1]
    ... 6 more

Caused by: org.elasticsearch.ElasticsearchException: Unable to read /etc/elasticsearch/self_node.pem (/etc/elasticsearch/self_node.pem). Please make sure this files exists and is readable regarding to permissions. Property: opendistro_security.ssl.transport.pemcert_filepath
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.checkPath(DefaultOpenDistroSecurityKeyStore.java:820) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.resolve(DefaultOpenDistroSecurityKeyStore.java:210) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initSSLConfig(DefaultOpenDistroSecurityKeyStore.java:330) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.(DefaultOpenDistroSecurityKeyStore.java:151) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.OpenDistroSecuritySSLPlugin.(OpenDistroSecuritySSLPlugin.java:194) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin.(OpenDistroSecurityPlugin.java:212) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.1.1.jar:7.1.1]

    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.node.Node.<init>(Node.java:308) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.1.jar:7.1.1]
    ... 6 more
2

Here are the perms of the files:
-rw-------. 1 root elasticsearch 1675 Sep 20 10:15 self_admin-key-temp.pem
-rw-------. 1 root elasticsearch 1704 Sep 20 10:15 self_admin-key.pem
-rw-------. 1 root elasticsearch 1054 Sep 20 10:17 self_admin.csr
-rw-------. 1 root elasticsearch 1306 Sep 20 10:17 self_admin.pem
-rw-------. 1 root elasticsearch 1679 Sep 20 10:41 self_node-key-temp.pem
-rw-------. 1 root elasticsearch 1704 Sep 20 10:41 self_node-key.pem
-rw-------. 1 root elasticsearch 1054 Sep 20 10:43 self_node.csr
-rw-------. 1 root elasticsearch 1306 Sep 20 10:44 self_node.pem
-rw-------. 1 root elasticsearch 1679 Sep 20 10:13 self_root-ca-key.pem
-rw-------. 1 root elasticsearch 1424 Sep 20 10:15 self_root-ca.pem
-rw-------. 1 root elasticsearch 17 Sep 20 10:44 self_root-ca.srl
[root@LAELK02L elasticsearch]# pwd
/etc/elasticsearch

3

ok i changed the perms to 644 and now it load them even though there is a warning to change it back to 600 perms.

Now when i try to run the security admin script it says this:
ERR: Seems you use a node certificate which is also an admin certificate

  • That may have worked with older Open Distro Security versions but it indicates*

  • a configuration error and is therefore forbidden now.*

Unable to check whether cluster is sane
ERR: An unexpected ElasticsearchSecurityException occured: No user found for cluster:monitor/nodes/info

Any idea on what this means?

4

this issue is resolved

5

Please share what resolved it so that your posts are useful to others that land here. Even if your solution might seem “simple”, it can help others.

6

@luistro

Appreciate if you drop the solution. Thanks