Replacing certs gives error it cannot read

Hello,

I am following the ODfE guide to replace the certs and I am getting an error that it cannot read the certs after generating self signed. I followed the guide exactly so i do not know where to go from here.

Here is the error ( Unable to read /etc/elasticsearch/self_node.pem (/etc/elasticsearch/self_node.pem). Please make sure this files exists and is readable regarding to permissions. Property: opendistro_security.ssl.transport.pemcert_filepath) and below is the whole elasticsearch.log for start up, any idea what I am doing wrong?:

[2019-09-20T10:52:58,581][ERROR][o.e.b.Bootstrap ] [LAELK02L] Exception
java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.node.Node.(Node.java:308) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-7.1.1.jar:7.1.1]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.1.1.jar:7.1.1]
Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.1.1.jar:7.1.1]
… 15 more
Caused by: org.elasticsearch.ElasticsearchException: Unable to read /etc/elasticsearch/self_node.pem (/etc/elasticsearch/self_node.pem). Please make sure this files exists and is readable regarding to permissions. Property: opendistro_security.ssl.transport.pemcert_filepath
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.checkPath(DefaultOpenDistroSecurityKeyStore.java:820) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.resolve(DefaultOpenDistroSecurityKeyStore.java:210) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initSSLConfig(DefaultOpenDistroSecurityKeyStore.java:330) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.(DefaultOpenDistroSecurityKeyStore.java:151) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.OpenDistroSecuritySSLPlugin.(OpenDistroSecuritySSLPlugin.java:194) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin.(OpenDistroSecurityPlugin.java:212) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.1.1.jar:7.1.1]
… 15 more
[2019-09-20T10:52:58,596][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [LAELK02L] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.1.1.jar:7.1.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.1.1.jar:7.1.1]
Caused by: java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:614) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.1.1.jar:7.1.1]

    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.node.Node.<init>(Node.java:308) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.1.jar:7.1.1]
    ... 6 more

Caused by: java.lang.reflect.InvocationTargetException
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.1.1.jar:7.1.1]

    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.node.Node.<init>(Node.java:308) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.1.jar:7.1.1]
    ... 6 more

Caused by: org.elasticsearch.ElasticsearchException: Unable to read /etc/elasticsearch/self_node.pem (/etc/elasticsearch/self_node.pem). Please make sure this files exists and is readable regarding to permissions. Property: opendistro_security.ssl.transport.pemcert_filepath
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.checkPath(DefaultOpenDistroSecurityKeyStore.java:820) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.resolve(DefaultOpenDistroSecurityKeyStore.java:210) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.initSSLConfig(DefaultOpenDistroSecurityKeyStore.java:330) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.DefaultOpenDistroSecurityKeyStore.(DefaultOpenDistroSecurityKeyStore.java:151) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.OpenDistroSecuritySSLPlugin.(OpenDistroSecuritySSLPlugin.java:194) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin.(OpenDistroSecurityPlugin.java:212) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:605) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:556) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:471) ~[elasticsearch-7.1.1.jar:7.1.1]

    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:163) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.node.Node.<init>(Node.java:308) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.node.Node.<init>(Node.java:252) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.1.jar:7.1.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.1.jar:7.1.1]
    ... 6 more

Here are the perms of the files:
-rw-------. 1 root elasticsearch 1675 Sep 20 10:15 self_admin-key-temp.pem
-rw-------. 1 root elasticsearch 1704 Sep 20 10:15 self_admin-key.pem
-rw-------. 1 root elasticsearch 1054 Sep 20 10:17 self_admin.csr
-rw-------. 1 root elasticsearch 1306 Sep 20 10:17 self_admin.pem
-rw-------. 1 root elasticsearch 1679 Sep 20 10:41 self_node-key-temp.pem
-rw-------. 1 root elasticsearch 1704 Sep 20 10:41 self_node-key.pem
-rw-------. 1 root elasticsearch 1054 Sep 20 10:43 self_node.csr
-rw-------. 1 root elasticsearch 1306 Sep 20 10:44 self_node.pem
-rw-------. 1 root elasticsearch 1679 Sep 20 10:13 self_root-ca-key.pem
-rw-------. 1 root elasticsearch 1424 Sep 20 10:15 self_root-ca.pem
-rw-------. 1 root elasticsearch 17 Sep 20 10:44 self_root-ca.srl
[root@LAELK02L elasticsearch]# pwd
/etc/elasticsearch

ok i changed the perms to 644 and now it load them even though there is a warning to change it back to 600 perms.

Now when i try to run the security admin script it says this:
ERR: Seems you use a node certificate which is also an admin certificate

  • That may have worked with older Open Distro Security versions but it indicates*
    
  • a configuration error and is therefore forbidden now.*
    

Unable to check whether cluster is sane
ERR: An unexpected ElasticsearchSecurityException occured: No user found for cluster:monitor/nodes/info

Any idea on what this means?

this issue is resolved

Please share what resolved it so that your posts are useful to others that land here. Even if your solution might seem “simple”, it can help others.