Helm and Security Plugin - Not Yet Initialized Errors

Hey, all. I have followed the instructions in the Helm README as well as various docs around certs and whatnot. I have the certificates installed correctly, or so it appears. I have also loaded all of the configs via secrets. I have verified in the pods that the configs are mounted correctly. However, Client still won’t become ready, as far as I can tell, and it shows these errors over and over:

[2020-05-06T14:51:42,259][ERROR][c.a.o.s.a.BackendRegistry] [es-stac-opendistro-es-client-b99f76656-h5q6k] Not yet initialized (you may need to run securityadmin)
[2020-05-06T14:51:44,759][ERROR][c.a.o.s.a.BackendRegistry] [es-stac-opendistro-es-client-b99f76656-h5q6k] Not yet initialized (you may need to run securityadmin)
[2020-05-06T14:51:47,262][ERROR][c.a.o.s.a.BackendRegistry] [es-stac-opendistro-es-client-b99f76656-h5q6k] Not yet initialized (you may need to run securityadmin)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for internalusers while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for actiongroups while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for config while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for roles while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for rolesmapping while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for tenants while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,764][ERROR][c.a.o.s.a.BackendRegistry] [es-stac-opendistro-es-client-b99f76656-h5q6k] Not yet initialized (you may need to run securityadmin)

So I think maybe I still need to run securityadmin.sh to initialize the index. When attempting to run it, I get this error

k exec -it es-stac-opendistro-es-master-0 -- bash /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -cacert /usr/share/elasticsearch/config/admin-root-ca.pem -cert /usr/share/elasticsearch/config/admin-crt.pem -key /usr/share/elasticsearch/config/admin-key.pem -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/ -nhnv
Open Distro Security Admin v7
Will connect to localhost:9300 ... done
15:05:59.351 [elasticsearch[_client_][transport_worker][T#1]] ERROR com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLNettyTransport - Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
	at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
	at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
	at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:285) ~[?:?]
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:180) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) ~[?:?]
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?]
	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:280) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
	at java.lang.Thread.run(Thread.java:835) [?:?]
ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{Ay2tcwuoTzu1noGGv1BI3Q}{localhost}{127.0.0.1:9300}]]
	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352)
	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248)
	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57)
	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:394)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:396)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:385)
	at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.execute(OpenDistroSecurityAdmin.java:520)
	at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.main(OpenDistroSecurityAdmin.java:153)

I had added SANs to the certs to make them a bit more friendly. That was the only variation from this recipe

I redeployed without the SANs, and I get a new error:

[2020-05-06T15:45:13,540][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [es-stac-opendistro-es-master-0] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (1

Some brief searches shows that this is a common problem, but I am not sure what the solution is just yet.

I’m not using passwords in the certs, and I have the passphrases disabled:

  transportKeyPassphrase:
    enabled: false
    passPhrase:

  sslKeyPassphrase:
    enabled: false
    passPhrase:

That’s a red herring according to : Troubleshoot - Open Distro Documentation

I have tried several variations of cert generation and subject name settings and I can’t get securityadmin.sh to connect. I keep getting the Handshake failed, and in the master node, I see

[2020-05-06T16:26:00,500][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [es-stac-opendistro-es-master-0] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Invalid CertificateVerify signature
javax.net.ssl.SSLHandshakeException: Invalid CertificateVerify signature

I would assume that this is directly relating to the DN for the admin certificate, but I can’t come up with a DN and setting for opendistro_security.authcz.admin_dn

Running into the same issue on 1.9.0. Did you figure it out?

Trying it with curl against port 9300 returns
curl: (35) SSL peer was unable to negotiate an acceptable set of security parameters.

I don’t think it’s the DN. The error changes when I change mine to be wrong.

LOL My problem was that my root ca cert did not show the authorityKeyIdentifier. Turns out the order of extensions matters a lot. Once i switched subjectKeyIdentifier to be first and authorityKeyIdentifier to be second, the key showed up on the cert and securityadmin.sh will run…

@rick_asi if you are still having this issue, did you try to run default config for opendistro and examining the certificates that get generated. This might point you in the right direction when you compare the DN of the admin cert and the opendistro_security.authcz.admin_dn setting.