Helm and Security Plugin - Not Yet Initialized Errors

Hey, all. I have followed the instructions in the Helm README as well as various docs around certs and whatnot. I have the certificates installed correctly, or so it appears. I have also loaded all of the configs via secrets. I have verified in the pods that the configs are mounted correctly. However, Client still won’t become ready, as far as I can tell, and it shows these errors over and over:

[2020-05-06T14:51:42,259][ERROR][c.a.o.s.a.BackendRegistry] [es-stac-opendistro-es-client-b99f76656-h5q6k] Not yet initialized (you may need to run securityadmin)
[2020-05-06T14:51:44,759][ERROR][c.a.o.s.a.BackendRegistry] [es-stac-opendistro-es-client-b99f76656-h5q6k] Not yet initialized (you may need to run securityadmin)
[2020-05-06T14:51:47,262][ERROR][c.a.o.s.a.BackendRegistry] [es-stac-opendistro-es-client-b99f76656-h5q6k] Not yet initialized (you may need to run securityadmin)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for internalusers while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for actiongroups while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for config while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for roles while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for rolesmapping while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for tenants while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS]  (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,764][ERROR][c.a.o.s.a.BackendRegistry] [es-stac-opendistro-es-client-b99f76656-h5q6k] Not yet initialized (you may need to run securityadmin)

So I think maybe I still need to run securityadmin.sh to initialize the index. When attempting to run it, I get this error

k exec -it es-stac-opendistro-es-master-0 -- bash /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -cacert /usr/share/elasticsearch/config/admin-root-ca.pem -cert /usr/share/elasticsearch/config/admin-crt.pem -key /usr/share/elasticsearch/config/admin-key.pem -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/ -nhnv
Open Distro Security Admin v7
Will connect to localhost:9300 ... done
15:05:59.351 [elasticsearch[_client_][transport_worker][T#1]] ERROR com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLNettyTransport - Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
	at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
	at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
	at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:285) ~[?:?]
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:180) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) ~[?:?]
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?]
	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:280) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
	at java.lang.Thread.run(Thread.java:835) [?:?]
ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{Ay2tcwuoTzu1noGGv1BI3Q}{localhost}{127.0.0.1:9300}]]
	at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352)
	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248)
	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57)
	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:394)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:396)
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:385)
	at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.execute(OpenDistroSecurityAdmin.java:520)
	at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.main(OpenDistroSecurityAdmin.java:153)

I had added SANs to the certs to make them a bit more friendly. That was the only variation from this recipe

I redeployed without the SANs, and I get a new error:

[2020-05-06T15:45:13,540][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [es-stac-opendistro-es-master-0] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (1

Some brief searches shows that this is a common problem, but I am not sure what the solution is just yet.

I’m not using passwords in the certs, and I have the passphrases disabled:

  transportKeyPassphrase:
    enabled: false
    passPhrase:

  sslKeyPassphrase:
    enabled: false
    passPhrase:

That’s a red herring according to : https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/#java-error-during-startup

I have tried several variations of cert generation and subject name settings and I can’t get securityadmin.sh to connect. I keep getting the Handshake failed, and in the master node, I see

[2020-05-06T16:26:00,500][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [es-stac-opendistro-es-master-0] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Invalid CertificateVerify signature
javax.net.ssl.SSLHandshakeException: Invalid CertificateVerify signature

I would assume that this is directly relating to the DN for the admin certificate, but I can’t come up with a DN and setting for opendistro_security.authcz.admin_dn