Hey, all. I have followed the instructions in the Helm README as well as various docs around certs and whatnot. I have the certificates installed correctly, or so it appears. I have also loaded all of the configs via secrets. I have verified in the pods that the configs are mounted correctly. However, Client still won’t become ready, as far as I can tell, and it shows these errors over and over:
[2020-05-06T14:51:42,259][ERROR][c.a.o.s.a.BackendRegistry] [es-stac-opendistro-es-client-b99f76656-h5q6k] Not yet initialized (you may need to run securityadmin)
[2020-05-06T14:51:44,759][ERROR][c.a.o.s.a.BackendRegistry] [es-stac-opendistro-es-client-b99f76656-h5q6k] Not yet initialized (you may need to run securityadmin)
[2020-05-06T14:51:47,262][ERROR][c.a.o.s.a.BackendRegistry] [es-stac-opendistro-es-client-b99f76656-h5q6k] Not yet initialized (you may need to run securityadmin)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for internalusers while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS] (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for actiongroups while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS] (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for config while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS] (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for roles while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS] (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for rolesmapping while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS] (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,597][WARN ][c.a.o.s.c.ConfigurationLoaderSecurity7] [es-stac-opendistro-es-client-b99f76656-h5q6k] No data for tenants while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS] (index=.opendistro_security and type=null)
[2020-05-06T14:51:49,764][ERROR][c.a.o.s.a.BackendRegistry] [es-stac-opendistro-es-client-b99f76656-h5q6k] Not yet initialized (you may need to run securityadmin)
So I think maybe I still need to run securityadmin.sh to initialize the index. When attempting to run it, I get this error
k exec -it es-stac-opendistro-es-master-0 -- bash /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -cacert /usr/share/elasticsearch/config/admin-root-ca.pem -cert /usr/share/elasticsearch/config/admin-crt.pem -key /usr/share/elasticsearch/config/admin-key.pem -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/ -nhnv
Open Distro Security Admin v7
Will connect to localhost:9300 ... done
15:05:59.351 [elasticsearch[_client_][transport_worker][T#1]] ERROR com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLNettyTransport - Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:285) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:180) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:280) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1332) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) ~[netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281) ~[netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:835) [?:?]
ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{Ay2tcwuoTzu1noGGv1BI3Q}{localhost}{127.0.0.1:9300}]]
at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248)
at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57)
at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:394)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:396)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:385)
at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.execute(OpenDistroSecurityAdmin.java:520)
at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.main(OpenDistroSecurityAdmin.java:153)