TLS/SSL issue - javax.net.ssl.SSLHandshakeException

I am getting the forrlow ERRORs/WARNINGs in my logs:

2020-11-13T10:20:49,465][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:49,465][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:49,465][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:49,473][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:49972, remoteAddress=DATANODE04_P/246.802.468.187:9300}], closing connection
[2020-11-13T10:20:49,465][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:49,473][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:47688, remoteAddress=DATANODE05_P/246.802.468.188:9300}], closing connection
[2020-11-13T10:20:49,474][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:49254, remoteAddress=DATANODE02_P/246.802.468.185:9300}], closing connection
[2020-11-13T10:20:49,479][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:60736, remoteAddress=DATANODE03_P/246.802.468.186:9300}], closing connection
[2020-11-13T10:20:49,465][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:49,482][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:43392, remoteAddress=DATANODE01_P/246.802.468.184:9300}], closing connection
[2020-11-13T10:20:49,501][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
[2020-11-13T10:20:49,503][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:9300, remoteAddress=/246.802.468.188:43492}], closing connection
[2020-11-13T10:20:49,509][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
[2020-11-13T10:20:49,513][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:9300, remoteAddress=/246.802.468.184:40572}], closing connection
[2020-11-13T10:20:50,317][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:50,318][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:50,319][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:60744, remoteAddress=DATANODE03_P/246.802.468.186:9300}], closing connection
[2020-11-13T10:20:50,321][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:50,322][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:50,317][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:50,320][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:47698, remoteAddress=DATANODE05_P/246.802.468.188:9300}], closing connection
[2020-11-13T10:20:50,323][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:43408, remoteAddress=DATANODE01_P/246.802.468.184:9300}], closing connection
[2020-11-13T10:20:50,324][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:49266, remoteAddress=DATANODE02_P/246.802.468.185:9300}], closing connection
[2020-11-13T10:20:50,324][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:49976, remoteAddress=DATANODE04_P/246.802.468.187:9300}], closing connection
[2020-11-13T10:20:50,493][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
[2020-11-13T10:20:50,494][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:9300, remoteAddress=/246.802.468.188:43508}], closing connection
[2020-11-13T10:20:50,500][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
[2020-11-13T10:20:50,501][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:9300, remoteAddress=/246.802.468.184:40582}], closing connection
[2020-11-13T10:20:51,324][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:51,326][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:51,326][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:51,328][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:49270, remoteAddress=DATANODE02_P/246.802.468.185:9300}], closing connection
[2020-11-13T10:20:51,328][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:51,329][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:60762, remoteAddress=DATANODE03_P/246.802.468.186:9300}], closing connection

Here is the bottom portion of my elasticsearch.yml:

WARNING: revise all the lines below before you go into production

opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.transport.resolve_hostname: false
opendistro_security.ssl.transport.pemcert_filepath: master001.pem
opendistro_security.ssl.transport.pemkey_filepath: master001.key
opendistro_security.ssl.transport.pemtrustedcas_filepath: ca-chain-bundle.pem
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: master001.pem
opendistro_security.ssl.http.pemkey_filepath: master001.key
opendistro_security.ssl.http.pemtrustedcas_filepath: ca-chain-bundle.pem
opendistro_security.allow_unsafe_democertificates: false
opendistro_security.allow_default_init_securityindex: true
opendistro_security.ssl.http.enabled_protocols:

  • “TLSv1.2”
  • “TLSv1.3”
    opendistro_security.ssl.http.enabled_ciphers:
  • “TLS_AES_256_GCM_SHA384”
  • “TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256”
  • “TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384”
  • “TLS_DHE_RSA_WITH_AES_256_GCM_SHA384”
  • “TLS_DHE_RSA_WITH_AES_128_GCM_SHA256”
    opendistro_security.authcz.admin_dn:
  • ‘O=SomeOrg1,O=SomeOrg2,L=SomeCity,S=SomeState,C=SomeCountry’
    opendistro_security.nodes_dn:
  • 'CN= NODE
    opendistro_security.ssl.transport.enabled_protocols:
  • “TLSv1.2”
  • “TLSv1.3”
    opendistro_security.ssl.transport.enabled_ciphers:
  • “TLS_AES_256_GCM_SHA384”
  • “TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256”
  • “TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384”
  • “TLS_DHE_RSA_WITH_AES_256_GCM_SHA384”
  • “TLS_DHE_RSA_WITH_AES_128_GCM_SHA256”
    opendistro_security.ssl.http.clientauth_mode: OPTIONAL
    cluster.routing.allocation.disk.threshold_enabled: true
    opendistro_security.audit.type: internal_elasticsearch
    opendistro_security.enable_snapshot_restore_privilege: true
    opendistro_security.check_snapshot_restore_write_privileges: true
    opendistro_security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
    opendistro_security.system_indices.enabled: true
    opendistro_security.system_indices.indices: [".opendistro-alerting-config", “.opendistro-alerting-alert*”]

The master01.pem and master.key were create via a csr to my corporate CA – I am not using an internal elasticsearch CA.

Seems to suggest that this is a keystore issue. However, my understanding per

Is that using a keystore is optional. I have my RootCA and my IssuingCA certs concatenated in ca-chain-bundle.pem

I created keystore using these directions:

and the bottom of this link to create the truststore:

This seems to have solved the issue.

I am still seeing errors:

Exception during establishing a SSL connection: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

Okay for ODFE version 1.11 - the keystore / truststore instruction I posted previously work. However, I found through another issue I posted about regarding another issue - you need to add this to this config file:

opendistro_security.ssl.http.keystore_keypassword

And whatever the password that was created.

I thought I had this solved - but still seeing errors

PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors