Receiving SSLHandShakeException: certificate_unkown

Scenario: I am trying to start OS server from java code and pointing to my custom config folder. So, for that I am creating my own opensearch.yaml file and modified the config in install_demo_configuration.sh file to point my generated config. This all happens using java code. Now when I try to start opensearch, it starts correctly but after a point in time it gives this error (From logs):
Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

Config file:

---------------------------------- Cluster -----------------------------------

cluster.name: MyTestCluster

------------------------------------ Node ------------------------------------

bootstrap.system_call_filter: false
node.name: node-1
node.roles: [data, master]

----------------------------------- Paths ------------------------------------

path.data: /data/opensearchdata/contentDir/data
path.logs: /data/opensearchdata/contentDir/logs

----------------------------------- Memory -----------------------------------

bootstrap.memory_lock: true

indices.breaker.total.use_real_memory: true
#indices.breaker.total.limit: 95%

---------------------------------- Network -----------------------------------

network.host: [site, localhost]
transport.tcp.port: 9300
http.port: 9200
#network.publish_host: hostname

--------------------------------- Discovery ----------------------------------

discovery.seed_hosts: [localhost:9300]
cluster.initial_master_nodes: [“node-1”]
cluster.max_shards_per_node: 6000

--------------------------------- Indices ------------------------------------

indices.query.bool.max_clause_count: 2048

---------------------------------- Gateway -----------------------------------

gateway.recover_after_nodes: 3

---------------------------------- Various -----------------------------------

node.max_local_storage_nodes: 1

---------------------------------- Thread pool -------------------------------

thread_pool.write.queue_size: 10000
thread_pool.search.queue_size: 10000

---------------------------------- Repository --------------------------------

path.repo:

#xpack.security.enabled: false

######## Start OpenSearch Security Demo Configuration ########

WARNING: revise all the lines below before you go into production

plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opendistro-notifications-”, “.opendistro-notebooks”, “.opendistro-asynchronous-search-response*”]
######## End OpenSearch Security Demo Configuration ########

@pablo can you shed some light on this issue?

@coderzzzz

If you start this cluster with the original file, does it still throw the same FATAL error?
Can you confirm that all certs and keys (esnode.pem, esnode-key.pem, root-ca.pem) are in the same config folder as opensearch.yml?

How do you create and start the cluster? Docker or binaries?

Yes, @pablo I confirmed that the certs and keys as you mentioned are in the same folder as opensearch.yml. I start cluster using binaries (running opensearch under bin/)

@coderzzzz could you share the command line which starts the node?
Is it a single node?
Do you see any preceding errors or warnings? Could you share the OpenSearch log file?

@pablo its a java command which starts opensearch… but otherwise it’s as simple as running ./opensearch. Yes its a single node. Nope no preceding warnings. I can’t share the entire log file for security reasons but will share excerpts of it:
[INFO ][o.o.i.i.ManagedIndexCoordinator] [node-1] Performing move cluster state metadata.
[2021-10-19T14:47:03,880][INFO ][o.o.i.i.MetadataService ] [node-1] Move Metadata succeed, set finish flag to true. Indices failed to get indexed: {}
[2021-10-19T14:47:03,879][INFO ][o.o.s.a.s.SinkProvider ] [node-1] Closing InternalOpenSearchSink
[2021-10-19T14:47:03,880][INFO ][o.o.i.i.MetadataService ] [node-1] There is a move metadata process running…
[2021-10-19T14:47:03,892][INFO ][o.o.s.a.s.SinkProvider ] [node-1] Closing DebugSink
[2021-10-19T14:47:04,094][INFO ][o.o.n.Node ] [node-1] stopped
[2021-10-19T14:47:04,095][INFO ][o.o.n.Node ] [node-1] closing …
[2021-10-19T14:47:04,104][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Closing AuditLogImpl
[2021-10-19T14:47:04,108][INFO ][o.o.n.Node ] [node-1] closed
[2021-10-19T14:48:17,501][INFO ][o.o.n.Node ] [node-1] version[1.0.0], pid[2455901], build[tar/34550c5b17124ddc59458ef774f6b43a086522e3/2021-07-02T23:22:21.383695Z], OS[Linux/5.4.0-37-generic/amd64], JVM[Eclipse Foundation/OpenJDK 64-Bit Server VM/11.0.12/11.0.12+7]
[2021-10-19T14:48:17,504][INFO ][o.o.n.Node ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/opensearch-12664107107897359870, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/data/opensearchdata/opensearch-1.0.1/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/data/opensearchdata/opensearch-1.0.1, -Dopensearch.path.conf=/data/opensearchdata/contentDir/etc, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2021-10-19T14:48:19,634][INFO ][o.o.s.s.t.SSLConfig ] [node-1] SSL dual mode is disabled
[2021-10-19T14:48:19,634][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] OpenSearch Config path is /data/opensearchdata/contentDir/etc
[2021-10-19T14:48:19,910][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] JVM supports TLSv1.3
[2021-10-19T14:48:19,912][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Config directory is /data/opensearchdata/contentDir/etc/, from there the key- and truststore files are resolved relatively
[2021-10-19T14:48:20,215][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS Transport Client Provider : JDK
[2021-10-19T14:48:20,216][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS Transport Server Provider : JDK
[2021-10-19T14:48:20,216][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS HTTP Provider : JDK
[2021-10-19T14:48:20,216][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2021-10-19T14:48:20,217][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2021-10-19T14:48:20,484][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Clustername: MyTestCluster
[2021-10-19T14:48:20,490][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] Directory /data/opensearchdata/contentDir/etc has insecure file permissions (should be 0700)
[2021-10-19T14:48:20,490][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /data/opensearchdata/contentDir/etc/kirk.pem has insecure file permissions (should be 0600)
[2021-10-19T14:48:20,491][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /data/opensearchdata/contentDir/etc/log4j2.properties has insecure file permissions (should be 0600)
[2021-10-19T14:48:20,491][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /data/opensearchdata/contentDir/etc/.opensearch.yml.swp has insecure file permissions (should be 0600)
[2021-10-19T14:48:20,491][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /data/opensearchdata/contentDir/etc/kirk-key.pem has insecure file permissions (should be 0600)
[2021-10-19T14:48:20,492][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /data/opensearchdata/contentDir/etc/jvm.options has insecure file permissions (should be 0600)
[2021-10-19T14:48:20,492][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /data/opensearchdata/contentDir/etc/opensearch.yml has insecure file permissions (should be 0600)
[2021-10-19T14:48:20,492][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /data/opensearchdata/contentDir/etc/esnode.pem has insecure file permissions (should be 0600)
[2021-10-19T14:48:20,493][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /data/opensearchdata/contentDir/etc/root-ca.pem has insecure file permissions (should be 0600)
[2021-10-19T14:48:20,493][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /data/opensearchdata/contentDir/etc/esnode-key.pem has insecure file permissions (should be 0600)

2021-10-19T14:48:29,419][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No endpoint configured for categories [BAD_HEADERS, FAILED_LOGIN, MISSING_PRIVILEGES, GRANTED_PRIVILEGES, OPENDISTRO_SECURITY_INDEX_ATTEMPT, SSL_EXCEPTION, AUTHENTICATED, INDEX_EVENT, COMPLIANCE_DOC_READ, COMPLIANCE_DOC_WRITE, COMPLIANCE_EXTERNAL_CONFIG, COMPLIANCE_INTERNAL_CONFIG_READ, COMPLIANCE_INTERNAL_CONFIG_WRITE], using default endpoint
[2021-10-19T14:48:29,419][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing of external configuration is disabled.
[2021-10-19T14:48:29,419][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing of internal configuration is enabled.
[2021-10-19T14:48:29,419][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing only metadata information for read request is enabled.
[2021-10-19T14:48:29,420][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing will watch {} for read requests.
[2021-10-19T14:48:29,420][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing read operation requests from kibanaserver users is disabled.
[2021-10-19T14:48:29,420][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing only metadata information for write request is enabled.
[2021-10-19T14:48:29,420][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing diffs for write requests is disabled.
[2021-10-19T14:48:29,420][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing write operation requests from kibanaserver users is disabled.
[2021-10-19T14:48:29,421][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Auditing will watch for write requests.
[2021-10-19T14:48:29,421][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] .opendistro_security is used as internal security index.
[2021-10-19T14:48:29,421][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Internal index used for posting audit logs is null
[2021-10-19T14:48:29,422][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Hot-reloading of audit configuration is enabled
[2021-10-19T14:48:29,422][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Node ‘node-1’ initialized
[2021-10-19T14:49:23,101][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:336) ~[?:?]
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1387) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1282) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.59.Final.jar:4.1.59.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.59.Final.jar:4.1.59.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]

@coderzzzz could you change network.host: [ site , localhost] to network.host: 0.0.0.0?

@pablo actually that’s pointing to my internal service hostname actually, I just replaced it here for completeness

So @pablo initially, the server hosts and is up on internal_host:9200 port but after some time it keeps giving this exception and I can’t figure out where exactly is the issue as it is a demo security configuration

@coderzzzz do you run opensearch with opensearch-dashboards? If so, could you run just opensearch and check for errors?

@pablo I ran opensearch only and then sent you these logs

You’ve stated that install_demo_configuration.sh was executed with a custom opensearch.yml file.
Have you made any changes in the certificates section of install_demo_configuration.sh?

Since this is a test environment, have you tried to use the latest version 1.1.0?

@pablo haven’t made any changes to the certificates section of install_demo_configuration.sh

I can, but will it make a difference? It was running fine till this morning actually

@pablo I tried using opensearch-1.1.0 but after making those modification changes the same issue persists. Any other suggestions?

@coderzzzz would you mind sending the modified install_demo_configuration.yml? You could alter sensitive data and message it to me directly.

If you keep this OpenSearch running for while, does the ERROR repeat? If so, how often does that?

@pablo I have sent you the file, can you tell me what’s the issue?