Error when starting a single-node opensearch on docker

Hello everyone!

Before opening an issue on GitHub, I wanted to see if there may be something I missed on my end, as I’m having trouble starting a simple OpenSearch on Docker.

Here’s my docker run command:

docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" opensearchproject/opensearch:1.0.0

And here’s the result (infinite looping) log:

[2021-07-28T11:17:22,451][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [350fac761c4a] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206c6f63616c686f73743a393230300d0a557365722d4167656e743a20476f2d687474702d636c69656e742f312e310d0a4163636570743a206170706c69636174696f6e2f6a736f6e0d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206c6f63616c686f73743a393230300d0a557365722d4167656e743a20476f2d687474702d636c69656e742f312e310d0a4163636570743a206170706c69636174696f6e2f6a736f6e0d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1261) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.59.Final.jar:4.1.59.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]

OpenSearch seems to be running though, as I can curl it without problems, but I cannot start a development OpenSearch Dashboard though, it gives me this log:

server log [11:17:14.260] [error][data][opensearch] [ConnectionError]: socket hang up

I haven’t tried it recently, but afaik I had the same problem using the sample docker-compose.

Is anyone else having this problem?

Thanks :slight_smile:

Docker version: 20.10.7
Docker Compose version: 1.29.2
System: Ubuntu 20.04

Update on the issue:

I managed to start a dev OSD (I still have the constant exception on OpenSearch though), but I think we should update the documentation or the default config values of opensearch_dashboards.yml.

Considering that the defaults of the docker image are to set the admin:admin credentials and enable ssl, I think we should match these settings in the OSD defaults:

diff --git a/config/opensearch_dashboards.yml b/config/opensearch_dashboards.yml
index d281b987ee..65aa1df2e4 100644
--- a/config/opensearch_dashboards.yml
+++ b/config/opensearch_dashboards.yml
@@ -25,7 +25,7 @@
 #server.name: "your-hostname"

 # The URLs of the OpenSearch instances to use for all your queries.
-#opensearch.hosts: ["http://localhost:9200"]
+#opensearch.hosts: ["https://localhost:9200"]
 
 # OpenSearch Dashboards uses an index in OpenSearch to store saved searches, visualizations and
 # dashboards. OpenSearch Dashboards creates a new index if the index doesn't already exist.
@@ -48,8 +48,8 @@
 # the username and password that the OpenSearch Dashboards server uses to perform maintenance on the OpenSearch Dashboards
 # index at startup. Your OpenSearch Dashboards users still need to authenticate with OpenSearch, which
 # is proxied through the OpenSearch Dashboards server.
-#opensearch.username: "opensearch_dashboards_system"
-#opensearch.password: "pass"
+#opensearch.username: "admin"
+#opensearch.password: "admin"
 
 # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
 # These settings enable SSL for outgoing requests from the OpenSearch Dashboards server to the browser.
@@ -68,7 +68,7 @@
 #opensearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
 
 # To disregard the validity of SSL certificates, change this setting's value to 'none'.
-#opensearch.ssl.verificationMode: full
+#opensearch.ssl.verificationMode: none

What do you think ? I will surely open an issue for this case.

Hey @closingin - welcome!

That all seems a bit weird. The log you’re posted in the first message - is that coming from the OpenSearch docker logs?

The docker-compose should work out-of-the-box - to confirm, you’re using this docker compose file, right?

https://opensearch.org/samples/docker-compose.yml

Thanks for your reply :slight_smile:

So yes and yes for your questions.

Technically, the docker-compose works out-of-the-box, as I can launch it and get access to OSD without any trouble, but the exception logs still spawn.

For the “dev” OSD I speak about in my messages, I mean that I cloned the repo and yarn start-ed it. In this case it doesn’t work with a dockerized OS if I don’t apply the changes I posted above.

OK. Gotcha.

Are you running anything else on your box? Perhaps something that’s trying to connect over and over to OpenSearch?

With regards to OpenSearch Dashboards, that makes more sense. I believe that the docker/docker compose initialization process does a lot of the work you are describing - which makes sense for a quick start. However, it’s not baked into OpenSearch as those are not sane defaults for a production environment.

Your intention was to start developing on OpenSearch Dashboards? Or just running? [I ask because I want to know if we should make a change in documentation or developer guides]

Aah yes you’re right I guess. I have multiple services that may be trying to connect to it. I can’t debug it right now, but I don’t think that there should be this huge exception thrown in the logs anyway. Could be investigated maybe!

EDIT: I can confirm that it’s one of my services that triggers the exception, they were trying to contact a previously-running ElasticSearch instance.

I’m currently migrating a Kibana plugin to OSD, so I couldn’t just use the docker image for OSD. I may also be developing for OSD in the near future.

I agree with you that having “none” as a default for the certificate checks is a no-no, that’s why I thought about updating the documentation (or the dev guides) rather than the config. It may be worth adding a small appendix to it!

Totally makes sense. That being said, I think there are two distinct issues here.

  1. The SSL connection exception is is totally unhelpful and gigantic. If you don’t mind, file a issue on GitHub - opensearch-project/security: 🔐 OpenSearch Security plugin for that.

  2. Totally agree with you on the default. The dev guide for an issue on GitHub - opensearch-project/OpenSearch-Dashboards: Open source visualization dashboards for OpenSearch. regarding regarding OpenSearch-Dashboards/DEVELOPER_GUIDE.md at main · opensearch-project/OpenSearch-Dashboards · GitHub …and… GitHub - opensearch-project/documentation-website: The documentation for OpenSearch, OpenSearch Dashboards, and their associated plugins.

I’ve opened one issue and one PR for these problems.

1 Like