Error when starting a single-node opensearch on docker

OpenSearch
1

Hello everyone!

Before opening an issue on GitHub, I wanted to see if there may be something I missed on my end, as I’m having trouble starting a simple OpenSearch on Docker.

Here’s my docker run command:

docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" opensearchproject/opensearch:1.0.0

And here’s the result (infinite looping) log:

[2021-07-28T11:17:22,451][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [350fac761c4a] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206c6f63616c686f73743a393230300d0a557365722d4167656e743a20476f2d687474702d636c69656e742f312e310d0a4163636570743a206170706c69636174696f6e2f6a736f6e0d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a206c6f63616c686f73743a393230300d0a557365722d4167656e743a20476f2d687474702d636c69656e742f312e310d0a4163636570743a206170706c69636174696f6e2f6a736f6e0d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1261) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.59.Final.jar:4.1.59.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.59.Final.jar:4.1.59.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]

OpenSearch seems to be running though, as I can curl it without problems, but I cannot start a development OpenSearch Dashboard though, it gives me this log:

server log [11:17:14.260] [error][data][opensearch] [ConnectionError]: socket hang up

I haven’t tried it recently, but afaik I had the same problem using the sample docker-compose.

Is anyone else having this problem?

Thanks :slight_smile:

Docker version: 20.10.7
Docker Compose version: 1.29.2
System: Ubuntu 20.04

2

Update on the issue:

I managed to start a dev OSD (I still have the constant exception on OpenSearch though), but I think we should update the documentation or the default config values of opensearch_dashboards.yml.

Considering that the defaults of the docker image are to set the admin:admin credentials and enable ssl, I think we should match these settings in the OSD defaults:

diff --git a/config/opensearch_dashboards.yml b/config/opensearch_dashboards.yml
index d281b987ee..65aa1df2e4 100644
--- a/config/opensearch_dashboards.yml
+++ b/config/opensearch_dashboards.yml
@@ -25,7 +25,7 @@
 #server.name: "your-hostname"

 # The URLs of the OpenSearch instances to use for all your queries.
-#opensearch.hosts: ["http://localhost:9200"]
+#opensearch.hosts: ["https://localhost:9200"]
 
 # OpenSearch Dashboards uses an index in OpenSearch to store saved searches, visualizations and
 # dashboards. OpenSearch Dashboards creates a new index if the index doesn't already exist.
@@ -48,8 +48,8 @@
 # the username and password that the OpenSearch Dashboards server uses to perform maintenance on the OpenSearch Dashboards
 # index at startup. Your OpenSearch Dashboards users still need to authenticate with OpenSearch, which
 # is proxied through the OpenSearch Dashboards server.
-#opensearch.username: "opensearch_dashboards_system"
-#opensearch.password: "pass"
+#opensearch.username: "admin"
+#opensearch.password: "admin"
 
 # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
 # These settings enable SSL for outgoing requests from the OpenSearch Dashboards server to the browser.
@@ -68,7 +68,7 @@
 #opensearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
 
 # To disregard the validity of SSL certificates, change this setting's value to 'none'.
-#opensearch.ssl.verificationMode: full
+#opensearch.ssl.verificationMode: none

What do you think ? I will surely open an issue for this case.

3

Hey @closingin - welcome!

That all seems a bit weird. The log you’re posted in the first message - is that coming from the OpenSearch docker logs?

The docker-compose should work out-of-the-box - to confirm, you’re using this docker compose file, right?

https://opensearch.org/samples/docker-compose.yml

4

Thanks for your reply :slight_smile:

So yes and yes for your questions.

Technically, the docker-compose works out-of-the-box, as I can launch it and get access to OSD without any trouble, but the exception logs still spawn.

For the “dev” OSD I speak about in my messages, I mean that I cloned the repo and yarn start-ed it. In this case it doesn’t work with a dockerized OS if I don’t apply the changes I posted above.

5

OK. Gotcha.

Are you running anything else on your box? Perhaps something that’s trying to connect over and over to OpenSearch?

With regards to OpenSearch Dashboards, that makes more sense. I believe that the docker/docker compose initialization process does a lot of the work you are describing - which makes sense for a quick start. However, it’s not baked into OpenSearch as those are not sane defaults for a production environment.

Your intention was to start developing on OpenSearch Dashboards? Or just running? [I ask because I want to know if we should make a change in documentation or developer guides]

6

Aah yes you’re right I guess. I have multiple services that may be trying to connect to it. I can’t debug it right now, but I don’t think that there should be this huge exception thrown in the logs anyway. Could be investigated maybe!

EDIT: I can confirm that it’s one of my services that triggers the exception, they were trying to contact a previously-running ElasticSearch instance.

I’m currently migrating a Kibana plugin to OSD, so I couldn’t just use the docker image for OSD. I may also be developing for OSD in the near future.

I agree with you that having “none” as a default for the certificate checks is a no-no, that’s why I thought about updating the documentation (or the dev guides) rather than the config. It may be worth adding a small appendix to it!

7

Totally makes sense. That being said, I think there are two distinct issues here.

  1. The SSL connection exception is is totally unhelpful and gigantic. If you don’t mind, file a issue on GitHub - opensearch-project/security: 🔐 Secure your cluster with TLS, numerous authentication backends, data masking, audit logging as well as role-based access control on indices, documents, and fields for that.

  2. Totally agree with you on the default. The dev guide for an issue on GitHub - opensearch-project/OpenSearch-Dashboards: 📊 Open source visualization dashboards for OpenSearch. regarding regarding https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/DEVELOPER_GUIDE.md …and… GitHub - opensearch-project/documentation-website: The documentation for OpenSearch, OpenSearch Dashboards, and their associated plugins.

8

I’ve opened one issue and one PR for these problems.

1 Like
9

Hello there!

I faced the same issue when trying to run discovery type with single-node, and fix that for local development removing the security plugin, as of:

Dockerfile:

# Remove opensearch-security plugin
FROM opensearchproject/opensearch:1.3.6
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-security

docker-compose.yml:

    opensearch:
        build:
          context: .
          dockerfile: Dockerfile
        container_name: opensearch
        restart: unless-stopped
        environment:
          - discovery.type=single-node
        ports:
          - 9200:9200
          - 9600:9600
        volumes:
          - 'opensearch:/usr/share/opensearch/data'