SSl/TSL connection error - Elasticsearch

Hi,

I have created the opendistro cluster for Elasticsearch in k8s environment, so cluster is up and I have ingress to connect to the cluster, like cluster external address would be like https://app.example.com/clusterid. So I want to connect the cluster using this url for any http REST API’s.
I have ClusterIP type service which exposes the port 9200.
So when I try to connect the cluster using https://app.example.com/clusterid address, I am getting below logs in Elasticsearch pods.

I able to connect to cluster using port-forward with admin/admi credentials. but I try to connect using ingress it gives the error. I tried to disable the opendistro-security plugins, disabling all the security related config in the elasticsearch.yaml but still it did not work.

Any help would be much appreciated.

[2021-07-27T10:31:00,461][ERROR][c.a.o.s.s.h.n.OpenDistroSecuritySSLNettyHttpServerTransport] [elasticsearch] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-07-27T10:31:00,463][WARN ][o.e.h.AbstractHttpServerTransport] [elasticsearch] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/11.11.11.11:9200, remoteAddress=/12.12.12.12:58840}

Moved this to the security category, that will get it in front of the right people.

@nk2812 Are you using helm? Can you please provide your configuration files (redact any sensitive details)

No, I am not using helm. Below is my configuration files.

this is my elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0

Using below file I am creating the cluster.

apiVersion: abc.example.com/v1alpha1
kind: OpenDistroCluster
metadata:
  name: opendistro-sample
spec:
  nodeSets:
    - name: master
      count: 3
      podTemplate:
        metadata:
            name: opendistro-sample
            labels:
                opendistro-cluster: "true"
        spec:
            initContainers:
              - name: sysctl
                image: busybox
                command: ['sh', '-c', "sysctl -w vm.max_map_count=262144"]
            containers:
              - name: opendistro
                image: amazon/opendistro-for-elasticsearch:1.13.2
                env:
                    - name: cluster.name
                      value: "opendistro-cluster"
                    - name: bootstrap.memory_lock
                      value: "true"
                ports:
                    - containerPort: 9200
                    - containerPort: 9600 # required for Performance Analyzer
                resources:
                    requests:
                        memory: 4Gi
                        cpu: 1
                    limits:
                        memory: 4Gi
                        cpu: 1

Also can you guide me on how can I provide below felids, my pods will be created dynamically so I do not have host names before start of the cluster.

     - discovery.seed_hosts=odfe-node1,odfe-node2
     - cluster.initial_master_nodes=odfe-node1,odfe-node2

Thanks.