Errors io.netty.handler.codec.DecoderException: java.lang.IllegalArgumentException: newPosition > limit

Open Source Elasticsearch and Kibana
1

ES Version: 6.8.6
Lucene Version: 7.7.2

We have a 3 nodes cluster where all 3 nodes are marked as “mdi”.
When we loop on /_cat/nodes, we saw one of the node randomly go out of cluster and entire cluster is in unstable stable. It will be random that any node can go out of cluster. When I say, go out of cluster, i don’t see that in cat/nodes.

From logs, we see below errors continuously logging.
Please note this state exist even with 0 load where no new search/ingest requests are coming.

[2020-11-09T06:40:31,260][WARN ][o.e.t.TcpTransport       ] [psrnativecec030920-esdata0] exception caught on transport layer [Netty4TcpChannel{localAddress=/100.73.50.18:9300, remoteAddress=/100.73.50.120:20266}], closing connection
    io.netty.handler.codec.DecoderException: java.lang.IllegalArgumentException: newPosition > limit: (5898240 > 5455480)
            at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:275) ~[netty-codec-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) [netty-transport-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.44.Final.jar:4.1.44.Final]
            at java.lang.Thread.run(Thread.java:834) [?:?]
    Caused by: java.lang.IllegalArgumentException: newPosition > limit: (5898240 > 5455480)
            at java.nio.Buffer.createPositionException(Buffer.java:318) ~[?:?]
            at java.nio.Buffer.position(Buffer.java:293) ~[?:?]
            at java.nio.ByteBuffer.position(ByteBuffer.java:1086) ~[?:?]
            at java.nio.MappedByteBuffer.position(MappedByteBuffer.java:226) ~[?:?]
            at java.nio.MappedByteBuffer.position(MappedByteBuffer.java:67) ~[?:?]
            at io.netty.buffer.PoolArena$DirectArena.memoryCopy(PoolArena.java:794) ~[netty-buffer-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.buffer.PoolArena$DirectArena.memoryCopy(PoolArena.java:704) ~[netty-buffer-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.buffer.PoolArena.reallocate(PoolArena.java:405) ~[netty-buffer-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:118) ~[netty-buffer-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:306) ~[netty-buffer-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:282) ~[netty-buffer-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1104) ~[netty-buffer-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1097) ~[netty-buffer-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1088) ~[netty-buffer-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:94) ~[netty-codec-4.1.44.Final.jar:4.1.44.Final]
            at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:269) ~[netty-codec-4.1.44.Final.jar:4.1.44.Final]
            ... 15 more