OpenDistro cannot downgrade a node from version 7.6.2 to 7.6.1

Open Source Elasticsearch and Kibana
1

Hello,

I’m currently setting up a Kubernetes cluster using opendistro-for-elasticsearch:1.7.0 without AWS. For testing purposes I run 1 replica of each client, master and data pod. However the data pod is giving me trouble I can’t follow. I’m using fresh local-storage PVs without any data on them and other troubleshooting posts I found regarding this topic were trying to downgrade when this problem arrived. I’m at a loss what to do.

Thanks for helping!

[2020-06-05T17:19:48,736][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-data-0] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: cannot downgrade a node from version [7.6.2] to version [7.6.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.6.1.jar:7.6.1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.6.1.jar:7.6.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.6.1.jar:7.6.1]
Caused by: java.lang.IllegalStateException: cannot downgrade a node from version [7.6.2] to version [7.6.1]
at org.elasticsearch.env.NodeMetaData.upgradeToCurrentVersion(NodeMetaData.java:94) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.env.NodeEnvironment.loadNodeMetaData(NodeEnvironment.java:432) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:329) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.node.Node.(Node.java:277) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.node.Node.(Node.java:257) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.1.jar:7.6.1]
… 6 more
uncaught exception in thread [main]
java.lang.IllegalStateException: cannot downgrade a node from version [7.6.2] to version [7.6.1]
at org.elasticsearch.env.NodeMetaData.upgradeToCurrentVersion(NodeMetaData.java:94)
at org.elasticsearch.env.NodeEnvironment.loadNodeMetaData(NodeEnvironment.java:432)
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:329)
at org.elasticsearch.node.Node.(Node.java:277)
at org.elasticsearch.node.Node.(Node.java:257)
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349)
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)

2

@lulzpulz Just to be clear: Your deployment was a completely new deployment, correct? You were NOT trying to upgrade/downgrade an existing deployment, correct?

3

Yes that is correct. Also on a fresh test server without any other applications which could interfere. I want to set it up for the first time while I’m learning Kubernetes.

4

es-data.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    component: elasticsearch
    role: data
  name: es-data
spec:
  serviceName: elasticsearch-data
  replicas: 1
  selector:
    matchLabels:
      component: elasticsearch
      role: data
  template:
    metadata:
      labels:
        component: elasticsearch
        role: data
      annotations:
    spec:
      # Add toleration for not scheduling on dedicated node
      #tolerations:
      #- key: dedicated
      #  value: "true"
      #  effect: NoSchedule
      initContainers:
      - name: init-sysctl
        image: busybox:1.27.2
        command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        securityContext:
          privileged: true
      - name: fixmount
        command: [ 'sh', '-c', 'chown -R 1000:1000 /usr/share/elasticsearch/data' ]
        image: busybox
        volumeMounts:
          - mountPath: /usr/share/elasticsearch/data
            name: data
      serviceAccountName: elasticsearch
      containers:
      - name: elasticsearch
        env:
        - name: CLUSTER_NAME
          value: logs
        - name: NODE_MASTER
          value: "false"
        - name: NODE_INGEST
          value: "false"
        - name: NETWORK_HOST
          value: "_eth0_"
        - name: TRANSPORT_TLS_PEM_PASS
          value: ""
        - name: HTTP_TLS_PEM_PASS
          value: ""
        - name: NUMBER_OF_MASTERS
          value: "1"
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: DISCOVERY_SERVICE
          value: elasticsearch-discovery
        - name: KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: NODE_DATA
          value: "true"
        - name: PROCESSORS
          valueFrom:
            resourceFieldRef:
              resource: limits.cpu
        - name: ES_JAVA_OPTS
          value: -Xms256m -Xmx256m
        # Official Image from Open Distro Team
        image: amazon/opendistro-for-elasticsearch:1.7.0
        imagePullPolicy: Always
        # only publish the transport port
        ports:
        - containerPort: 9300
          name: transport
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            cpu: 500m
            memory: 1Gi
        livenessProbe:
          tcpSocket:
            port: transport
          initialDelaySeconds: 60
          periodSeconds: 10
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: data
        - mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
          name: config
          subPath: elasticsearch.yml
        - mountPath: /usr/share/elasticsearch/config/logging.yml
          name: config
          subPath: logging.yml
        - mountPath: /usr/share/elasticsearch/config/elk-crt.pem
          name: certs
          subPath: elk-crt.pem
          readOnly: true
        - mountPath: /usr/share/elasticsearch/config/elk-key.pem
          name: certs
          subPath: elk-key.pem
          readOnly: true
        - mountPath: /usr/share/elasticsearch/config/elk-root-ca.pem
          name: certs
          subPath: elk-root-ca.pem
          readOnly: true
        - mountPath: /usr/share/elasticsearch/config/admin-crt.pem
          name: certs
          subPath: admin-crt.pem
          readOnly: true
        - mountPath: /usr/share/elasticsearch/config/admin-key.pem
          name: certs
          subPath: admin-key.pem
          readOnly: true
        - mountPath: /usr/share/elasticsearch/config/admin-root-ca.pem
          name: certs
          subPath: admin-root-ca.pem
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: elasticsearch
      - name: certs
        secret:
          secretName: elasticsearch-tls-data
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        component: elasticsearch
        role: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: local-storage
      resources:
        requests:
          storage: 5Gi
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-data
  labels:
    component: elasticsearch
    role: data
spec:
  ports:
  - port: 9300
    name: transport
  - port: 9200
    name: http
  - port: 9600
    name: metrics
  clusterIP: None
  selector:
    component: elasticsearch
    role: data
5

Updating to 1.8.0 solved the issue…