K8S Statefulset, readinessProbe and security bootstrap

Hi everyone,

My issue is quite similar to this one

I have a bunch of k8s manifests bootstrapping a basic Elasticsearch cluster:

  • 1 statefulset / 3 master pods
  • 1 statefulset / 2 data pods
  • Kibana and Elastic ingest node deployement

All of the ES master/data pods have init containers to install opendistro_security plugins on the elasticsearch OSS 7.7.0 container

Elasticsearch.yml is like:

opendistro_security.disabled: false

opendistro_security.ssl.transport.pemkey_filepath: tls/tls.key
opendistro_security.ssl.transport.pemcert_filepath: tls/tls.crt
opendistro_security.ssl.transport.pemtrustedcas_filepath: tls/ca.crt

opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.transport.resolve_hostname: false

opendistro_security.allow_default_init_securityindex: true

  - "CN=elasticsearch-data,OU=elasticsearch+OU=production,O=home,C=FR"
  - "CN=elasticsearch-master,OU=elasticsearch+OU=production,O=home,C=FR"
  - "CN=*"
  - "CN=elasticsearch-admin,OU=elasticsearch+OU=production,O=home,C=FR"

I’m using the following readinessProbe for the elasticsearch container:

    path: /_cluster/health?local=true
    port: 9200
  initialDelaySeconds: 3
  periodSeconds: 3

My problem is that /_cluster/_health?local=true keeps getting a 500 error “OpenDistro security not initialized”, preventing the Elasticsearch pods to go to the running state and K8S to start the other statefulset members of the ES cluster.

My only solution so far is to temporary remove the readiness probe at cluster bootstrap time and set it again once security has been initialized.

Could’nt we just have the security plugin to only activate the node-to-node encryption without requiring to fully initialize security ?
Any hints to bootstrap the cluster and keeps the readinessProbe (which are necessary to apply upgrade during the ES cluster lifecycle)