Unable to configure Kibana

TLDR; Cannot get Kibana to use SSL with two elastic nodes. We are not using logstash, following standard guides.

We are trying to install the Elasticstack using docker. We are having troubles gettings Kibana to communicate with the Elasticstack. We have tried to install the entire system multiple times and each time something else fails. Prior to this attempt, we have HTTPS to Kibana but we believe that the Elastic nodes were not encrypted and using the default certs.

We are using self signing certificates for all nodes. We are following this guide https://opendistro.github.io/for-elasticsearch-docs/docs/security/configuration/generate-certificates/ We are generating a cert for Admin, Node1, Node2, and Kibana. These are our current keys.

-rw-rw-rw- 1 docker docker 1704 Jul  8 15:05 admin-key.pem
-rw-rw-rw- 1 docker docker 1188 Jul  8 15:08 admin.pem
-rw------- 1 docker docker 1708 Jul  9 08:51 kibana-key.pem
-rw-r--r-- 1 docker docker 1188 Jul  9 08:52 kibana.pem
-rw-rw-rw- 1 docker docker 1704 Jul  8 15:29 node1-key.pem
-rw-rw-rw- 1 docker docker 1188 Jul  8 15:32 node1.pem
-rw-rw-rw- 1 docker docker 1704 Jul  8 15:33 node2-key.pem
-rw-rw-rw- 1 docker docker 1188 Jul  8 15:34 node2.pem
-rw-rw-rw- 1 docker docker 1679 Jul  8 15:01 root-ca-key.pem
-rw-rw-rw- 1 docker docker 1310 Jul  8 15:03 root-ca.pem

Our DockerFile is as follows.

version: ‘3’
services:
odfe-node1:
image: amazon/opendistro-for-elasticsearch:1.8.0
container_name: odfe-node1
environment:
- cluster.name=odfe-cluster
- node.name=odfe-node1
- discovery.seed_hosts=odfe-node1,odfe-node2
- cluster.initial_master_nodes=odfe-node1,odfe-node2
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- “ES_JAVA_OPTS=-Xms512m -Xmx512m” # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
- network.host=0.0.0.0 # required if not using the demo security configuration
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- odfe-data1:/usr/share/elasticsearch/data
- ./root-ca.pem:/usr/share/elasticsearch/config/root-ca.pem
- ./node1.pem:/usr/share/elasticsearch/config/node1.pem
- ./node1-key.pem:/usr/share/elasticsearch/config/node1-key.pem
- ./admin.pem:/usr/share/elasticsearch/config/admin.pem
- ./admin-key.pem:/usr/share/elasticsearch/config/admin-key.pem
- ./custom-elasticsearch1.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- ./internal_users.yml:/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml
- ./roles_mapping.yml:/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles_mapping.yml
- ./tenants.yml:/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/tenants.yml
- ./roles.yml:/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles.yml
- ./action_groups.yml:/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/action_groups.yml
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- odfe-net
odfe-node2:
image: amazon/opendistro-for-elasticsearch:1.8.0
container_name: odfe-node2
environment:
- cluster.name=odfe-cluster
- node.name=odfe-node2
- discovery.seed_hosts=odfe-node1,odfe-node2
- cluster.initial_master_nodes=odfe-node1,odfe-node2
- bootstrap.memory_lock=true
- “ES_JAVA_OPTS=-Xms512m -Xmx512m”
- network.host=0.0.0.0
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- odfe-data2:/usr/share/elasticsearch/data
- ./root-ca.pem:/usr/share/elasticsearch/config/root-ca.pem
- ./node2.pem:/usr/share/elasticsearch/config/node2.pem
- ./node2-key.pem:/usr/share/elasticsearch/config/node2-key.pem
- ./admin.pem:/usr/share/elasticsearch/config/admin.pem
- ./admin-key.pem:/usr/share/elasticsearch/config/admin-key.pem
- ./custom-elasticsearch2.yml:/usr/share/elasticsearch/config/elasticsearch.yml
- ./internal_users.yml:/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml
- ./roles_mapping.yml:/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles_mapping.yml
- ./tenants.yml:/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/tenants.yml
- ./roles.yml:/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles.yml
- ./action_groups.yml:/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/action_groups.yml
networks:
- odfe-net
kibana:
image: amazon/opendistro-for-elasticsearch-kibana:1.8.0
container_name: odfe-kibana
ports:
- 5601:5601
expose:
- “5601”
environment:
ELASTICSEARCH_URL: https ://odfe-node1:9200 # I had to break this apart due to new user two link posting limit
ELASTICSEARCH_HOSTS: https ://odfe-node1:9200 # I had to break this apart due to new user two link posting limit
SERVER_SSL_CERT: /usr/share/kibana/config/kibana.pem
SERVER_SSL_KEY: /usr/share/kibana/config/kibana-key.pem
SERVER_SSL_ENABLED: “true”
#SERVER_SSL_CERT: /usr/share/kibana/config/node2.pem
#SERVER_SSL_KEY: /usr/share/kibana/config/node2-key.pem
#ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: /usr/share/elasticsearch/config/root-ca.pem
volumes:
#- ./custom-kibana.yml:/usr/share/kibana/config/kibana.yml
- ./kibana-key.pem:/usr/share/kibana/config/kibana-key.pem
- ./kibana.pem:/usr/share/kibana/config/kibana.pem
- ./root-ca.pem:/usr/share/elasticsearch/config/root-ca.pem
#- ./node2-key.pem:/usr/share/kibana/config/node2-key.pem
#- ./node2.pem:/usr/share/kibana/config/node2.pem
networks:
- odfe-net

volumes:
odfe-data1:
odfe-data2:

networks:
odfe-net:

When we run docker compose up we are getting the following error.

odfe-kibana |
odfe-kibana | FATAL Error: [config validation of [server].ssl.cert]: definition for this key is missing
odfe-kibana |

With a stack trace preceding this message.

odfe-kibana | {“type”:“log”,"@timestamp":“2020-07-09T13:16:08Z”,“tags”:[“fatal”,“root”],“pid”:1,“message”:"{ Error: [config validation of [server].ssl.cert]: definition for this key is missing\n at ObjectType.validate (/usr/share/kibana/node_modules/@kbn/config-schema/target/out/types/type.js:59:19)\n at ConfigService.validateAtPath (/usr/share/kibana/src/core/server/config/config_service.js:188:19)\n at MapSubscriber.getDistinctConfig.pipe.config [as project] (/usr/share/kibana/src/core/server/config/config_service.js:196:81)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:49:35)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at DistinctUntilChangedSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:76:28)\n at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:44:25)\n at ReplaySubject.Subject._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Subject.js:102:51)\n at ReplaySubject.Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:30:22)\n at MapSubscriber.shareReplayOperation (/usr/share/kibana/node_modules/rxjs/internal/operators/shareReplay.js:44:32)\n at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:25:31)\n at MapOperator.call (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:32:23)\n at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:25:31)\n at DistinctUntilChangedOperator.call (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:27:23)\n at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:25:31)\n at MapOperator.call (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:32:23)\n at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:25:31)\n at TakeOperator.call (/usr/share/kibana/node_modules/rxjs/internal/operators/take.js:38:23)\n cause:\n { Error: definition for this key is missing\n at ObjectType.onError (/usr/share/kibana/node_modules/@kbn/config-schema/target/out/types/type.js:84:20)\n at type.Type.schema.error (/usr/share/kibana/node_modules/@kbn/config-schema/target/out/types/type.js:49:53)\n at finish (/usr/share/kibana/node_modules/joi/lib/types/any/index.js:529:50)\n at type._validate (/usr/share/kibana/node_modules/joi/lib/types/any/index.js:618:24)\n at type._base (/usr/share/kibana/node_modules/joi/lib/types/object/index.js:212:45)\n at type._validate (/usr/share/kibana/node_modules/joi/lib/types/any/index.js:614:37)\n at type._validateWithOptions (/usr/share/kibana/node_modules/joi/lib/types/any/index.js:674:29)\n at module.exports.internals.Any.root.validate (/usr/share/kibana/node_modules/joi/lib/index.js:146:23)\n at ObjectType.validate (/usr/share/kibana/node_modules/@kbn/config-schema/target/out/types/type.js:54:72)\n at ConfigService.validateAtPath (/usr/share/kibana/src/core/server/config/config_service.js:188:19)\n at MapSubscriber.getDistinctConfig.pipe.config [as project] (/usr/share/kibana/src/core/server/config/config_service.js:196:81)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:49:35)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at DistinctUntilChangedSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:76:28)\n at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:44:25)\n at ReplaySubject.Subject._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Subject.js:102:51)\n at ReplaySubject.Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:30:22)\n at MapSubscriber.shareReplayOperation (/usr/share/kiba
na/node_modules/rxjs/internal/operators/shareReplay.js:44:32) cause: undefined, path: [ ‘ssl’, ‘cert’ ] } }"}

If you need any other information from me let me know and ill post it.

Hey, hopefully this is better late than never, but your variable for the certificate needs to be SERVER_SSL_CERTIFICATE

cheers,
amor