Cannot get TLS to work

#1

Using sample docker-compose.yml

I tried the following approaches for generating the certificates:

I used the output of openssl x509 -subject -nameopt RFC2253 -noout -in node.pem to fill out the config for nodes_dn and admin_dn.

I am always stuck on this error. Getting cluster health only shows one node.

odfe-node1 | [2019-05-14T13:26:29,436][ERROR][c.a.o.s.t.OpenDistroSecurityRequestHandler] [vMkcCJQ] ElasticsearchException[Illegal parameter in http or transport request found.
odfe-node1 | This means that one node is trying to connect to another with
odfe-node1 | a non-node certificate (no OID or opendistro_security.nodes_dn incorrect configured) or that someone
odfe-node1 | is spoofing requests. Check your TLS certificate setup as described in documentation]

I̶f̶ ̶I̶ ̶e̶x̶e̶c̶u̶t̶e̶ ̶s̶e̶c̶u̶r̶i̶t̶y̶a̶d̶m̶i̶n̶.̶s̶h̶ ̶I̶ ̶a̶m̶ ̶b̶e̶i̶n̶g̶ ̶t̶o̶l̶d̶ ̶t̶h̶a̶t̶ ̶I̶ ̶a̶m̶ ̶n̶o̶t̶ ̶a̶n̶ ̶a̶d̶m̶i̶n̶ ̶u̶s̶e̶r̶.̶ ̶E̶v̶e̶n̶ ̶t̶h̶o̶u̶g̶h̶ ̶I̶ ̶c̶o̶p̶y̶ ̶p̶a̶s̶t̶e̶d̶ ̶f̶r̶o̶m̶ ̶t̶h̶e̶ ̶d̶o̶c̶s̶.̶

Problem still persists after running securityadmin successfully

elasticsearch-node1.yml (node2 is the same but with node2 certs/CN)

cluster.name: "docker-cluster"

network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP

# set to 1 to allow single node clusters

# Details: https://github.com/elastic/elasticsearch/pull/17288

discovery.zen.minimum_master_nodes: 1

######## Start OpenDistro for Elasticsearch Security Demo Configuration ########

# WARNING: revise all the lines below before you go into production

opendistro_security.ssl.transport.pemcert_filepath: odfe-node1.pem

opendistro_security.ssl.transport.pemkey_filepath: odfe-node1.key

opendistro_security.ssl.transport.pemtrustedcas_filepath: root-ca.pem

opendistro_security.ssl.transport.enforce_hostname_verification: false

opendistro_security.ssl.http.enabled: true

opendistro_security.ssl.http.pemcert_filepath: odfe-node1.pem

opendistro_security.ssl.http.pemkey_filepath: odfe-node1.key

opendistro_security.ssl.http.pemtrustedcas_filepath: root-ca.pem

opendistro_security.allow_default_init_securityindex: true

opendistro_security.authcz.admin_dn:

- emailAddress=support@example.com,CN=admin,OU=Inc.,O=EXAMPLE,L=Munich,ST=Bavaria,C=DE

opendistro_security.nodes_dn:

- emailAddress=support@example.com,CN=node1.example.com,OU=Inc.,O=EXAMPLE,L=Munich,ST=Bavaria,C=DE

opendistro_security.audit.type: internal_elasticsearch

opendistro_security.enable_snapshot_restore_privilege: true

opendistro_security.check_snapshot_restore_write_privileges: true

opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]

cluster.routing.allocation.disk.threshold_enabled: false

node.max_local_storage_nodes: 3

######## End OpenDistro for Elasticsearch Security Demo Configuration ########

docker-compose.yml

version: '3'
services:
  odfe-node1:
    image: amazon/opendistro-for-elasticsearch:0.9.0
    container_name: odfe-node1
    environment:
      - cluster.name=odfe-cluster
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - odfe-data1:/usr/share/elasticsearch/data
      - ./root-ca.pem:/usr/share/elasticsearch/config/root-ca.pem
      - ./admin.pem:/usr/share/elasticsearch/config/admin.pem
      - ./admin.key:/usr/share/elasticsearch/config/admin.key
      - ./odfe-node1.pem:/usr/share/elasticsearch/config/odfe-node1.pem
      - ./odfe-node1.key:/usr/share/elasticsearch/config/odfe-node1.key
      - ./elasticsearch-node1.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - odfe-net
  odfe-node2:
    image: amazon/opendistro-for-elasticsearch:0.9.0
    container_name: odfe-node2
    environment:
      - cluster.name=odfe-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.zen.ping.unicast.hosts=odfe-node1
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - odfe-data2:/usr/share/elasticsearch/data
      - ./root-ca.pem:/usr/share/elasticsearch/config/root-ca.pem
      - ./admin.pem:/usr/share/elasticsearch/config/admin.pem
      - ./admin.key:/usr/share/elasticsearch/config/admin.key
      - ./odfe-node2.pem:/usr/share/elasticsearch/config/odfe-node2.pem
      - ./odfe-node2.key:/usr/share/elasticsearch/config/odfe-node2.key
      - ./elasticsearch-node2.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    networks:
      - odfe-net
  kibana:
    image: amazon/opendistro-for-elasticsearch-kibana:0.9.0
    container_name: odfe-kibana
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_URL: https://odfe-node1:9200
      SERVER_SSL_ENABLED: "true"
      SERVER_SSL_KEY: /usr/share/kibana/config/kibana.key
      SERVER_SSL_CERTIFICATE: /usr/share/kibana/config/kibana.pem
    volumes:
      - ./root-ca.pem:/usr/share/kibana/config/root-ca.pem
      - ./kibana.pem:/usr/share/kibana/config/kibana.pem
      - ./kibana.key:/usr/share/kibana/config/kibana.key
    networks:
      - odfe-net
volumes:
  odfe-data1:
  odfe-data2:

networks:
  odfe-net:

How to fix this?

#2

I guess node_dn follow format as below with quotes.

opendistro_security.nodes_dn:
  - "CN=*.example.com, OU=SSL, O=Test, L=Test, C=DE"
  - "CN=node.other.com, OU=SSL, O=Test, L=Test, C=DE"

Can you please try similar format?

#3

I suspect single or double quotes are both fine (I’ve always used single), but it’s important not to include whitespace between elements. So "CN=*.example.com,OU=SSL,O=Test,L=Test,C=DE". I’ll tweak the docs to be more clear about the quotes.

#4

As @aetter mentioned you should not include any whitespace in that string. I got it resolved by adding both nodes on each elasticsearch-nodeX.yml file (sadly not mentioned in the docs).

#5

Hi ecklf, I’d argue that it was included on a different page, but I changed a couple sentences to further clarify. I appreciate the heads-up. :+1:

1 Like
#6

Awesome, thank you! In case you are able to moderate the blog post it might be a great addition aswell :wink: