Kibana on k8s cannot connect to elasticsearch

Hello guys,

I am having an issue with integrating Kiabana with Okta OpenID.

  1. Kibana config:
opendistro_security.auth.type: "openid"
opendistro_security.openid.connect_url: "https://my_domain/.well-known/openid-configuration"
opendistro_security.openid.client_id: "0oa9dl1e4VRStlugu5d6"
opendistro_security.openid.client_secret: "my_key"
server.name: kibana
server.host: "0"
elasticsearch.hosts: https://localhost:9200
elasticsearch.ssl.verificationMode: none
elasticsearch.username: kibanaserver
elasticsearch.password: kibanaserver
elasticsearch.requestHeadersWhitelist: [ "securitytenant","Authorization" ]

opendistro_security.multitenancy.enabled: true
opendistro_security.multitenancy.tenants.preferred: [ "Private", "Global" ]
opendistro_security.readonly_mode.roles: [ "kibana_read_only" ]

# Use this setting if you are running kibana without https
opendistro_security.cookie.secure: false

newsfeed.enabled: false
telemetry.optIn: false
telemetry.enabled: false
security.showInsecureClusterWarning: false
  1. Elasticsearch config.yaml:
_meta:
type: "config"
config_version: 2

config:
dynamic:
  # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
  # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
  # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
  #filtered_alias_mode: warn
  #do_not_fail_on_forbidden: false
  #kibana:
  # Kibana multitenancy
  #multitenancy_enabled: true
  #server_username: kibanaserver
  #index: '.kibana'
  http:
    anonymous_auth_enabled: false
    xff:
      enabled: false
      internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
      #internalProxies: '.*' # trust all internal proxies, regex pattern
      #remoteIpHeader:  'x-forwarded-for'
      ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
      ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
      ###### and here https://tools.ietf.org/html/rfc7239
      ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
  authc:
    basic_internal_auth_domain:
      description: "Authenticate via HTTP Basic against internal users database"
      http_enabled: true
      transport_enabled: true
      order: 0
      http_authenticator:
        type: basic
        challenge: false
      authentication_backend:
        type: internal
    openid_auth_domain:
      http_enabled: true
      transport_enabled: true
      order: 1
      http_authenticator:
        type: openid
        challenge: false
        config:
          subject_key: preffered_username
          roles_key: roles
          openid_connect_url: https://my_domain/.well-known/openid-configuration
      authentication_backend:
        type: noop

  1. Internal_useras.yml: basic demo users

_meta:

type: "internalusers"
config_version: 2

new-user:
hash: "$2y$12$88IFVl6IfIwCFh5aQYfOmuXVL9j2hz/GusQb35o.4sdTDAEMTOD.K"
reserved: false
hidden: false
opendistro_security_roles:
- "some-security-role"
backend_roles:
- "some-backend-role"
attributes:
  attribute1: "value1"
static: false

admin:
hash: "$2a$12$VcCDgh2NDk07JGN0rjGbM.Ad41qVR/YFJcgHp0UGns5JDymv..TOG"
reserved: true
backend_roles:
- "admin"
description: "Demo admin user"

kibanaserver:
hash: "$2a$12$4AcgAt3xwOWadA5s5blL6ev39OXDNhmOesEoo33eZtrq2N0YrU3H."
reserved: true
description: "Demo kibanaserver user"

  • Issue:

As soon as i am installing the helm chart after adding the openid settings i get this error in Kibana none stop:

{“type”:“log”,"@timestamp":“2021-02-25T16:25:30Z”,“tags”:[“error”,“elasticsearch”,“data”],“pid”:1,“message”:"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}

The above prevents the server from going up.

  • If i am reverting back to regular authentication without the openid, everything works fine.
  • Tried to switch the hosts to “http”
  • Tried to switch the openid auth in config.yaml to “0”
  • Tried to inject an elasticearch url as per the kubernetes FQDN
  • Tried different users besides “kibanaserver”
  • Played around with multiple different configuration in kibana.yml

I tried vertuallye everything for the past 7 hours and i do hope i am missing someting here.

Thank you for yout time.

Something else that is extremely strange.

If i am installing the default helm chart, kibana is working and connects to elastcsearch.
If i take the exact same config from the default kibana.yml and paste it to the values.yml under kibana.config, the same error is happening.

why are you using elasticsearch.hosts: https://localhost:9200 in kibana config? Are you running kibana and elasticsearch on the same pod? Sorry, did not use helm to install.

there is an iptable that redirects it to the host name of the releval elastisearch client service.
The issue now is different:

Redirects to Okta succesffuly, but getting an error in the kibana pod

{“type”:“log”,"@timestamp":“2021-02-27T16:00:38Z”,“tags”:[“error”,“plugins”,“opendistroSecurityKibana”],“pid”:1,“message”:“OpenId authentication failed: Error: Authentication Exception”}

how jwt from okta look like?

i am not using tls or the token. Just the preffered_username.

after kibana redirects user to OpenIDP, user provides username and password, OpenID provider checked credentials and generate token with result and some additional info and sends it back to kibana. This is how Auth0 or Keycloak are communicating with kibana, I am sure Okta is working the same way

Hmm any idea where can i check the token that was generated?

https://oidcdebugger.com/ or you can find browser addons for jwt debugging. Also do you have any logs from Okta side about authentication? I remember that for instance Auth0 shows log for authentication

Okta side is perfectly fine.

In kibana pod the only error i see is the one i already sent

Can you see token generated in Okta? Also you can increase log level to DEBUG in elasticsearch/kibana to have more information.

Something i had not noticed before. From elasticsearch:

[2021-02-28T06:50:38,136][WARN ][c.a.o.s.h.HTTPBasicAuthenticator] [od-es-opendistro-es-master-0] No ‘Basic Authorization’ header, send 401 and ‘WWW-Authenticate Basic’

[2021-02-28T06:50:38,136][INFO ][c.a.d.a.h.j.k.JwtVerifier] [od-es-opendistro-es-master-0] Escaped Key ID from JWT Token

[2021-02-28T06:50:38,137][WARN ][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [od-es-opendistro-es-master-0] Failed to get subject from JWT claims, check if subject_key ‘preffered_username’ is correct.

[2021-02-28T06:50:38,137][ERROR][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [od-es-opendistro-es-master-0] No subject found in JWT token

[2021-02-28T06:50:38,137][WARN ][c.a.o.s.a.BackendRegistry] [od-es-opendistro-es-master-0] Authentication finally failed for null from 10.64.138.3:38886

I am an idiot. I mispelled preferred_username…
Thank you very much for your replies

Hi @johnmops
Which “preferred_username” you means?