OpenID authentication and "Authentication finally failed for null"

Hi

I have been strugling for a while to get OpenID authentication (Keycloak as IdP) to work with OD for ES & Kibana 1.8 and finally run out of options to try and would need help to get this working. Everything used here is ran on top of Kubernetes.

But starting from configs.

ES config.yml (imported after changes with securityadmin.sh):

_meta:
type: “config”
config_version: 2

config:
dynamic:
http:
anonymous_auth_enabled: false
xff:
enabled: false
internalProxies: .+
authc:
basic_internal_auth_domain:
description: “Authenticate via HTTP Basic against internal users database”
http_enabled: true
transport_enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: internal
openid_auth_domain:
http_enabled: true
transport_enabled: true
order: 1
http_authenticator:
type: openid
challenge: false
config:
subject_key: preferred_username
roles_key: roles
enable_ssl: true
verify_hostnames: false
pemtrustedcas_filepath: /usr/share/elasticsearch/config/keycloak-root-ca.pem
openid_connect_url: https:///auth/realms//.well-known/openid-configuration
authentication_backend:
type: noop
authz:
roles_from_myldap:
description: “Authorize via LDAP or Active Directory”
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap
config:
enable_ssl: false
enable_start_tls: false
enable_ssl_client_auth: false
verify_hostnames: true
hosts:
- localhost:8389
bind_dn: null
password: null
rolebase: ‘ou=groups,dc=example,dc=com’
rolesearch: ‘(member={0})’
userroleattribute: null
userrolename: disabled
rolename: cn
resolve_nested_roles: true
userbase: ‘ou=people,dc=example,dc=com’
usersearch: ‘(uid={0})’
roles_from_another_ldap:
description: “Authorize via another Active Directory”
http_enabled: false
transport_enabled: false
authorization_backend:
type: ldap

And kibana.yml:

server.name: kibana
server.host: “0.0.0.0”
elasticsearch.hosts: {ELASTICSEARCH_URL} elasticsearch.requestTimeout: 360000 server.ssl.enabled: true server.ssl.key: /usr/share/kibana/config/kibana-key.pem server.ssl.certificate: /usr/share/kibana/config/kibana-crt.pem elasticsearch.ssl.certificateAuthorities: /usr/share/kibana/config/kibana-root-ca.pem elasticsearch.ssl.verificationMode: none elasticsearch.username: "kibanaserver" elasticsearch.password: "<password>" elasticsearch.requestHeadersWhitelist: ["Authorization", "security_tenant", "securitytenant", "x-forwarded-for", "x-forwarded-by"] opendistro_security.cookie.secure: true opendistro_security.cookie.password: {COOKIE_PASS}
opendistro_security.auth.type: “openid”
opendistro_security.openid.connect_url: “https:///auth/realms//.well-known/openid-configuration”
opendistro_security.openid.client_id: “kibana”
opendistro_security.openid.client_secret: “”
opendistro_security.openid.root_ca: “/usr/share/kibana/config/keycloak-root-ca.pem”
opendistro_security.openid.scope: “openid”
opendistro_security.openid.base_redirect_url:
logging.verbose: true

I also set all possible options for logging in ES side, but those don’t seem to have any impact to logs:

status = error

appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n

rootLogger.level = info
rootLogger.appenderRef.console.ref = console

logger.token.name = com.amazon.dlic.auth.http.saml.Token
logger.token.level = debug

logger.opendistro_security.name = com.amazon.opendistroforelasticsearch.security
logger.opendistro_security.level = trace
logger.opendistro_security.appenderRef.rolling.ref = rolling
logger.opendistro_security.appenderRef.rolling_old.ref = rolling_old
logger.opendistro_security.additivity = false

logger.opendistro_security.name = com.amazon.dlic.auth.http.jwt
logger.opendistro_security.level = trace

Anyway, whenever trying authentication the only thing coming up in the logs (via K8S) is this:

[2020-06-29T10:27:56,203][WARN ][c.a.o.s.h.HTTPBasicAuthenticator] [es-master-2] No ‘Basic Authorization’ header, send 401 and ‘WWW-Authenticate Basic’
[2020-06-29T10:27:56,777][WARN ][c.a.o.s.a.BackendRegistry] [es-master-2] Authentication finally failed for null from 10.144.211.228:46872

I first thought that something is wrong in Kibana side, but after finally trying authentication with curl it looks that the issue is in ElasticSearch. I used following to fetch token from Keycloak and then trying to authenticate towards ES:

RESULT=curl -k --noproxy '*' -d 'client_id=kibana' -d 'username=<uid>' -d 'password=<password>' -d 'grant_type=password' -d 'client_secret=<secret>' -d 'scope=openid' 'https://<Keycloak address>/auth/realms/<realm>/protocol/openid-connect/token'
TOKEN=echo $RESULT | sed 's/.*access_token":"\([^"]*\).*/\1/'
curl -k --noproxy ‘*’ -H “Authorization: Bearer $TOKEN”

The result was exactly the same as trying with Kibana. So something seems to click between ES and Keycloak, but I just can’t figure what. Everything looks to be OK in keycloak’s client config and I also allowed “Web Origins” from * just in case.

Note: Authenthication with Keycloak works fine with Grafana. so IdP side looks to be OK.

Any ideas what to try next?

I forgot to mention that access token provided by Keycloak looks OK:

{
“exp”: 1593424216,
“iat”: 1593423916,
“jti”: “14d3fab1-ba86-4ba5-bd51-617473b8313a”,
“iss”: “https://keycloak address/auth/realms/realm”,
“aud”: “account”,
“sub”: “e6fdfbc4-d552-4824-8466-92249601c496”,
“typ”: “Bearer”,
“azp”: “kibana”,
“session_state”: “91746596-c205-4a4b-9376-e62f9507471e”,
“acr”: “1”,
“allowed-origins”: ["*"],
“realm_access”: {
“roles”: [“offline_access”, “uma_authorization”]
},
“resource_access”: {
“account”: {
“roles”: [“manage-account”, “manage-account-links”, “view-profile”]
}
},
“scope”: “openid email profile”,
“email_verified”: false,
“name”: “XXX”,
“preferred_username”: “xxx”,
“given_name”: “XXX”,
“email”: “xxx@xxx.com
}