ES is not able to find "roles" in JWT payload from Okta OpenID

Hello All! We are testing SSO in ES and kibana with Opendistro ES. Found some issues. Please find below what I did.

  1. I have installed Opendistro ES helm chart in Kubernetes! ES and kibana installed successfully!
  2. Created config.yml and roles_mapping.yml secrets and loaded to ES master pod.
  3. One issue here. The secrets loading only to master pod of ES. They are not loading into client and data pods even though I ran securityadmindemo.sh.
  4. But somehow ES is not able to find roles_key * “roles” from okta openID jwt payload.I see the following warnings in client pod. I have added this attribute in okta and able to see in the jwt payload when I tested with localhost redirect!

2020-09-22T03:25:28.115268136Z [2020-09-22T03:25:28,115][WARN ][c.a.d.a.h.j.AbstractHTTPJwtAuthenticator] [okta-efk-client-589f5ccfd5-jfmv7] Failed to get roles from JWT claims with roles_key ‘roles’. Check if this key is correct and available in the JWT payload.
2020-09-22T03:25:28.605088779Z [2020-09-22T03:25:28,604][WARN ][c.a.o.s.h.HTTPBasicAuthenticator] [okta-efk-client-589f5ccfd5-jfmv7] No ‘Basic Authorization’ header, send 401 and ‘WWW-Authenticate Basic’

Configs I have used:
ES config.yml (please note this one loading only into master pod)
_meta:
type: “config”
config_version: 2

config:
dynamic:
kibana:
multitenancy_enabled: false
server_username: admin
index: ‘.kibana’
do_not_fail_on_forbidden: false
authc:
basic_internal_auth_domain:
description: “Sign in with Username and Password”
http_enabled: true
transport_enabled: true
order: 0
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: internal
openid_auth_domain:
description: “Sign in with Okta”
http_enabled: true
transport_enabled: true
order: 1
http_authenticator:
type: openid
challenge: true
config:
subject_key: preferred_username
roles_key: “roles” (from jwt payload)
openid_connect_url: https://.okta.com/.well-known/openid-configuration
authentication_backend:
type: noop

kibana.yml: Added following config settings in values.yml under kibana section.
config:
opendistro_security.openid.client_id: “<ClientID”
opendistro_security.openid.client_secret: “<SecretID”
opendistro_security.openid.connect_url: https://.okta.com/.well-known/openid-configuration
opendistro_security.auth.type: “openid”
opendistro_security.openid.base_redirect_url: https://kibana.somedomain.com
opendistro_security.openid.logout_url: https://<company.okta.com
opendistro_security.openid.scope: “openid profile email”
elasticsearch.ssl.verificationMode: none
elasticsearch.requestHeadersWhitelist: [“Authorization”, “security_tenant”, “securitytenant”]
elasticsearch.hosts: https://okta-efk-client-service:9200
elasticsearch.username: “admin”
elasticsearch.password: “admin”
elasticsearch.requestTimeout: 360000
server.name: kibana
server.host: “0”
newsfeed.enabled: false
telemetry.optIn: false
telemetry.enabled: false
opendistro_security.multitenancy.enabled: true
opendistro_security.multitenancy.tenants.enable_global: true
opendistro_security.multitenancy.tenants.enable_private: false
opendistro_security.multitenancy.tenants.preferred: [“Global”]
opendistro_security.readonly_mode.roles: [“kibana_read_only”]

roles_mapping.yml

_meta:
type: “rolesmapping”
config_version: 2

all_access:
reserved: false
backend_roles:

  • “admin”
  • “write” (this is the role I created in okta. It was in the JWT payload as “roles”: [“write”])
    description: “Maps admin to all_access”

own_index:
reserved: false
users:

  • “*”
    description: “Allow full access to an index named like the username”

logstash:
reserved: false
backend_roles:

  • “logstash”

kibana_user:
reserved: false
backend_roles:

  • “kibanauser”
    description: “Maps kibanauser to kibana_user”

readall:
reserved: false
backend_roles:

  • “read”
  • “kibana_user”

manage_snapshots:
reserved: false
backend_roles:

  • “snapshotrestore”

kibana_server:
reserved: true
users:

  • “kibanaserver”

What worked?

  1. With all the above configs I’m able to redirect to kibana with full access and create indexes,all admin stuff. All the okta app assigned users doing same since no backend roles.

Expected behaviour/requirements:

**1. If we select “write” role in kibana (mapped backend role for all_access rolemapping) that user can create indexes,dashboards,etc…in global tenant.
2. If “read” role selected(mapped to readall backend role) those users can read dashboards,indexes in global tenant.

Any help would be much appreciated.Thank you in advance!