Kibana stopped working after upgrade Opendistro 1.10.2 to 1.13.3

Hi All,
After upgrading Opendistro 1.10.2 to to 1.13.3, my Kibana forces me to choose tenant on every login.
Here are some log excerpts:
Elasticsearch log:

[2021-12-17T09:22:04,249][INFO ][c.a.o.s.p.PrivilegesEvaluator] [h161.company.com] No index-level perm match for User [name=kibanaserver, backend_roles=[], requestedTenant=null] Resolved [aliases=[ *], allIndices=[* ], types=[ *], originalRequested=[* ], remoteIndices=[]] [Action [indices:monitor/settings/get]] [RolesChecked [own_index, kibana_server]]
[2021-12-17T09:22:04,249][INFO ][c.a.o.s.p.PrivilegesEvaluator] [h161.company.com] No permissions for [indices:monitor/settings/get]

Here is exerpt from my ‘internal_users.yml’:

`kibanaserver:
hash: “$2y$12$K…”
reserved: true
description: “Kibanaserver user”

Here is an excerpt from my ‘roles_mapping.yml’:

kibana_server:
reserved: true
users:

  • “kibanaserver”

Here is an excerpt from my ‘roles.yml’

kibana_server:
cluster_permissions:

  • “cluster:*”
  • “indices:*”
    index_permissions:
  • index_patterns:
    • “*”
      allowed_actions:
    • “indices_all”
    • “indices:*”

Could you please advise on identifying the issue?

@rlevitsky Could you share your kibana.yml and config.yml files?

@pablo Thank you for your reply.
Here is my kibana.yml file:

---

server.name: "h161.company.com"
server.host: "0"
elasticsearch.hosts:
- https://h161.company.com:9200
elasticsearch.ssl.verificationMode: full
elasticsearch.ssl.certificateAuthorities: ["/usr/share/kibana/config/opendistroCA.crt"]
elasticsearch.username: kibanaserver
elasticsearch.password: *********
elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization"]
elasticsearch.requestTimeout: 30000

opendistro_security.multitenancy.enabled: true
opendistro_security.multitenancy.tenants.enable_private: false
opendistro_security.multitenancy.tenants.preferred: ["Global","Private"]
opendistro_security.readonly_mode.roles: ["kibana_read_only"]

uiSettings.overrides.defaultRoute: /app/discover

opendistro_security.auth.type: "saml"
server.xsrf.whitelist: ["/_opendistro/_security/saml/acs", "/_opendistro/_security/saml/logout"]

newsfeed.enabled: false
telemetry.optIn: false
telemetry.enabled: false

logging:
  root:
    appenders: [default]
    level: debug

Here is my config.yml:

---
_meta:
  type: "config"
  config_version: 2
config:
  dynamic:
    do_not_fail_on_forbidden: false
    kibana:
      multitenancy_enabled: true
      server_username: kibanaserver
      index: '.kibana'
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: false
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
      saml_auth_domain:
        http_enabled: true
        transport_enabled: false
        order: 2
        http_authenticator:
          type: saml
          challenge: true
          config:
            idp:
              enable_ssl: true
              verify_hostnames: true
              metadata_url: https://login.company.com/auth/realms/company/protocol/saml/descriptor
              entity_id: https://login.company.com/auth/realms/company-uat
              pemtrustedcas_filepath: ca-bundle.crt
            sp:
              entity_id: ELK-STAGING
            roles_key: Role
            kibana_url: https://h161.company.com/
            exchange_key: hohnge9ujaiF1ooCei2zo9phizoYoo2f
        authentication_backend:
          type: noop
      ldap:
        description: "Authenticate via LDAP or Active Directory"
        http_enabled: true
        transport_enabled: false
        order: 1
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: ldap
          config:
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            pemtrustedcas_filepath: ca-bundle.crt
            hosts:
            - ldap.company.com
            bind_dn: 'cn=lookup,ou=Special,dc=company,dc=com'
            password: '*********'
            userbase: 'ou=addressbook,dc=company,dc=com'
            usersearch: '(uid={0})'
            username_attribute: 'uid'
    authz:
      roles_from_myldap:
        description: "Authorize via LDAP or Active Directory"
        http_enabled: true
        transport_enabled: false
        authorization_backend:
          type: ldap
          config:
            enable_ssl: true
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            pemtrustedcas_filepath: ca-bundle.crt
            hosts:
            - ldap.company.com
            bind_dn: 'cn=lookup,ou=Special,dc=company,dc=com'
            password: '**********'
            rolebase: 'ou=Groups,dc=company,dc=com'
            rolesearch: '(member={0})'
            userroleattribute: null
            userrolename: disabled
            rolename: cn
            resolve_nested_roles: true
            userbase: 'ou=addressbook,dc=company,dc=com'
            usersearch: '(uid={0})'
            skip_users:
            - admin
            - logstash
            - kibanaserver
            - zabbix
            - grafana
            - retention
            - curator

@rlevitsky In ODFE you’ll need to select the option Remember my selection... to stop the tenant window appreaing.

In OpenSearch, this action is default and the tenant window will appear only once.
Please be aware that tenant selection is kept as a cookie, so if you run your browser in the private mode it will always ask you about the tenant.

1 Like

Thank you very much Pawel,

I didn’t notice that option. After setting it, Kibana is no longer pestering me to select a tenant.

However, I still see those “No permissions” messages at my elasticsearch logs.

Could you please advise on fixing it?

Best,
Roman.

@rlevitsky That is only INFO level message. This is a service account and OpenDistro is treating it as a regular user. No functionality should be affected here.

I’ve checked OpenSearch and it seems to be fixed there.

1 Like