Multitenancy - even with separated roles and tenants everything saved ends up in global

Hi,

We have an issue where multitenancy doesn’t seem to work, it is enabled and we can create tenants, assign them to roles etc… Users can select tenant upon login or switch.

So say user Banana has the tenant network, login, select that tenant out of private, global and network. Goes to dashboard and created a dashboard, saves it. Then everyone can see that saved object, but if I click share on that dashboard during edit you can see the tenant information in the URL.

Are we missing something obvious and are just blind or can it be more to it?

Br
Jan

Hi @Jan
I don’t follow the question, the tenant information is relevant as the dashboard (like any other kibana/opensearchDashboard object) is relevant to the tenant that it was created in, therefore the information of the tenant is present in the URL.

Is this what you are asking about?

Hi,
No not really, sorry was very late last night. I know that the tenant information should be in the link, that is the only thing about tenants that is working, the issue is that the dashboard and visualizations that are created for one tenant do not get stored for that tenant only, anyone will see that in saved objects and can just click it and use it.

The separation between tenants isn’t working at all, everything seems to be stored in the global context in .kibana_1, I do not see the creation of any .kibana_tenant or .kibana_user either.

Hope this clears up the question a bit more. All searches I have done on google have resulted in the opposite problem that others do not see what they should. I would have preferred that issue :slight_smile: Here everyone can see no matter what tenant they are in, even though they don’t have read access to any tenant but their own.

@Jan thats a very interesting case indeed.
What version of ODFE / opensearch are you running? Can you provide the config.yml, kibana.yml and elasticsearch.yml files? (redact any sensitive details)

Config.yml

_meta:
  type: "config"
  config_version: 2
 
config:
  dynamic:
    # Set filtered_alias_mode to 'disallow' to forbid more than 2 filtered aliases per index
    # Set filtered_alias_mode to 'warn' to allow more than 2 filtered aliases per index but warns about it (default)
    # Set filtered_alias_mode to 'nowarn' to allow more than 2 filtered aliases per index silently
    #filtered_alias_mode: warn
    #do_not_fail_on_forbidden: false
    kibana:
      # Kibana multitenancy
      multitenancy_enabled: true
      server_username: kibanaserver
      index: '.opensearch-dashboards'
    do_not_fail_on_forbidden: false
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
        internalProxies: '192\.168\.0\.10|192\.168\.0\.11' # regex pattern
        #internalProxies: '.*' # trust all internal proxies, regex pattern
        #remoteIpHeader:  'x-forwarded-for'
        ###### see https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html for regex help
        ###### more information about XFF https://en.wikipedia.org/wiki/X-Forwarded-For
        ###### and here https://tools.ietf.org/html/rfc7239
        ###### and https://tomcat.apache.org/tomcat-8.0-doc/config/valve.html#Remote_IP_Valve
    authc:
      egad:
        description: "Authenticate via Active Directory"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          # LDAP authentication backend (authenticate users against a LDAP or Active Directory)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: true
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            pemtrustedcas_filepath: root-ca.pem
            hosts:
            - <ldap1>:389
            - <ldap2>:389
            - <ldap3>.se:389
            bind_dn: 'CN=<user>,OU=<ou>,OU=<ou>,OU=<ou>,OU=<ou>,OU=<ou>,DC=<dc>,DC=<dc>'
            password: '<password>'
            userbase: 'OU=<ou>,OU=<ou>,DC=<dc>,DC=<dc>'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(sAMAccountName={0})'
            # Use this attribute from the user as username (if not set then DN is used)
            username_attribute: 'displayName'
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
    authz:
      roles_from_egad:
        description: "Authorize via Active Directory"
        http_enabled: true
        transport_enabled: true
        authorization_backend:
          # LDAP authorization backend (gather roles from a LDAP or Active Directory, you have to configure the above LDAP authentication backend settings too)
          type: ldap
          config:
            # enable ldaps
            enable_ssl: false
            # enable start tls, enable_ssl should be false
            enable_start_tls: true
            # send client certificate
            enable_ssl_client_auth: false
            # verify ldap hostname
            verify_hostnames: true
            pemtrustedcas_filepath: root-ca.pem
            hosts:
            - <ldap1>:389
            - <ldap2>:389
            - <ldap3>.se:389
            bind_dn: 'CN=<user>,OU=<ou>,OU=<ou>,OU=<ou>,OU=<ou>,OU=<ou>,DC=<dc>,DC=<dc>'
            password: '<password>'
            rolebase: 'OU=<ou>,OU=<ou>,DC=<dc>,DC=<dc>'
            # Filter to search for roles (currently in the whole subtree beneath rolebase)
            # {0} is substituted with the DN of the user
            # {1} is substituted with the username
            # {2} is substituted with an attribute value from user's directory entry, of the authenticated user. Use userroleattribute to specify the name of the attribute
            rolesearch: '(member={0})'
            # Specify the name of the attribute which value should be substituted with {2} above
            userroleattribute: null
            # Roles as an attribute of the user entry
            userrolename: disabled
            #userrolename: memberOf
            # The attribute in a role entry containing the name of that role, Default is "name".
            # Can also be "dn" to use the full DN as rolename.
            rolename: cn
            # Resolve nested roles transitive (roles which are members of other roles and so on ...)
            resolve_nested_roles: true
            #userbase: 'ou=people,dc=example,dc=com'
            # Filter to search for users (currently in the whole subtree beneath userbase)
            # {0} is substituted with the username
            usersearch: '(uid={0})'
            # Skip users matching a user name, a wildcard or a regex pattern
            skip_users:
              - 'kibanaserver'
              - 'admin'
              - 'elasticsink'

opensearch.yml

cluster.name: docker-cluster

# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0

# # minimum_master_nodes need to be explicitly set when bound on a public IP
# # set to 1 to allow single node clusters
# discovery.zen.minimum_master_nodes: 1

# Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
#discovery.type: single-node

######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: node.pem
plugins.security.ssl.transport.pemkey_filepath: node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node.pem
plugins.security.ssl.http.pemkey_filepath: node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - 'CN=<cn>,OU=<ou>,O=<o>,L=<l>,ST=<st>,C=<c>'
plugins.security.nodes_dn:
  - 'CN=<cn>,OU=<ou>,O=<o>,L=<l>,ST=<st>,C=<c>'
  - 'CN=<cn>,OU=<ou>,O=<o>,L=<l>,ST=<st>,C=<c>'

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*"]
node.max_local_storage_nodes: 3
######## End OpenSearch Security Demo Configuration ########

opensearch_dashboard.yml

# Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License.
# A copy of the License is located at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# or in the "license" file accompanying this file. This file is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied. See the License for the specific language governing
# permissions and limitations under the License.

# Description:
# Default configuration for OpenSearch Dashboards

server.host: "0"
opensearch.hosts: ["https://localhost:9200"]
opensearch.ssl.verificationMode: none
opensearch.username: "kibanaserver"
opensearch.password: "<password>"
opensearch.requestHeadersWhitelist: [ authorization,securitytenant ]

server.ssl.enabled: true
server.ssl.certificate: /usr/share/opensearch-dashboards/config/<cert>.crt
server.ssl.key: /usr/share/opensearch-dashboards/config/<key>.key

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false

The version we are using is 1.0.1, latest docker release (tag latest as well).

@Jan
The config looks ok,
can you also provide the roles.yml file
also
can you connect as the user that can see more than expected and select View roles and identities (top right corner) to check which roles the user is mapped to?

The last test I did I made the roles and mappings inside opensearch instead of through the yaml files, I will include the json from the config instead.

Selection_165

roles.json

{
    "NEETOP_role" : {
      "reserved" : false,
      "hidden" : false,
      "cluster_permissions" : [ ],
      "index_permissions" : [
        {
          "index_patterns" : [
            "opensearch_*"
          ],
          "dls" : "",
          "fls" : [ ],
          "masked_fields" : [ ],
          "allowed_actions" : [
            "data_access",
            "kibana_all_read",
            "read",
            "cluster:admin/opendistro/reports/definition/create",
            "cluster:admin/opendistro/reports/definition/update",
            "cluster:admin/opendistro/reports/definition/on_demand",
            "cluster:admin/opendistro/reports/definition/delete",
            "cluster:admin/opendistro/reports/definition/get",
            "cluster:admin/opendistro/reports/definition/list",
            "cluster:admin/opendistro/reports/instance/list",
            "cluster:admin/opendistro/reports/instance/get",
            "cluster:admin/opendistro/reports/menu/download",
            "create_index"
          ]
        }
      ],
      "tenant_permissions" : [
        {
          "tenant_patterns" : [
            "NETOP_tenant"
          ],
          "allowed_actions" : [
            "kibana_all_write"
          ]
        }
      ],
      "static" : false
    }
  }

  
  {
    "kibana_user_NETOP" : {
      "reserved" : false,
      "hidden" : false,
      "cluster_permissions" : [
        "cluster_composite_ops"
      ],
      "index_permissions" : [
        {
          "index_patterns" : [
            ".kibana",
            ".kibana-6",
            ".kibana_*"
          ],
          "fls" : [ ],
          "masked_fields" : [ ],
          "allowed_actions" : [
            "read",
            "delete",
            "manage",
            "index",
            "create_index"
          ]
        },
        {
          "index_patterns" : [
            ".tasks",
            ".management-beats",
            "*:.tasks",
            "*:.management-beats"
          ],
          "fls" : [ ],
          "masked_fields" : [ ],
          "allowed_actions" : [
            "indices_all"
          ]
        }
      ],
      "tenant_permissions" : [
        {
          "tenant_patterns" : [
            "NETOP_tenant"
          ],
          "allowed_actions" : [
            "kibana_all_write"
          ]
        }
      ],
      "static" : false
    }
  }

roles_mappings.json

{
    "NEETOP_role" : {
      "hosts" : [ ],
      "users" : [ ],
      "reserved" : false,
      "hidden" : false,
      "backend_roles" : [
        "adgroup1"
      ],
      "and_backend_roles" : [ ]
    }
  }

  {
    "kibana_user_NETOP" : {
      "hosts" : [ ],
      "users" : [ ],
      "reserved" : false,
      "hidden" : false,
      "backend_roles" : [
        "adgroup1"
      ],
      "and_backend_roles" : [ ]
    }
  }
  

roles.yml

---

_meta:
  type: "roles"
  config_version: 2

# Restrict users so they can only view visualization and dashboard on OpenSearchDashboards
kibana_read_only:
  reserved: true

# The security REST API access role is used to assign specific users access to change the security settings through the REST API.
security_rest_api_access:
  reserved: true

# Allows users to view monitors, destinations and alerts
alerting_read_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/alerting/alerts/get'
    - 'cluster:admin/opendistro/alerting/destination/get'
    - 'cluster:admin/opendistro/alerting/monitor/get'
    - 'cluster:admin/opendistro/alerting/monitor/search'

# Allows users to view and acknowledge alerts
alerting_ack_alerts:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/alerting/alerts/*'

# Allows users to use all alerting functionality
alerting_full_access:
  reserved: true
  cluster_permissions:
    - 'cluster_monitor'
    - 'cluster:admin/opendistro/alerting/*'
  index_permissions:
    - index_patterns:
        - '*'
      allowed_actions:
        - 'indices_monitor'
        - 'indices:admin/aliases/get'
        - 'indices:admin/mappings/get'

# Allow users to read Anomaly Detection detectors and results
anomaly_read_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/ad/detector/info'
    - 'cluster:admin/opendistro/ad/detector/search'
    - 'cluster:admin/opendistro/ad/detectors/get'
    - 'cluster:admin/opendistro/ad/result/search'
    - 'cluster:admin/opendistro/ad/tasks/search'

# Allows users to use all Anomaly Detection functionality
anomaly_full_access:
  reserved: true
  cluster_permissions:
    - 'cluster_monitor'
    - 'cluster:admin/opendistro/ad/*'
  index_permissions:
    - index_patterns:
        - '*'
      allowed_actions:
        - 'indices_monitor'
        - 'indices:admin/aliases/get'
        - 'indices:admin/mappings/get'

# Allows users to read Notebooks
notebooks_read_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/notebooks/list'
    - 'cluster:admin/opendistro/notebooks/get'

# Allows users to all Notebooks functionality
notebooks_full_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/notebooks/create'
    - 'cluster:admin/opendistro/notebooks/update'
    - 'cluster:admin/opendistro/notebooks/delete'
    - 'cluster:admin/opendistro/notebooks/get'
    - 'cluster:admin/opendistro/notebooks/list'

# Allows users to read and download Reports
reports_instances_read_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/reports/instance/list'
    - 'cluster:admin/opendistro/reports/instance/get'
    - 'cluster:admin/opendistro/reports/menu/download'

# Allows users to read and download Reports and Report-definitions
reports_read_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/reports/definition/get'
    - 'cluster:admin/opendistro/reports/definition/list'
    - 'cluster:admin/opendistro/reports/instance/list'
    - 'cluster:admin/opendistro/reports/instance/get'
    - 'cluster:admin/opendistro/reports/menu/download'

# Allows users to all Reports functionality
reports_full_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/reports/definition/create'
    - 'cluster:admin/opendistro/reports/definition/update'
    - 'cluster:admin/opendistro/reports/definition/on_demand'
    - 'cluster:admin/opendistro/reports/definition/delete'
    - 'cluster:admin/opendistro/reports/definition/get'
    - 'cluster:admin/opendistro/reports/definition/list'
    - 'cluster:admin/opendistro/reports/instance/list'
    - 'cluster:admin/opendistro/reports/instance/get'
    - 'cluster:admin/opendistro/reports/menu/download'

# Allows users to use all asynchronous-search functionality
asynchronous_search_full_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/asynchronous_search/*'
  index_permissions:
    - index_patterns:
        - '*'
      allowed_actions:
        - 'indices:data/read/search*'

# Allows users to read stored asynchronous-search results
asynchronous_search_read_access:
  reserved: true
  cluster_permissions:
    - 'cluster:admin/opendistro/asynchronous_search/get'

# Allow user lcm collector to write to and create indexes
lcm_user:
  reserved: false
  cluster_permissions:
    - 'cluster:monitor/main'
  index_permissions:
    - index_patterns:
        - 'lcmdata-sw-*'
      allowed_actions:
        - 'write'
        - 'read'
        - 'view_index_metadata'
        - 'create_index'
        - 'indices:admin/get'
        - 'indices:admin/mappings/get'

@Jan If this user is able to view dashboards created in global tenant, then I am not able to reproduce the issue. I have set up roles exactly the same as your files, except I went further and put ‘*’ under index patterns for both roles, and still the tenant limitation is there.

Can you try to upload the configuration using securityadmin.sh script? Just in case API caused some issues.

SOLUTION:

Hi all, ok I found the solution after lots of head scratching. My colleague upon installing and doing initial setup jumped to the conclusion that opensearch dashboard isn’t kibana anymore so he renamed the index name in config.yml accordingly (index: ‘.opensearch-dashboards’). This caused all the rights that was set in roles to fail since they are looking for the .kibana pattern.

So long story short, make sure you use the same name everywhere.

Thanks for the help!