Error fetching fields for index pattern metricbeat-*

iamthealex-es · October 23, 2020, 4:48pm

I’m using ODFE via debian packages.

opendistro-security/2021.1,now 1.7.0.0-0 amd64 [installed,automatic]
opendistroforelasticsearch-kibana/2021.1,now 1.7.0 amd64 [installed,automatic]
elasticsearch-oss/2021.1,now 7.6.1 amd64 [installed,automatic]
opendistroforelasticsearch-kibana/2021.1,now 1.7.0 amd64 [installed,automatic]

I have a readonly user that I want to be able to see the metricbeat visualiazations.
The metricbeat system dashboard installed by metricbeat does the display all the visualizations to my readonly user.

Probably Kibana is trying to write (according to the elasticsearch log message – see below, but not according to the displayed error message in the Kibana dashboard which claims there are fetch/read problems).

Randomly at the end of putting up the visualizations on the dashboard, there will be a spurious error message. That error message and error in the elasticsearch log file don’t occur when I’m logged in as the admin user.

At the time of the problem, I get a message in the elasticsearch log stating that

`2020-10-23T12:06:29,000][WARN ][c.a.o.s.c.PrivilegesInterceptorImpl] [10.43.186.10] Tenant global_tenant is not allowed to write (user: xxxx)

At the same time in the Kibana GUI, I get this:
forbidden

and if I click on “See the full error”, I get this:

I have tried various ways to make the problem go away by editing the security role associated with this user in my roles.yml ( I tried both “*” and “global_tenant” for the tenant_patterns)

read_only_index:
  reserved: true
  hidden: false
  cluster_permissions:
    - cluster_composite_ops_ro
    - cluster_monitor
  index_permissions:
    - index_patterns:
      - "*"
      allowed_actions:
      -  read
      -  search
      - cluster_monitor
      - indices_monitor
  tenant_permissions:
    - tenant_patterns:
      - "*"
      allowed_actions:
        - kibana_all_read

I have attempted to simplify tenancy issues, by disabling mult-tenancy.
Here is the relevant snippet from my kibana.yml:

opendistro_security.multitenancy.enabled: false
opendistro_security.readonly_mode.roles: ["kibana_read_only", "read_only_index"]

iamthealex-es · October 23, 2020, 8:36pm

It may be that I need to wait for ODFE to merge in the fix from [Discover] Prevent error message of read only user without default index pattern set by kertal · Pull Request #54122 · elastic/kibana · GitHub.

Anthony · March 15, 2021, 5:37pm

@iamthealex-es can you confirm how you created read_only_user, have you also mapped the user to readall role to be able to view the data?
If you disabling the multitenancy, did you also set the multitenancy_enabled setting to false in config.yml:

config:
  dynamic:
     kibana:
        multitenancy_enabled: false

iamthealex-es · March 18, 2021, 6:47pm

I did not set multitenancy to false in my config.yml.
But in my kibana.yml, I have

opendistro_security.multitenancy.enabled: false

in roles.yml, I created “read_only_index”.

read_only_index:
  reserved: true
  hidden: false
  cluster_permissions:
    - cluster_composite_ops_ro
    - cluster_monitor
  index_permissions:
    - index_patterns:
      - "*"
      allowed_actions:
      -  read
      -  search
      - cluster_monitor
  tenant_permissions:
    - tenant_patterns:
      - "global_tenant"

in roles_mapping I connect my readonly user with that read_only_index role:

read_only_index:
  reserved: true
  hidden: false
  users:
  - "readonly_user"

Anthony · July 20, 2021, 12:19pm

@iamthealex-es did you manage to resolve this?

If not, what version of odfe are you currently using? I will try to reproduce from my end

iamthealex-es · August 2, 2021, 5:12pm

The issue is not resolved.
I continue to see:

[2021-08-02T13:09:28,401][WARN ][c.a.o.s.c.PrivilegesInterceptorImpl] [10.43.122.10] Tenant global_tenant is not allowed to write (user: editshare)

I no longer see the long messages Kibana, I now only see a complaint about 403:

iamthealex-es · August 2, 2021, 5:18pm

My config.yml:


_meta:
  type: "config"
  config_version: 2

config:
  dynamic:
    http:
      anonymous_auth_enabled: false

    authc:
      basic_internal_auth_domain:
      	description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 4
	http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
    authz:

iamthealex-es · August 2, 2021, 5:20pm

My kibana.yml

lasticsearch.hosts: ['https://10.43.122.10:9200']
elasticsearch.ssl.verificationMode: none
elasticsearch.username: xxxxx
elasticsearch.password: yyyyy
elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization"]

opendistro_security.multitenancy.enabled: false
opendistro_security.readonly_mode.roles: ["kibana_read_only"]
opendistro_security.cookie.ttl: 86400000
opendistro_security.session.ttl: 86400000
opendistro_security.session.keepalive: true

newsfeed.enabled: false

telemetry.optIn: false
telemetry.enabled: false

server.ssl.enabled: true
server.ssl.certificate: /etc/aaaaa/ssl/server.crt
server.ssl.key: /etc/aaaaa/ssl/server.key
server.host: 0.0.0.0

kibana.defaultAppId: "dashboard/fa8f5fa0-a43a-11ea-8f7c-d358310f8df1"

iamthealex-es · August 2, 2021, 5:21pm

elasticsearch-oss/now 7.6.1 amd64 [installed,local]
opendistro-index-management/now 1.7.0.0-1 amd64 [installed,local]
opendistro-job-scheduler/now 1.7.0.0-1 amd64 [installed,local]
opendistro-security/now 1.7.0.0-0 amd64 [installed,local]
opendistroforelasticsearch-kibana/now 1.7.0 amd64 [installed,local]

Anthony · August 3, 2021, 8:56am

@iamthealex-es
Can you confirm that the user is mapped to the readonly role by running below command and pasting output here:

curl --insecure -u user:password -XGET “https://localhost:9200/_opendistro/_security/authinfo?pretty”

If the user is indeed mapped to the role you’ve mentioned, than I am not able to reproduce the issue. Have you tried upgrading from odfe 1.7 to more recent releases?

iamthealex-es · August 3, 2021, 2:01pm

The user is mapped to the backend_role not by adding backend_roles in internal_users.yml, but by the mapping in roles.yml:

read_only_index:
  reserved: true
  hidden: false
  users:
  - "editshare"

$ curl --insecure -u editshare https://localhost:9200/_opendistro/_security/authinfo?pretty
Enter host password for user 'editshare':
{
  "user" : "User [name=editshare, backend_roles=[], requestedTenant=null]",
  "user_name" : "editshare",
  "user_requested_tenant" : null,
  "remote_address" : "[::1]:51830",
  "backend_roles" : [ ],
  "custom_attribute_names" : [ ],
  "roles" : [
    "read_only_index"
  ],
  "tenants" : {
    "global_tenant" : false,
    "editshare" : true
  },
  "principal" : null,
  "peer_certificates" : "0",
  "sso_logout_url" : null
}

and the permissions for the readonly index are given in roles.yml:


read_only_index:
  reserved: true
  hidden: false
  cluster_permissions:
    - cluster_composite_ops_ro
    - cluster_monitor
  index_permissions:
    - index_patterns:
      - "*"
      allowed_actions:
      -  read
      -  search
      - cluster_monitor
  tenant_permissions:
    - tenant_patterns:
      - "global_tenant"

I gave you the wrong information for my version.
I’m using the 7.10 version of elasticsearch: elasticsearch-oss/unknown,now 7.10.2 amd64 [installed,automatic]
and the 1.13.2.0-1 version of opendistro-security: opendistro-security/unknown,now 1.13.1.0-1 all [installed,automatic]

Apparently only some of my dashboards result in the error message.
And, it only happens occasionally and only on my dashboards that access metricbeat data.
I will try to narrow it down to a particular index, but I may just have for a full stack from opensearch.org to get around this issue.

I really appreciate you trying to debug this for me (and teaching me the “_security/authinfo?pretty” trick.

Anthony · August 3, 2021, 3:02pm

@iamthealex-es Are you able to see all the fields using “editshare” user in discover tab for index pattern metricbeat-* ?

iamthealex-es · August 3, 2021, 4:09pm

Yes, I can see all the fields in metricbeat-* in the discover tab when I’m logged in as editshare.

Anthony · August 3, 2021, 4:16pm

based on previous output the editshare user is using “editshare” tenant. Can you either enable the full tenancy in kibana.yml or disable from config.yml to see if this has any effect on the issue.
Also, what tenant are the visualisations stored on?

Topic		Replies	Views
Cross cluster search with secured local and remote clusters Open Source Elasticsearch and Kibana	4	1214	April 6, 2021
BUG: Indices:data/read/async_search/submit error Security	2	864	March 15, 2021
Error during creation of user for Filebeat in order to connect General Feedback	1	1457	August 21, 2020
Kibana wont import visualizations and dashboards Open Source Elasticsearch and Kibana	4	1920	January 6, 2020
Unable to create kibana index patterns Open Source Elasticsearch and Kibana	3	1066	August 6, 2021

Error fetching fields for index pattern metricbeat-*

Related Topics