Saml cookie refresh/sso redirect issue (worked before upgrade)

Security
1

Hi all!

I have been using saml/sso with opendistro since version 1.8.0 but recently I noticed issues with the redirect to sso once the session length hits 1hr. Here is what happens:

Saml/sso is working fine to log into the platform (I am upgraded to opensearch 1.0.1 now running on docker swarm). Opensearch Dashboards is behind an f5 loadbalancer which provides certificates for the communication to the client.
If I click logout manually, it sends me back to sso which in turn forwards me to opensearch as it should since my sso session is still active which is how it is configured.

The problem is that after an hour, instead of redirecting to saml/sso it shows

{"statusCode":401,"error":"Unauthorized","message":"Response Error"}

in the browser. Once I am in this state, I can still see the security_authentication cookie change if I refresh.
The way to get out of this state is to delete the security_authentication cookie at which point it sends me back to sso and I get back into a good session (without needing to authenticate).

Here is my opensearch.yml jinja file

cluster.name: {{ clustername }}
node.name: {{ item.name }}
network.host: 0.0.0.0

discovery.seed_hosts: ["{{ master[0].name }}","{{ master[1].name }}","{{ master[2].name }}"]
cluster.initial_master_nodes: ["{{ master[0].name }}","{{ master[1].name }}","{{ master[2].name }}"]

node.master: {{ item.master }}
node.data: {{ item.data }}
node.ingest: {{ item.ingest }}

######## Start OpenDistro for Elasticsearch Security Configuration ########
plugins.security.ssl.transport.pemcert_filepath: odfe-{{ item.name }}.pem
plugins.security.ssl.transport.pemkey_filepath: odfe-{{ item.name }}-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: odfe-es-root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: odfe-{{ item.name }}.pem
plugins.security.ssl.http.pemkey_filepath: odfe-{{ item.name }}-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: odfe-es-root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=admin,OU=SSL,O={{ clustername }},L=YEG,ST=AB,C=CA
plugins.security.nodes_dn:
  - 'CN=*,OU=ES,O={{ clustername }},L=YEG,ST=AB,C=CA'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
######### End OpenDistro for Elasticsearch Security Configuration ########

Here is my opensearch_dashboards.yml jinja file:

server.name: {{ item.name }}
server.host: "0.0.0.0"
opensearch.hosts: ["https://{{ coordinating[0].name }}:9200", "https://{{ coordinating[1].name }}:9200"]
opensearch.username: kibanaserver
opensearch.password: {{ elastic_kibana_pass }}
opensearch.requestHeadersWhitelist: ["securitytenant","Authorization"]

opensearch.ssl.verificationMode: full
opensearch.ssl.certificateAuthorities: ["/usr/share/opensearch-dashboards/config/odfe-es-root-ca.pem"]

opensearch_security.multitenancy.enabled: false
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]

# Use this setting if you are running kibana without https
#plugins.security.cookie.secure: true This is in the documentation but does not work!
opensearch_security.cookie.secure: true

server.port: 5601

# Enable SAML authentication
#plugins.security.auth.type: "saml"
opensearch_security.auth.type: "saml"
server.xsrf.whitelist: ["/_opendistro/_security/saml/acs"]

My config.yaml jinja looks like this

      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
      saml_auth_domain:
        http_enabled: true
        transport_enabled: false
        order: 1
        http_authenticator:
            type: 'saml'
            challenge: true
            config:
                idp:
                    metadata_file: {{ es_saml_idp_metadata_file }}
                    entity_id: {{ es_saml_idp_entity_id }}
                sp:
                    entity_id: {{ es_saml_sp_entity_id }}
                    forceAuthn: false
                kibana_url: {{ es_saml_kibana_url }}
                subject_key: {{ es_saml_subject_key }}
                roles_key: {{ es_saml_roles_key }}
                exchange_key: '{{ es_saml_exchange_key }}'
        authentication_backend:
            type: noop

To me this looks like a bug that was introduced along the way. Did anyone else run into this?

The opensearch desktop container logs the following error when I refresh the browser in this state:

{"type":"log","@timestamp":"2021-09-10T20:58:02Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"response","@timestamp":"2021-09-10T20:58:02Z","tags":[],"pid":1,"method":"post","statusCode":401,"req":{"url":"/internal/search/opensearch","method":"post","headers":{"host":"dashboards.domain.com","connection":"keep-alive","content-length":"1414","sec-ch-ua":"\"Google Chrome\";v=\"93\", \" Not;A Brand\";v=\"99\", \"Chromium\";v=\"93\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36","osd-version":"1.0.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://dashboards.domain.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://dashboards.domain.com/app/discover","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-for":"10.253.7.241","x-forwarded-proto":"https","x-forwarded-port":"443"},"remoteAddress":"10.255.0.8","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36","referer":"https://dashboards.domain.com/app/discover"},"res":{"statusCode":401,"responseTime":38,"contentLength":9},"message":"POST /internal/search/opensearch 401 38ms - 9.0B"}
{"type":"log","@timestamp":"2021-09-10T20:58:03Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"response","@timestamp":"2021-09-10T20:58:03Z","tags":[],"pid":1,"method":"get","statusCode":401,"req":{"url":"/app/discover","method":"get","headers":{"host":"dashboards.domain.com","connection":"keep-alive","cache-control":"max-age=0","sec-ch-ua":"\"Google Chrome\";v=\"93\", \" Not;A Brand\";v=\"99\", \"Chromium\";v=\"93\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"same-origin","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","referer":"https://dashboards.domain.com/app/discover","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-for":"10.253.7.241","x-forwarded-proto":"https","x-forwarded-port":"443"},"remoteAddress":"10.255.0.8","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36","referer":"https://dashboards.domain.com/app/discover"},"res":{"statusCode":401,"responseTime":13,"contentLength":9},"message":"GET /app/discover 401 13ms - 9.0B"}
{"type":"response","@timestamp":"2021-09-10T20:58:03Z","tags":[],"pid":1,"method":"get","statusCode":404,"req":{"url":"/favicon.ico","method":"get","headers":{"host":"dashboards.domain.com","connection":"keep-alive","sec-ch-ua":"\"Google Chrome\";v=\"93\", \" Not;A Brand\";v=\"99\", \"Chromium\";v=\"93\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"image","referer":"https://dashboards.domain.com/app/discover","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-for":"10.253.7.241","x-forwarded-proto":"https","x-forwarded-port":"443"},"remoteAddress":"10.255.0.8","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36","referer":"https://dashboards.domain.com/app/discover"},"res":{"statusCode":404,"responseTime":7,"contentLength":9},"message":"GET /favicon.ico 404 7ms - 9.0B"}

What I am thinking is that instead of showing a 401, dashboards should redirect to sso and extend the session.

2

Hi @lorenzo95

Have you found a solution to this issue? What is your SAML IDP solution?

3

Hello!

Thank you for following up. No, I have not found a solution.
I am using SAML against shibboleth idp (https://www.shibboleth.net/). I worked with the identity team to check the logs on the shibboleth end. When I receive the 401 error, nothing from the opensearch stack is trying to reach out to idp. No message is received and no error is logged.

It feels like an issue inside elastic security but I don’t know how to further debug it.

Thank you,
Gero

4

Hello,

I was having this problem also but not all the time. I could logout and back in, no problem. After an hour user was automatically logged out and tried to log back in I would receive error 401. Unfortunately, I’m using Keycloak. I solved my problem with

Logout Service Redirect Binding URL

https://elastic-stack.domain.com:5601/
5

@lorenzo95 All your configuration values are updated with opensearch specific settings. May I know why the following setting is still pointing to open distro API?

6

Oh that is interesting! Thank you for pointing that out. As per the documentation here

I should change that to:

server.xsrf.whitelist: ["/_plugins/_security/saml/acs"]

I am not sure why I left it. I changed the first part. Let me change it and report back. I would love it if the fix was this obvious!

7

Hello, thanks for the reply. We actually don’t let anyone log out of from here since it would also disconnect all other services such as 365 etc. I did add

forceAuthn: false

to my sp config in config.yaml above. When you click logout you are just going through a loop with sso.

8

I gave that a try.

I changed

server.xsrf.whitelist: ["/_opendistro/_security/saml/acs"]

to

server.xsrf.whitelist: ["/_plugins/_security/saml/acs"]

and restarted opensearch-dashboards.
On the idp side I changed the sp metadata to match and restarted

https://dashboards.domain.com/_plugins/_security/saml/acs

However!!!

Using an incognito browser, it then doesn’t authenticate at all anymore. The reason being is that opensearch-dashboards still generates the opendistro ednpoint url

auth request:AssertionConsumerServiceURL="https://dashboards.domain.com/_opendistro/_security/saml/acs"

which then generates the error in the browser

{
statusCode: 400,
error: "Bad Request",
message: "Request must contain a osd-xsrf header."
}

and of course it throws an error in the idp logs since it’s being told to use the opendistro endpoint by dashboards but it’s not configured anymore.

I also tried leaving idp on opendistro and only changing the whitelist which of course doesn’t work as well. It all has to match.

It looks like the current plugin still uses the opendistro endpoint and did not quite migrated over.
That is at least using the current opensearchproject/opensearch-dashboards:1.0.1 container.
Is there a step in the migration to change that or is it supposed to be like this?

Thank you!

9

Exactly!!

I am facing the same issue while providing the new opensearch API. This must be a bug.

When I tried with opendistro endpoint, I could login without any issues.

10

Hello!

Just to update, this 401 error issue is still occurring in 1.1.0

Thank you

11

Hello,
I am facing the same issue in 1.1.0

12

Since we all upgraded to 1.2.0/1 now (cough log4j), the issue persists.

1 Like
13

This is happening to me also. Upgraded to 1.2.3 opensearch and 1.2.0 opensearch-dashboards (for some reason they’re not in lock-step…) and replaced the following param in opensearch_dashboards.yml

server.xsrf.whitelist:
[
“/_opendistro/_security/saml/acs/idpinitiated”,
“/_opendistro/_security/saml/acs”,
“/_opendistro/_security/saml/logout”,
]

with:

server.xsrf.whitelist:
[
“/_plugins/_security/saml/acs/idpinitiated”,
“/_plugins/_security/saml/acs”,
“/_plugins/_security/saml/logout”,
]

as well as updating my IdP (Okta) with the same URL.

It fails with the following error:

{"statusCode":404,"error":"Not Found","message":"Not Found"}

Any update on when this is going to be documented? And fixed? There is a lot to be desired from the lack of documentation on this project. It was an issue with opendistro and seems to have carried over to opensearch. Even going through the upgrade/migration docs was not fully comprehensive as to everything that was required. Sigh…

14

This post is really to track a saml authentication (401 unauthorized) issue.
The issue you are describing has a bug open on github here:
https://github.com/opensearch-project/security-dashboards-plugin/issues/836

Cheers

1 Like