Saml cookie refresh/sso redirect issue (worked before upgrade)

Hi all!

I have been using saml/sso with opendistro since version 1.8.0 but recently I noticed issues with the redirect to sso once the session length hits 1hr. Here is what happens:

Saml/sso is working fine to log into the platform (I am upgraded to opensearch 1.0.1 now running on docker swarm). Opensearch Dashboards is behind an f5 loadbalancer which provides certificates for the communication to the client.
If I click logout manually, it sends me back to sso which in turn forwards me to opensearch as it should since my sso session is still active which is how it is configured.

The problem is that after an hour, instead of redirecting to saml/sso it shows

{"statusCode":401,"error":"Unauthorized","message":"Response Error"}

in the browser. Once I am in this state, I can still see the security_authentication cookie change if I refresh.
The way to get out of this state is to delete the security_authentication cookie at which point it sends me back to sso and I get back into a good session (without needing to authenticate).

Here is my opensearch.yml jinja file

cluster.name: {{ clustername }}
node.name: {{ item.name }}
network.host: 0.0.0.0

discovery.seed_hosts: ["{{ master[0].name }}","{{ master[1].name }}","{{ master[2].name }}"]
cluster.initial_master_nodes: ["{{ master[0].name }}","{{ master[1].name }}","{{ master[2].name }}"]

node.master: {{ item.master }}
node.data: {{ item.data }}
node.ingest: {{ item.ingest }}

######## Start OpenDistro for Elasticsearch Security Configuration ########
plugins.security.ssl.transport.pemcert_filepath: odfe-{{ item.name }}.pem
plugins.security.ssl.transport.pemkey_filepath: odfe-{{ item.name }}-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: odfe-es-root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: odfe-{{ item.name }}.pem
plugins.security.ssl.http.pemkey_filepath: odfe-{{ item.name }}-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: odfe-es-root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=admin,OU=SSL,O={{ clustername }},L=YEG,ST=AB,C=CA
plugins.security.nodes_dn:
  - 'CN=*,OU=ES,O={{ clustername }},L=YEG,ST=AB,C=CA'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
######### End OpenDistro for Elasticsearch Security Configuration ########

Here is my opensearch_dashboards.yml jinja file:

server.name: {{ item.name }}
server.host: "0.0.0.0"
opensearch.hosts: ["https://{{ coordinating[0].name }}:9200", "https://{{ coordinating[1].name }}:9200"]
opensearch.username: kibanaserver
opensearch.password: {{ elastic_kibana_pass }}
opensearch.requestHeadersWhitelist: ["securitytenant","Authorization"]

opensearch.ssl.verificationMode: full
opensearch.ssl.certificateAuthorities: ["/usr/share/opensearch-dashboards/config/odfe-es-root-ca.pem"]

opensearch_security.multitenancy.enabled: false
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]

# Use this setting if you are running kibana without https
#plugins.security.cookie.secure: true This is in the documentation but does not work!
opensearch_security.cookie.secure: true

server.port: 5601

# Enable SAML authentication
#plugins.security.auth.type: "saml"
opensearch_security.auth.type: "saml"
server.xsrf.whitelist: ["/_opendistro/_security/saml/acs"]

My config.yaml jinja looks like this

      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 0
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: internal
      saml_auth_domain:
        http_enabled: true
        transport_enabled: false
        order: 1
        http_authenticator:
            type: 'saml'
            challenge: true
            config:
                idp:
                    metadata_file: {{ es_saml_idp_metadata_file }}
                    entity_id: {{ es_saml_idp_entity_id }}
                sp:
                    entity_id: {{ es_saml_sp_entity_id }}
                    forceAuthn: false
                kibana_url: {{ es_saml_kibana_url }}
                subject_key: {{ es_saml_subject_key }}
                roles_key: {{ es_saml_roles_key }}
                exchange_key: '{{ es_saml_exchange_key }}'
        authentication_backend:
            type: noop

To me this looks like a bug that was introduced along the way. Did anyone else run into this?

The opensearch desktop container logs the following error when I refresh the browser in this state:

{"type":"log","@timestamp":"2021-09-10T20:58:02Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"response","@timestamp":"2021-09-10T20:58:02Z","tags":[],"pid":1,"method":"post","statusCode":401,"req":{"url":"/internal/search/opensearch","method":"post","headers":{"host":"dashboards.domain.com","connection":"keep-alive","content-length":"1414","sec-ch-ua":"\"Google Chrome\";v=\"93\", \" Not;A Brand\";v=\"99\", \"Chromium\";v=\"93\"","content-type":"application/json","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36","osd-version":"1.0.0","sec-ch-ua-platform":"\"Windows\"","accept":"*/*","origin":"https://dashboards.domain.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://dashboards.domain.com/app/discover","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-for":"10.253.7.241","x-forwarded-proto":"https","x-forwarded-port":"443"},"remoteAddress":"10.255.0.8","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36","referer":"https://dashboards.domain.com/app/discover"},"res":{"statusCode":401,"responseTime":38,"contentLength":9},"message":"POST /internal/search/opensearch 401 38ms - 9.0B"}
{"type":"log","@timestamp":"2021-09-10T20:58:03Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"response","@timestamp":"2021-09-10T20:58:03Z","tags":[],"pid":1,"method":"get","statusCode":401,"req":{"url":"/app/discover","method":"get","headers":{"host":"dashboards.domain.com","connection":"keep-alive","cache-control":"max-age=0","sec-ch-ua":"\"Google Chrome\";v=\"93\", \" Not;A Brand\";v=\"99\", \"Chromium\";v=\"93\"","sec-ch-ua-mobile":"?0","sec-ch-ua-platform":"\"Windows\"","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"same-origin","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","referer":"https://dashboards.domain.com/app/discover","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-for":"10.253.7.241","x-forwarded-proto":"https","x-forwarded-port":"443"},"remoteAddress":"10.255.0.8","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36","referer":"https://dashboards.domain.com/app/discover"},"res":{"statusCode":401,"responseTime":13,"contentLength":9},"message":"GET /app/discover 401 13ms - 9.0B"}
{"type":"response","@timestamp":"2021-09-10T20:58:03Z","tags":[],"pid":1,"method":"get","statusCode":404,"req":{"url":"/favicon.ico","method":"get","headers":{"host":"dashboards.domain.com","connection":"keep-alive","sec-ch-ua":"\"Google Chrome\";v=\"93\", \" Not;A Brand\";v=\"99\", \"Chromium\";v=\"93\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36","sec-ch-ua-platform":"\"Windows\"","accept":"image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8","sec-fetch-site":"same-origin","sec-fetch-mode":"no-cors","sec-fetch-dest":"image","referer":"https://dashboards.domain.com/app/discover","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9","x-forwarded-for":"10.253.7.241","x-forwarded-proto":"https","x-forwarded-port":"443"},"remoteAddress":"10.255.0.8","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.63 Safari/537.36","referer":"https://dashboards.domain.com/app/discover"},"res":{"statusCode":404,"responseTime":7,"contentLength":9},"message":"GET /favicon.ico 404 7ms - 9.0B"}

What I am thinking is that instead of showing a 401, dashboards should redirect to sso and extend the session.