Users lost access to own_index and tenant indices

I patched the OS of our ES cluster today and suddenly users are seeing a blank white screen when trying to access the “Discover” app in Kibana.

Looking at the logs, its because the user can no longer access the indices in question

No index-level perm match for User [name=user.name, backend_roles=[Incident Response Users], requestedTenant=Incident Response] Resolved [aliases=[.kibana_-1461739569_incidentresponse], indices=[], allIndices=[.kibana_-1461739569_incidentresponse_2], types=[*], originalRequested=[.kibana_-1461739569_incidentresponse, .kibana_-1461739569_incidentresponse_2], remoteIndices=[]] [Action [indices:data/read/mget[shard]]] [RolesChecked [Incident Response Users, own_index]]

I am able to create a new tenant and grant access to that and it all works. If I delete the users .kibana indices they get created and that works too…

As an administrator I have access without issue. If I gave access specifically to these indices it works, but that is not how it’s supposed to be.

Any idea what went wrong so I can keep this from happening again?

After more troubleshooting, it seems that kibana_all_read and kibana_all_write is not enough. I get these errors until I add the crud permission to the index.