Snapshot and Restore of .opendistro_security index is behaving improper

General Feedback
1

Hi,
I am using ES, Kibana 7.10.2 and using opendistro-security plugin 1.13.x. I have tried below 2 scenarios on OD security index Snapshot/Restore and details are as below:

Case 1:
a. Add user and respective roles, rolesmapping. Access the ES cluster using this user.
b. Take Snapshot of .opendistro_security index using admin certificates.
c. Delete .opendistro_security index from the cluster
d. Restore .opendistro_security index into the cluster.
This scenario works fine. After restore I am able to access ES cluster using the user created in step a.

Case 2:
a. Add user and respective roles, rolesmapping. Access the ES cluster using this user.
b. Take Snapshot of .opendistro_security index using admin certificates.
c. Delete only user. Roles, rolesmappings are not deleted.
d. Delete .opendistro_security index from the cluster before restore
e. Restore .opendistro_security index into the cluster.
After restore the user is restored back(I can see user in Kibana UI Internalusers).
But I am unable to access ES cluster using the user which have deleted in step c.
When I try to curl to ES it says Unauthorized. Also I can’t login to Kibana UI using this user.

Command used for Snapshot

curl -X PUT --cert <path-to-client.crt> --key <path-to-client.key>  https://<ELASTICSEARCH_SERVICE>:<ES_PORT>/_snapshot/es_backup/snapshot_1?wait_for_completion=true -H 'Content-Type: application/json' -d' {"ignore_unavailable": true,"include_global_state": false, "indices": ".opendistro_security" }'

Command used for Restore

curl  -XPOST --cert <path-to-client.crt> --key <path-to-client.key> https://<ELASTICSEARCH_SERVICE>:<ES_PORT>/_snapshot/es_backup/snapshot_1/_restore?wait_for_completion=true -H 'Content-Type: application/json' -d' { "indices": ".opendistro_security", "ignore_unavailable": true,"include_global_state": false }'

Why is it not working as expected?

2

I learned from this post: Users gone after using securityadmin.sh - #3 by DotDat

Question: Can i export the settings i did to the .yml files?

Answer: 
https://opendistro.github.io/for-elasticsearch-docs/docs/security/configuration/security-admin/#backup-restore-and-migrate

For user and permissions and others in those config files.

./securityadmin.sh -backup /root/odfe/ -h HOST_IP -icl -nhnv -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/kirk.pem -key /etc/elasticsearch/kirk-key.pem

After that, you have to import again. I created cronjob for backup every day for those settings since it can’t be taken in the snapshot.

3

Hi,
It’s mentioned in Take and Restore Snapshots - Open Distro Documentation that using admin certificates snapshot/restore could be performed. I’m also trying to use the same.
I was able to overcome with the above issue when I tried running Purge cache API API - Open Distro for Elasticsearch Documentation after ES Restore API

DELETE _opendistro/_security/api/cache
4

I’ll just drop this here for how it can be done in kubernetes with OpenSearch.
The “master” node has permanent storage, so regular backups can be dropped there. It could of course also be copied to another node.

kubectl exec -it opensearch-master-0 -- bash -c '
  umask 022
  BDIR=data/backup/backup-$(date +%Y%m%d%H%M%S)
  mkdir -p ${BDIR}
  bash plugins/opensearch-security/tools/securityadmin.sh \
    -backup ${BDIR} \
    -icl \
    -nhnv \
    -cacert config/certificates/root-ca.pem \
    -cert config/certificates/admin.pem \
    -key config/certificates/admin-key.pem \
    -h localhost
  ls -l ${BDIR}'