Snapshot stuck in IN_PROGRESS in _all, but shows as SUCCEEDED in _status

I upgraded to OpenSearch 1.2.1 from OpenDistro. When I take a snapshot, it is shown as in IN_PROGRESS:

backup-20211213112202 SUCCESS 1639394522 11:22:02 1639394525 11:22:05 2.6s 156 156 0 156
jz IN_PROGRESS 1639563632 10:20:32 0 00:00:00 55.8m 156 0 0 0

Here we see “jz” as the last snapshot taken by me. The previous snapshot took only 2.6 seconds, while the snapshot taken by me is taking 55 minutes and counting. This prevents other snapshots from being taken.

Strange thing is that it shows as SUCCEEDED when I look up the status of the snapshot:

{
    "snapshots": [
        {
            "snapshot": "jz",
            "repository": "os_snapshot",
            "uuid": "Jf2PujbdR22ViyYoFKtSUg",
            "state": "SUCCESS",

I tried to delete the “jz” snapshot but it hanging there. The delete request is “Sending request…” in Postman.

I have restarted all data nodes, and all master nodes. But I can still not delete the snapshot and it is still shown as IN_PROGRESS.

What more can I do to recover from this?

I see bug report has been created here: [BUG] repository-azure plugin hangs in OpenSearch >= 1.2.0 · Issue #1734 · opensearch-project/OpenSearch · GitHub

Fixed in OpenSearch 1.2.3, released today. Thanks for reporting the issue!

2 Likes