### Elasticsearch Version

All

### Problem Description

It's possible for ILM to attempt a double-invocation of the `create-snapshot` step due to a master fail-over during the snapshot. This causes ILM to get stuck in the `create-snapshot` ERROR step with the error that the snapshot already exists (or else is currently ongoing).

For remediation, you can either move the index back to the `cleanup-snapshot` step (**IN THE SAME PHASE**), which will delete the snapshot and start taking a new one:

```
POST _ilm/move/<your-index-here>
{
  "current_step": { 
    "phase": "cold",
    "action": "searchable_snapshot",
    "name": "ERROR"
  },
  "next_step": { 
    "phase": "cold",
    "action": "searchable_snapshot",
    "name": "cleanup-snapshot"
  }
}
```

Or, use the [delete snapshot API](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-snapshot-api.html) and simply delete the existing snapshots, since ILM keeps retrying the create-snapshot step and will recreate the snapshot with that name.

### Steps to Reproduce

It manifests in the following way:

1. Enter `create-snapshot` step
2. ILM issues the create snapshot request, which is ongoing
3. The master node fails or is shut down
4. A new master is elected
5. The new master invokes the `create-snapshot` step a second time
6. The step fails due the snapshot either already existing, or a snapshot request already ongoing


### Logs (if relevant)

You can usually diagnose this by looking to see if the master node changes between the first `create-snapshot` and subsequent `create-snapshot` invocation (ILM only executes on the master node, so the logs will come from a different node for the two invocations).

You may see logs such as:

```
[2021-09-27T18:20:22.182Z][WARN][org.elasticsearch.snapshots.SnapshotsService] [instance-0000000152] [myindex][mysnapshotname] failed to create snapshot
org.elasticsearch.snapshots.InvalidSnapshotNameException: [myrepo:mysnapshotname] Invalid snapshot name [mysnapshotname], snapshot with the same name is already in-progress
	at org.elasticsearch.snapshots.SnapshotsService.ensureSnapshotNameNotRunning(SnapshotsService.java:614) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.snapshots.SnapshotsService.access$5300(SnapshotsService.java:123) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.snapshots.SnapshotsService$2.execute(SnapshotsService.java:462) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$1.execute(BlobStoreRepository.java:480) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:48) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:691) ~[elasticsearch-7.14.2.jar:7.14.2]
```

And also:

```
org.elasticsearch.snapshots.SnapshotException: [myrepo-mysnapshotname/uuid] no longer master
```

And potentially a message about a snapshot with the same name already existing:

```
[2021-09-27T23:50:22.612Z][ERROR][org.elasticsearch.xpack.ilm.IndexLifecycleRunner] [instance-0000000152] policy [my-ilm-policy] for index [myindex] failed on step [{"phase":"cold","action":"searchable_snapshot","name":"create-snapshot"}]. Moving to ERROR step
org.elasticsearch.snapshots.InvalidSnapshotNameException: [myrepo:mysnapshotname] Invalid snapshot name [mysnapshotname], snapshot with the same name already exists
	at org.elasticsearch.snapshots.SnapshotsService.ensureSnapshotNameAvailableInRepo(SnapshotsService.java:754) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.snapshots.SnapshotsService.access$5200(SnapshotsService.java:123) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.snapshots.SnapshotsService$2.execute(SnapshotsService.java:459) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$1.execute(BlobStoreRepository.java:480) ~[elasticsearch-7.14.2.jar:7.14.2]
```
server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java
*
     * If the inner loop finds that a shard update task applies to a given snapshot and either a shard-snapshot or shard-clone operation in
     * it then it will update the state of the snapshot entry accordingly. If that update was a noop, then the task is removed from the
     * iteration as it was already applied before and likely just arrived on the master node again due to retries upstream.
     * If the update was not a noop, then it means that the shard it applied to is now available for another snapshot or clone operation
     * to be re-assigned if there is another snapshot operation that is waiting for the shard to become available. We therefore record the
     * fact that a task was executed by adding it to a collection of executed tasks. If a subsequent execution of the outer loop finds that
     * a task in the executed tasks collection applied to a shard it was waiting for to become available, then the shard snapshot operation
     * will be started for that snapshot entry and the task removed from the collection of tasks that need to be applied to snapshot
     * entries since it can not have any further effects.
     *
     * Package private to allow for tests.
     */
    static final ClusterStateTaskExecutor<ShardSnapshotUpdate> SHARD_STATE_EXECUTOR = (currentState, taskContexts) -> {
        for (var taskContext : taskContexts) {
            taskContext.success(new ClusterStateTaskExecutor.LegacyClusterTaskResultActionListener(taskContext.getTask(), currentState));
        }
        return new SnapshotShardsUpdateContext(
            currentState,
            () -> taskContexts.stream().map(ClusterStateTaskExecutor.TaskContext::getTask).iterator()
        ).computeUpdatedState();
    };
    private static boolean isQueued(@Nullable ShardSnapshotStatus status) {
        return status != null && status.state() == ShardState.QUEUED;
    }
    /**
     * State machine for updating existing {@link SnapshotsInProgress.Entry} by applying a given list of {@link ShardSnapshotUpdate} to
     * them.
     */
    private static final class SnapshotShardsUpdateContext {