### Elasticsearch Version

All

### Problem Description

It's possible for ILM to attempt a double-invocation of the `create-snapshot` step due to a master fail-over during the snapshot. This causes ILM to get stuck in the `create-snapshot` ERROR step with the error that the snapshot already exists (or else is currently ongoing).

For remediation, you can either move the index back to the `cleanup-snapshot` step (**IN THE SAME PHASE**), which will delete the snapshot and start taking a new one:

```
POST _ilm/move/<your-index-here>
{
  "current_step": { 
    "phase": "cold",
    "action": "searchable_snapshot",
    "name": "ERROR"
  },
  "next_step": { 
    "phase": "cold",
    "action": "searchable_snapshot",
    "name": "cleanup-snapshot"
  }
}
```

Or, use the [delete snapshot API](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-snapshot-api.html) and simply delete the existing snapshots, since ILM keeps retrying the create-snapshot step and will recreate the snapshot with that name.

### Steps to Reproduce

It manifests in the following way:

1. Enter `create-snapshot` step
2. ILM issues the create snapshot request, which is ongoing
3. The master node fails or is shut down
4. A new master is elected
5. The new master invokes the `create-snapshot` step a second time
6. The step fails due the snapshot either already existing, or a snapshot request already ongoing


### Logs (if relevant)

You can usually diagnose this by looking to see if the master node changes between the first `create-snapshot` and subsequent `create-snapshot` invocation (ILM only executes on the master node, so the logs will come from a different node for the two invocations).

You may see logs such as:

```
[2021-09-27T18:20:22.182Z][WARN][org.elasticsearch.snapshots.SnapshotsService] [instance-0000000152] [myindex][mysnapshotname] failed to create snapshot
org.elasticsearch.snapshots.InvalidSnapshotNameException: [myrepo:mysnapshotname] Invalid snapshot name [mysnapshotname], snapshot with the same name is already in-progress
	at org.elasticsearch.snapshots.SnapshotsService.ensureSnapshotNameNotRunning(SnapshotsService.java:614) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.snapshots.SnapshotsService.access$5300(SnapshotsService.java:123) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.snapshots.SnapshotsService$2.execute(SnapshotsService.java:462) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$1.execute(BlobStoreRepository.java:480) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:48) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:691) ~[elasticsearch-7.14.2.jar:7.14.2]
```

And also:

```
org.elasticsearch.snapshots.SnapshotException: [myrepo-mysnapshotname/uuid] no longer master
```

And potentially a message about a snapshot with the same name already existing:

```
[2021-09-27T23:50:22.612Z][ERROR][org.elasticsearch.xpack.ilm.IndexLifecycleRunner] [instance-0000000152] policy [my-ilm-policy] for index [myindex] failed on step [{"phase":"cold","action":"searchable_snapshot","name":"create-snapshot"}]. Moving to ERROR step
org.elasticsearch.snapshots.InvalidSnapshotNameException: [myrepo:mysnapshotname] Invalid snapshot name [mysnapshotname], snapshot with the same name already exists
	at org.elasticsearch.snapshots.SnapshotsService.ensureSnapshotNameAvailableInRepo(SnapshotsService.java:754) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.snapshots.SnapshotsService.access$5200(SnapshotsService.java:123) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.snapshots.SnapshotsService$2.execute(SnapshotsService.java:459) ~[elasticsearch-7.14.2.jar:7.14.2]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$1.execute(BlobStoreRepository.java:480) ~[elasticsearch-7.14.2.jar:7.14.2]
```
server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java
String repository,
        String name,
        SnapshotsInProgress snapshotsInProgress,
        SnapshotDeletionsInProgress deletionsInProgress
    ) {
        final int inProgressOperations = snapshotsInProgress.count() + deletionsInProgress.getEntries().size();
        final int maxOps = maxConcurrentOperations;
        if (inProgressOperations >= maxOps) {
            throw new ConcurrentSnapshotExecutionException(
                repository,
                name,
                "Cannot start another operation, already running ["
                    + inProgressOperations
                    + "] operations and the current"
                    + " limit for concurrent snapshot operations is set to ["
                    + maxOps
                    + "]"
            );
        }
    }
    /**
     * Throws {@link RepositoryMissingException} if no repository by the given name is found in the given cluster state.
     */
    public static void ensureRepositoryExists(String repoName, ClusterState state) {
        if (state.metadata().custom(RepositoriesMetadata.TYPE, RepositoriesMetadata.EMPTY).repository(repoName) == null) {
            throw new RepositoryMissingException(repoName);
        }
    }
    /**
     * Validates snapshot request