Seen in soak testing:

```
org.elasticsearch.snapshots.SnapshotStressTestsIT > testRandomActivities {seed=[AB70F7282B153829:AB3327F808885EA1]} FAILED
    java.lang.AssertionError: add restore for [[index-0][2]] that already has an existing restore
        at __randomizedtesting.SeedInfo.seed([AB70F7282B153829]:0)
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:3399)
        at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$14(StoreRecovery.java:569)
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
        at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:310)
        at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:230)
        at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:133)
        at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:108)
        at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$15(StoreRecovery.java:567)
        at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212)
        at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:182)
        at org.elasticsearch.index.CompositeIndexEventListener.callListeners(CompositeIndexEventListener.java:278)
        at org.elasticsearch.index.CompositeIndexEventListener.iterateBeforeIndexShardRecovery(CompositeIndexEventListener.java:287)
        at org.elasticsearch.index.CompositeIndexEventListener.beforeIndexShardRecovery(CompositeIndexEventListener.java:314)
        at org.elasticsearch.index.shard.IndexShard.preRecovery(IndexShard.java:1683)
        at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:519)
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:311)
        at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:2349)
        at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$21(IndexShard.java:3132)
        at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
```

~This looks to happen if we attempt to do two concurrent restores of an index, one where the index is closed and the other where it is deleted. This test shouldn't do that since it is focussing on the happy path, and one of those restores should fail, but nor should it be possible to trip an assertion if we try it.~

Edit: hmm no that doesn't make sense, there's some confusing logging in this area. Looking again I see that we restore the same index twice in quick succession, and there are some recovery cancellations in the logs. Cancelling a snapshot-based recovery won't stop the restore (and clean up the `ongoingRestores` marker) straight away. But I'm not sure how that could happen in `SnapshotStressTestsIT` yet, it looks like we always hold all the permits for an index until the restore succeeds. 

---

I think the production consequences of this are mild, we mostly track the ongoing restores so that tests can wait for the repository to become idle.
server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java
/**
         * The IDs of the snapshots to delete. This collection is empty if the deletion is a repository cleanup.
         */
        private final Collection<SnapshotId> snapshotIds;
        /**
         * The {@link RepositoryData} generation at the start of the process, to ensure that the {@link RepositoryData} does not change
         * while the new {@link BlobStoreIndexShardSnapshots} are being written.
         */
        private final long originalRepositoryDataGeneration;
        /**
         * The minimum {@link IndexVersion} of the nodes in the cluster and the snapshots remaining in the repository. The repository must
         * remain readable by all node versions which support this {@link IndexVersion}.
         */
        private final IndexVersion repositoryFormatIndexVersion;
        /**
         * Whether the {@link #repositoryFormatIndexVersion} is new enough to support naming {@link BlobStoreIndexShardSnapshots} blobs with
         * UUIDs (i.e. does not need to remain compatible with versions before v7.6.0). Older repositories use (unsafe) numeric indices for
         * these blobs instead.
         */
        private final boolean useShardGenerations;
        /**
         * All blobs in the repository root at the start of the operation, obtained by listing the repository contents. Note that this may
         * include some blobs which are no longer referenced by the current {@link RepositoryData}, but which have not yet been removed by
         * the cleanup that follows an earlier deletion. This cleanup may still be ongoing (we do not wait for it to complete before
         * starting the next repository operation) or it may have failed before completion (it could have been running on a different node,
         * which crashed for unrelated reasons) so we track all the blobs here and clean them up again at the end.
         */
        private final Map<String, BlobMetadata> originalRootBlobs;
        /**
         * All index containers at the start of the operation, obtained by listing the repository contents. Note that this may include some