Seen in soak testing:

```
org.elasticsearch.snapshots.SnapshotStressTestsIT > testRandomActivities {seed=[AB70F7282B153829:AB3327F808885EA1]} FAILED
    java.lang.AssertionError: add restore for [[index-0][2]] that already has an existing restore
        at __randomizedtesting.SeedInfo.seed([AB70F7282B153829]:0)
        at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:3399)
        at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$14(StoreRecovery.java:569)
        at org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
        at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:310)
        at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:230)
        at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:133)
        at org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:108)
        at org.elasticsearch.index.shard.StoreRecovery.lambda$restore$15(StoreRecovery.java:567)
        at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212)
        at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:182)
        at org.elasticsearch.index.CompositeIndexEventListener.callListeners(CompositeIndexEventListener.java:278)
        at org.elasticsearch.index.CompositeIndexEventListener.iterateBeforeIndexShardRecovery(CompositeIndexEventListener.java:287)
        at org.elasticsearch.index.CompositeIndexEventListener.beforeIndexShardRecovery(CompositeIndexEventListener.java:314)
        at org.elasticsearch.index.shard.IndexShard.preRecovery(IndexShard.java:1683)
        at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:519)
        at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:311)
        at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:2349)
        at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$21(IndexShard.java:3132)
        at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
```

~This looks to happen if we attempt to do two concurrent restores of an index, one where the index is closed and the other where it is deleted. This test shouldn't do that since it is focussing on the happy path, and one of those restores should fail, but nor should it be possible to trip an assertion if we try it.~

Edit: hmm no that doesn't make sense, there's some confusing logging in this area. Looking again I see that we restore the same index twice in quick succession, and there are some recovery cancellations in the logs. Cancelling a snapshot-based recovery won't stop the restore (and clean up the `ongoingRestores` marker) straight away. But I'm not sure how that could happen in `SnapshotStressTestsIT` yet, it looks like we always hold all the permits for an index until the restore succeeds. 

---

I think the production consequences of this are mild, we mostly track the ongoing restores so that tests can wait for the repository to become idle.
server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java
private static final Set<String> DYNAMIC_SETTING_NAMES = Set.of(
        MAX_SNAPSHOT_BYTES_PER_SEC.getKey(),
        MAX_RESTORE_BYTES_PER_SEC.getKey()
    );
    private final boolean readOnly;
    private final Object lock = new Object();
    private final SetOnce<BlobContainer> blobContainer = new SetOnce<>();
    private final SetOnce<BlobStore> blobStore = new SetOnce<>();
    private final BlobPath basePath;
    private final ClusterService clusterService;
    private final RecoverySettings recoverySettings;
    private final NamedXContentRegistry namedXContentRegistry;
    protected final BigArrays bigArrays;
    /**
     * Flag that is set to {@code true} if this instance is started with {@link #metadata} that has a higher value for
     * {@link RepositoryMetadata#pendingGeneration()} than for {@link RepositoryMetadata#generation()} indicating a full cluster restart
     * potentially accounting for the the last {@code index-N} write in the cluster state.
     * Note: While it is true that this value could also be set to {@code true} for an instance on a node that is just joining the cluster
     * during a new {@code index-N} write, this does not present a problem. The node will still load the correct {@link RepositoryData} in
     * all cases and simply do a redundant listing of the repository contents if it tries to load {@link RepositoryData} and falls back
     * to {@link #latestIndexBlobId()} to validate the value of {@link RepositoryMetadata#generation()}.
     */
    private boolean uncleanStart;
    /**
     * This flag indicates that the repository can not exclusively rely on the value stored in {@link #latestKnownRepoGen} to determine the
     * latest repository generation but must inspect its physical contents as well via {@link #latestIndexBlobId()}.
     * This flag is set in the following situations:
     * <ul>
     *     <li>All repositories that are read-only, i.e. for which {@link #isReadOnly()} returns {@code true} because there are no
     *     guarantees that another cluster is not writing to the repository at the same time</li>