**Build scan:**
https://gradle-enterprise.elastic.co/s/xvwrlxusnxdgu/tests/:qa:rolling-upgrade:v8.6.1%23oneThirdUpgradedTest/org.elasticsearch.upgrades.SnapshotBasedRecoveryIT/testSnapshotBasedRecovery

**Reproduction line:**
```
./gradlew ':qa:rolling-upgrade:v8.6.1#oneThirdUpgradedTest' -Dtests.class="org.elasticsearch.upgrades.SnapshotBasedRecoveryIT" -Dtests.method="testSnapshotBasedRecovery" -Dtests.seed=32151106DB58AD22 -Dtests.bwc=true -Dtests.locale=en-MT -Dtests.timezone=Asia/Colombo -Druntime.java=17
```

**Applicable branches:**
main

**Reproduces locally?:**
Didn't try

**Failure history:**
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.upgrades.SnapshotBasedRecoveryIT&tests.test=testSnapshotBasedRecovery

**Failure excerpt:**
```
java.lang.AssertionError: timed out waiting for green state for index [snapshot_based_recovery] cluster state [{
  "cluster_name" : "v8.6.1",
  "cluster_uuid" : "ETWjuovuQXC6pr6CN8pJXA",
  "version" : 391,
  "state_uuid" : "y7ss0aZLQvab9JIDk7jppg",
  "master_node" : "tsyOGVI1T5iBSTLS0zL29A",
  ...
```
qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/SnapshotBasedRecoveryIT.java
if (FIRST_MIXED_ROUND) {
                    List<String> upgradedNodeIds = getUpgradedNodeIds();
                    
                    
                    if (upgradedNodeIds.isEmpty() == false) {
                        assertThat(upgradedNodeIds.size(), is(equalTo(1)));
                        String upgradedNodeId = upgradedNodeIds.get(0);
                        logger.info("--> excluding [{}] from node [{}]", indexName, upgradedNodeId);
                        updateIndexSettings(indexName, Settings.builder().put("index.routing.allocation.exclude._id", upgradedNodeId));
                        ensureGreen(indexName);
                        logger.info("--> finished excluding [{}] from node [{}]", indexName, upgradedNodeId);
                    } else {
                        logger.info("--> no upgrading nodes, not adding any exclusions for [{}]", indexName);
                    }
                    String primaryNodeId = getPrimaryNodeIdOfShard(indexName, 0);
                    Version primaryNodeVersion = getNodeVersion(primaryNodeId);
                    
                    
                    
                    
                    
                    logger.info("--> Primary node in first mixed round {} / {}", primaryNodeId, primaryNodeVersion);
                    if (primaryNodeVersion.after(UPGRADE_FROM_VERSION)) {
                        logger.info("--> cancelling primary shard on node [{}]", primaryNodeId);
                        cancelShard(indexName, 0, primaryNodeId);
                        logger.info("--> done cancelling primary shard on node [{}]", primaryNodeId);
                        String currentPrimaryNodeId = getPrimaryNodeIdOfShard(indexName, 0);
                        assertThat(getNodeVersion(currentPrimaryNodeId), is(equalTo(UPGRADE_FROM_VERSION)));
                    }
                } else {