**Build scan:**
https://gradle-enterprise.elastic.co/s/xvwrlxusnxdgu/tests/:qa:rolling-upgrade:v8.6.1%23oneThirdUpgradedTest/org.elasticsearch.upgrades.SnapshotBasedRecoveryIT/testSnapshotBasedRecovery

**Reproduction line:**
```
./gradlew ':qa:rolling-upgrade:v8.6.1#oneThirdUpgradedTest' -Dtests.class="org.elasticsearch.upgrades.SnapshotBasedRecoveryIT" -Dtests.method="testSnapshotBasedRecovery" -Dtests.seed=32151106DB58AD22 -Dtests.bwc=true -Dtests.locale=en-MT -Dtests.timezone=Asia/Colombo -Druntime.java=17
```

**Applicable branches:**
main

**Reproduces locally?:**
Didn't try

**Failure history:**
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.upgrades.SnapshotBasedRecoveryIT&tests.test=testSnapshotBasedRecovery

**Failure excerpt:**
```
java.lang.AssertionError: timed out waiting for green state for index [snapshot_based_recovery] cluster state [{
  "cluster_name" : "v8.6.1",
  "cluster_uuid" : "ETWjuovuQXC6pr6CN8pJXA",
  "version" : 391,
  "state_uuid" : "y7ss0aZLQvab9JIDk7jppg",
  "master_node" : "tsyOGVI1T5iBSTLS0zL29A",
  ...
```
server/src/test/java/org/elasticsearch/cluster/routing/allocation/AllocationCommandsTests.java
ClusterInfo.EMPTY,
            SnapshotShardSizeInfo.EMPTY,
            System.nanoTime()
        );
        logger.info("--> executing move allocation command to non-data node");
        IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () -> command.execute(routingAllocation, false));
        assertEquals(
            "[move_allocation] can't move [test][0] from "
                + node1
                + " to "
                + node2
                + ": source ["
                + node2.getName()
                + "] is not a data node.",
            e.getMessage()
        );
    }
    public void testMoveShardFromNonDataNode() {
        AllocationService allocation = createAllocationService(
            Settings.builder().put("cluster.routing.allocation.node_concurrent_recoveries", 10).build()
        );
        logger.info("creating an index with 1 shard, no replica");
        Metadata metadata = Metadata.builder()
            .put(IndexMetadata.builder("test").settings(settings(Version.CURRENT)).numberOfShards(1).numberOfReplicas(0))
            .build();
        RoutingTable routingTable = RoutingTable.builder(TestShardRoutingRoleStrategies.DEFAULT_ROLE_ONLY)
            .addAsNew(metadata.index("test"))
            .build();
        ClusterState clusterState = ClusterState.builder(ClusterName.DEFAULT).metadata(metadata).routingTable(routingTable).build();
        logger.info("--> adding two nodes");