### CI Link

https://gradle-enterprise.elastic.co/s/7dleq35dp5stm

### Repro line

./gradlew ':x-pack:plugin:core:test' --tests "org.elasticsearch.xpack.core.ilm.SetSingleNodeAllocateStepTests.testPerformActionAttrsRequestFails" -Dtests.seed=6B438F3C3DADE5D2 -Dtests.locale=nl -Dtests.timezone=Asia/Chongqin

### Does it reproduce?

Yes

### Applicable branches

8.1, 8.0

### Failure history

Seems to fail rarely, but has failed before. Example: https://gradle-enterprise.elastic.co/s/d33b4fzn7yoh6

### Failure excerpt

```
org.elasticsearch.xpack.core.ilm.SetSingleNodeAllocateStepTests > testPerformActionAttrsRequestFails FAILED
    java.lang.AssertionError: Node roles must not be provided as attributes but saw attributes {ml=K}
        at __randomizedtesting.SeedInfo.seed([C679BACC1DA27FA7:CB16887322D9C088]:0)
        at org.elasticsearch.cluster.node.DiscoveryNode.<init>(DiscoveryNode.java:233)
        at org.elasticsearch.cluster.node.DiscoveryNode.<init>(DiscoveryNode.java:175)
```
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/ilm/SetSingleNodeAllocateStepTests.java
}
        Set<String> nodeIds = new HashSet<>();
        nodeIds.addAll(newNodeIds);
        nodeIds.addAll(oldNodeIds);
        DiscoveryNodes discoveryNodes = nodes.build();
        IndexRoutingTable.Builder indexRoutingTable = createRoutingTableWithOneShardOnSubset(indexMetadata, newNodeIds, nodeIds);
        
        assertNodeSelected(indexMetadata, indexMetadata.getIndex(), newNodeIds, discoveryNodes, indexRoutingTable.build());
    }
    public void testPerformActionSomeShardsOnlyOnNewNodesButNewNodesInvalidAttrs() {
        final Version oldVersion = VersionUtils.randomPreviousCompatibleVersion(random(), Version.CURRENT);
        final int numNodes = randomIntBetween(2, 20); 
        final int numNewNodes = randomIntBetween(1, numNodes - 1);
        final int numOldNodes = numNodes - numNewNodes;
        final int numberOfShards = randomIntBetween(1, 5);
        final String attribute = "box_type";
        final String validAttr = "valid";
        final String invalidAttr = "not_valid";
        Settings.Builder indexSettings = settings(oldVersion);
        indexSettings.put(IndexMetadata.INDEX_ROUTING_INCLUDE_GROUP_SETTING.getKey() + attribute, validAttr);
        IndexMetadata indexMetadata = IndexMetadata.builder(randomAlphaOfLength(10))
            .settings(indexSettings)
            .numberOfShards(numberOfShards)
            .numberOfReplicas(randomIntBetween(0, numNewNodes - 1))
            .build();
        DiscoveryNodes.Builder nodes = DiscoveryNodes.builder();
        Set<String> newNodeIds = new HashSet<>();
        for (int i = 0; i < numNewNodes; i++) {
            String nodeId = "new_node_id_" + i;
            String nodeName = "new_node_" + i;