When `cluster.routing.allocation.awareness.attributes` is set the ML trained model assignment planner assumes that the attributes listed are used to indicate the AZ of every node in the cluster, not just data nodes. For Cloud this is the correct assumption, but for self-managed clusters it's not necessarily true.

When an ML node doesn't have the expected attributes we end up in https://github.com/elastic/elasticsearch/blob/ed1b2c3fb612b684a6ee057e43c2a824d10c31d4/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/autoscaling/NodeAvailabilityZoneMapper.java#L144-L150

One option would be to make that situation an explicit error that prevents allocation but shows up as the reason why allocation wasn't possible. If we had done that for the first release it would have been the best option, however, since the functionality is already released and there's a chance that there are some users who have the expected attributes on some ML nodes but not others we can't really do it, because when such users upgraded to the version with the extra check their models would cease to be assigned.

So probably the best fix for making allocation more user-friendly is to take the value of any awareness attribute that is not set to be an empty string.
x-pack/plugin/ml/qa/native-multi-node-tests/src/javaRestTest/java/org/elasticsearch/xpack/ml/integration/PyTorchModelIT.java
public void testEvaluateWithResultFieldOverride() throws IOException {
        String modelId = "test_evaluate";
        createPassThroughModel(modelId);
        putModelDefinition(modelId);
        putVocabulary(List.of("these", "are", "my", "words"), modelId);
        startDeployment(modelId);
        String resultsField = randomAlphaOfLength(10);
        Response inference = infer("my words", modelId, resultsField);
        assertThat(
            EntityUtils.toString(inference.getEntity()),
            equalTo("{\"inference_results\":[{\"" + resultsField + "\":[[1.0,1.0]]}]}")
        );
        stopDeployment(modelId);
    }
    public void testEvaluateWithMinimalTimeout() throws IOException {
        String modelId = "test_evaluate_timeout";
        createPassThroughModel(modelId);
        putModelDefinition(modelId);
        putVocabulary(List.of("these", "are", "my", "words"), modelId);
        startDeployment(modelId);
        
        
        
        
        try {
            infer("my words", modelId, TimeValue.ZERO);
        } catch (ResponseException ex) {
            assertThat(ex.getResponse().getStatusLine().getStatusCode(), equalTo(408));
        }
        stopDeployment(modelId);