When `cluster.routing.allocation.awareness.attributes` is set the ML trained model assignment planner assumes that the attributes listed are used to indicate the AZ of every node in the cluster, not just data nodes. For Cloud this is the correct assumption, but for self-managed clusters it's not necessarily true.

When an ML node doesn't have the expected attributes we end up in https://github.com/elastic/elasticsearch/blob/ed1b2c3fb612b684a6ee057e43c2a824d10c31d4/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/autoscaling/NodeAvailabilityZoneMapper.java#L144-L150

One option would be to make that situation an explicit error that prevents allocation but shows up as the reason why allocation wasn't possible. If we had done that for the first release it would have been the best option, however, since the functionality is already released and there's a chance that there are some users who have the expected attributes on some ML nodes but not others we can't really do it, because when such users upgraded to the version with the extra check their models would cease to be assigned.

So probably the best fix for making allocation more user-friendly is to take the value of any awareness attribute that is not set to be an empty string.
x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/autoscaling/NodeAvailabilityZoneMapper.java
v = new ArrayList<>();
                    }
                    v.add(node);
                    return v;
                });
            }
        }
        return new NodesByAvailabilityZone(Map.copyOf(allNodesByAvailabilityZone), Map.copyOf(mlNodesByAvailabilityZone));
    }
    /**
     * This is different to {@link #getMlNodesByAvailabilityZone()} in that the latter returns the ML nodes by availability zone
     * of the latest cluster state, while this method does the same for a specific cluster state.
     *
     * @param clusterState The cluster state whose nodes will be used to detect ML nodes by availability zone.
     * @return A map whose keys are lists of awareness attribute values in the same order as the configured awareness attribute
     *         names, and whose values are collections of nodes that have that combination of attributes. If availability zones
     *         are not configured then the map will contain one entry mapping an empty list to a collection of all nodes. If
     *         it is too early in the lifecycle of the node to know the answer then an empty map will be returned. An empty
     *         map will also be returned if there are no ML nodes in the cluster. (These two empty map return scenarios can be
     *         distinguished by calling one of the other methods.)
     */
    public Map<List<String>, Collection<DiscoveryNode>> buildMlNodesByAvailabilityZone(ClusterState clusterState) {
        return buildNodesByAvailabilityZone(clusterState.nodes(), awarenessAttributes).mlNodes;
    }
    private record NodesByAvailabilityZone(
        Map<List<String>, Collection<DiscoveryNode>> allNodes,
        Map<List<String>, Collection<DiscoveryNode>> mlNodes
    ) {};
}