It's possible that in certain situations the ML tier in Cloud does not autoscale up as it should.

This happens because persistent task assignment and autoscaling both refuse to run if the memory tracker is "stale", but both use different definitions of staleness.  Persistent task assignment considers the memory tracker stale if it wasn't refreshed in the last 90 seconds (by default), and autoscaling considers the memory tracker stale if it wasn't refreshed in the last 421 seconds (= learnt interval between autoscaling calls, which is usually 301 seconds, plus 2 different 60 second tolerances).

Autoscaling checks here: https://github.com/elastic/elasticsearch/blob/3da04514a570e0eb8103b06d7b6c192c40993ef7/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/autoscaling/MlAutoscalingDeciderService.java#L477

Persistent task assignment checks here: https://github.com/elastic/elasticsearch/blob/1f24ffdacd2415b7b9eb6a188ab47ce084186203/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/task/AbstractJobPersistentTasksExecutor.java#L187

But the nasty problem is that if persistent task assignment finds the memory tracker stale then it changes the tasks assignment explanation to say this.  If autoscaling runs while the task has this assignment then it won't consider the task to be "waiting" here: https://github.com/elastic/elasticsearch/blob/3da04514a570e0eb8103b06d7b6c192c40993ef7/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/autoscaling/MlAutoscalingDeciderService.java#L388

For example, instead of `{"executor_node":null,"explanation":"persistent task is awaiting node assignment."}` we get `{"executor_node":null,"explanation":"Not opening job [my_job] because job memory requirements are stale - refresh requested"}`.  In fact, the assignment explanation oscillates between these two states, which is visible in the allocation ID counting upwards despite no changes in the cluster: `"allocation_id":222`.  For a job that had never been successfully assigned to a node in an unchanging cluster we'd hope that that allocation ID would be 1 or 2, not 222.

If autoscaling runs while the assignment explanation relates to stale memory then it will bypass the scale up code, and send us into the "weird" code block here: https://github.com/elastic/elasticsearch/blob/3da04514a570e0eb8103b06d7b6c192c40993ef7/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/autoscaling/MlAutoscalingDeciderService.java#L591

The safest way to avoid this discrepancy would seem to be to use the same definition of staleness everywhere. 
x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/autoscaling/MlAutoscalingDeciderService.java
import org.elasticsearch.cluster.node.DiscoveryNode;
import org.elasticsearch.cluster.node.DiscoveryNodeRole;
import org.elasticsearch.cluster.service.ClusterService;
import org.elasticsearch.common.settings.Setting;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.unit.ByteSizeValue;
import org.elasticsearch.common.xcontent.XContentElasticsearchExtension;
import org.elasticsearch.core.Nullable;
import org.elasticsearch.core.TimeValue;
import org.elasticsearch.core.Tuple;
import org.elasticsearch.persistent.PersistentTasksCustomMetadata;
import org.elasticsearch.persistent.PersistentTasksCustomMetadata.PersistentTask;
import org.elasticsearch.xpack.autoscaling.capacity.AutoscalingCapacity;
import org.elasticsearch.xpack.autoscaling.capacity.AutoscalingDeciderContext;
import org.elasticsearch.xpack.autoscaling.capacity.AutoscalingDeciderResult;
import org.elasticsearch.xpack.autoscaling.capacity.AutoscalingDeciderService;
import org.elasticsearch.xpack.core.ml.MlTasks;
import org.elasticsearch.xpack.core.ml.action.OpenJobAction;
import org.elasticsearch.xpack.core.ml.action.StartDataFrameAnalyticsAction;
import org.elasticsearch.xpack.core.ml.action.StartDatafeedAction.DatafeedParams;
import org.elasticsearch.xpack.core.ml.dataframe.DataFrameAnalyticsState;
import org.elasticsearch.xpack.core.ml.inference.assignment.AssignmentState;
import org.elasticsearch.xpack.core.ml.inference.assignment.TrainedModelAssignment;
import org.elasticsearch.xpack.core.ml.job.config.AnalysisLimits;
import org.elasticsearch.xpack.core.ml.job.config.JobState;
import org.elasticsearch.xpack.core.ml.job.snapshot.upgrade.SnapshotUpgradeState;
import org.elasticsearch.xpack.core.ml.job.snapshot.upgrade.SnapshotUpgradeTaskParams;
import org.elasticsearch.xpack.ml.MachineLearning;
import org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentMetadata;
import org.elasticsearch.xpack.ml.job.NodeLoad;