**Build scan:**
https://gradle-enterprise.elastic.co/s/b5amw7zci2plo/tests/:x-pack:qa:full-cluster-restart:v8.2.0%23upgradedClusterTest/org.elasticsearch.xpack.restart.MLModelDeploymentFullClusterRestartIT/testDeploymentSurvivesRestart

**Reproduction line:**
`./gradlew ':x-pack:qa:full-cluster-restart:v8.2.0#upgradedClusterTest' -Dtests.class="org.elasticsearch.xpack.restart.MLModelDeploymentFullClusterRestartIT" -Dtests.method="testDeploymentSurvivesRestart" -Dtests.seed=4810270929CED248 -Dtests.bwc=true -Dtests.locale=th -Dtests.timezone=Antarctica/Rothera -Druntime.java=17`

**Applicable branches:**
master

**Reproduces locally?:**
Didn't try

**Failure history:**
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.restart.MLModelDeploymentFullClusterRestartIT&tests.test=testDeploymentSurvivesRestart

**Failure excerpt:**
```
java.lang.AssertionError: {pipeline_count=0, model_size_stats={model_size_bytes=1630, required_native_memory_bytes=283118780}, model_id=trained-model-full-cluster-restart, deployment_stats={start_time=1648545410145, inference_threads=1, nodes=[{routing_state={routing_state=failed, reason=No known trained model with model_id [trained-model-full-cluster-restart]}, node={Nh039VelR4SuhoOzgsKnyg={transport_address=127.0.0.1:34973, roles=[data, data_cold, data_content, data_frozen, data_hot, data_warm, ingest, master, ml, remote_cluster_client, transform], name=v8.2.0-0, attributes={upgraded=true, ml.machine_memory=101368078336, xpack.installed=true, testattr=test, ml.max_jvm_size=536870912}, ephemeral_id=W63zR7WyQjyZxKLItf5eYA}}, throughput_last_minute=0, peak_throughput_per_minute=0}, {routing_state={routing_state=started}, node={zmZMAZ7eRmachPiYsdrY5A={transport_address=127.0.0.1:35267, roles=[data, data_cold, data_content, data_frozen, data_hot, data_warm, ingest, master, ml, remote_cluster_client, transform], name=v8.2.0-1, attributes={upgraded=true, ml.machine_memory=101368078336, xpack.installed=true, testattr=test, ml.max_jvm_size=536870912}, ephemeral_id=ZQGVMmscR7yhu_MnHEzqPA}}, start_time=1648545449551, inference_threads=1, inference_count=0, throughput_last_minute=0, model_threads=1, number_of_pending_requests=0, peak_throughput_per_minute=0}], allocation_status={target_allocation_count=2, state=started, allocation_count=1}, model_id=trained-model-full-cluster-restart, state=started, model_threads=1, queue_capacity=1024, peak_throughput_per_minute=0}}
Expected: "fully_allocated"
     but: was "started"

  at __randomizedtesting.SeedInfo.seed([4810270929CED248:A14B2E695F603468]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.junit.Assert.assertThat(Assert.java:956)
  at org.elasticsearch.xpack.restart.MLModelDeploymentFullClusterRestartIT.lambda$waitForDeploymentStarted$0(MLModelDeploymentFullClusterRestartIT.java:108)
  at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1066)
  at org.elasticsearch.xpack.restart.MLModelDeploymentFullClusterRestartIT.waitForDeploymentStarted(MLModelDeploymentFullClusterRestartIT.java:102)
  at org.elasticsearch.xpack.restart.MLModelDeploymentFullClusterRestartIT.testDeploymentSurvivesRestart(MLModelDeploymentFullClusterRestartIT.java:94)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:833)

```
x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/action/TransportCreateTrainedModelAllocationAction.java
private final TrainedModelAllocationClusterService trainedModelAllocationClusterService;
    @Inject
    public TransportCreateTrainedModelAllocationAction(
        TrainedModelAllocationClusterService trainedModelAllocationClusterService,
        TrainedModelAllocationService trainedModelAllocationService,
        DeploymentManager deploymentManager,
        TransportService transportService,
        ClusterService clusterService,
        ThreadPool threadPool,
        ActionFilters actionFilters,
        IndexNameExpressionResolver indexNameExpressionResolver,
        XPackLicenseState licenseState
    ) {
        super(
            CreateTrainedModelAllocationAction.NAME,
            false,
            transportService,
            clusterService,
            threadPool,
            actionFilters,
            Request::new,