### Elasticsearch Version

8.10 SNAPSHOT

### Installed Plugins

_No response_

### Java Version

_bundled_

### OS Version

Darwin Kernel Version 22.6.0: root:xnu-8796.141.3~6/RELEASE_ARM64_T6000 arm64

### Problem Description

When APM is enabled, importing a model (e.g. via /_ml/trained_models/.elser_model_1) fails to complete. The model is stuck in a "downloading" state (per the UI), and the task runs indefinitely.

### Steps to Reproduce

 The error is surfaced to the user on the first request as follows:

```
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "value for key [apm.local.context] already present"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "value for key [apm.local.context] already present"
  },
  "status": 400
}
```

Following invocations to the API return the model as if it was installed, but it never completes the import.

### Logs (if relevant)

<details>
	<summary>Logs</summary>

```
│ info [o.e.c.m.MetadataCreateIndexService] [Darios-MacBook-Pro-2.local] [.ml-inference-000005] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
   │ info [o.e.c.r.a.AllocationService] [Darios-MacBook-Pro-2.local] updating number_of_replicas to [0] for indices [.ml-inference-000005]
   │ info [o.e.i.m.MapperService] [Darios-MacBook-Pro-2.local] [.ml-inference-000005] reloading search analyzers
   │ info [o.e.c.m.MetadataCreateIndexService] [Darios-MacBook-Pro-2.local] [.ml-annotations-000001] creating index, cause [api], templates [], shards [1]/[1]
   │ info [o.e.c.r.a.AllocationService] [Darios-MacBook-Pro-2.local] updating number_of_replicas to [0] for indices [.ml-annotations-000001]
   │ info [o.e.i.m.MapperService] [Darios-MacBook-Pro-2.local] [.ml-annotations-000001] reloading search analyzers
   │ info [o.e.r.AbstractRestChannel] [Darios-MacBook-Pro-2.local] channel handling [/_ml/trained_models/.elser_model_1] reused
   │ info [r.suppressed] [Darios-MacBook-Pro-2.local] path: /_ml/trained_models/.elser_model_1, params: {model_id=.elser_model_1} java.lang.IllegalStateException: Channel is already closed
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.rest.RestController$ResourceHandlingHttpChannel.close(RestController.java:775)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.rest.RestController$ResourceHandlingHttpChannel.sendResponse(RestController.java:762)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:26)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:38)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onResponse(NodeClient.java:161)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:205)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:199)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:182)
   │            at org.elasticsearch.ml@8.10.0-SNAPSHOT/org.elasticsearch.xpack.ml.action.TransportPutTrainedModelAction.lambda$masterOperation$0(TransportPutTrainedModelAction.java:248)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177)
   │            at org.elasticsearch.ml@8.10.0-SNAPSHOT/org.elasticsearch.xpack.ml.action.TransportPutTrainedModelAction.triggerModelFetchIfNecessary(TransportPutTrainedModelAction.java:401)
   │            at org.elasticsearch.ml@8.10.0-SNAPSHOT/org.elasticsearch.xpack.ml.action.TransportPutTrainedModelAction.lambda$masterOperation$1(TransportPutTrainedModelAction.java:243)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177)
   │            at org.elasticsearch.ml@8.10.0-SNAPSHOT/org.elasticsearch.xpack.ml.inference.persistence.TrainedModelProvider.lambda$storeTrainedModelConfig$0(TrainedModelProvider.java:211)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onResponse(NodeClient.java:161)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:205)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:199)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportBulkAction.lambda$unwrappingSingleItemBulkResponse$0(TransportBulkAction.java:184)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:298)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:630)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:611)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:600)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onResponse(NodeClient.java:161)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:205)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:199)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishOnSuccess(TransportReplicationAction.java:1041)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:953)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:944)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1411)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1510)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1481)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:42)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.lambda$onResponse$0(ChannelActionListener.java:31)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:31)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:19)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:298)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$runWithPrimaryShardReference$3(TransportReplicationAction.java:489)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.ReplicationOperation.finish(ReplicationO
   │ info peration.java:474)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.ReplicationOperation.decPendingAndFinishIfNeeded(ReplicationOperation.java:460)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.ReplicationOperation$1.lambda$onResponse$0(ReplicationOperation.java:182)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.ReplicationOperation.updateCheckPoints(ReplicationOperation.java:361)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.ReplicationOperation$1.onResponse(ReplicationOperation.java:178)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.ReplicationOperation$1.onResponse(ReplicationOperation.java:173)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportWriteAction$WritePrimaryResult$1.onSuccess(TransportWriteAction.java:311)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.maybeFinish(TransportWriteAction.java:468)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.lambda$run$0(TransportWriteAction.java:520)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AsyncIOProcessor.notifyList(AsyncIOProcessor.java:111)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:89)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:73)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.index.engine.InternalEngine.asyncEnsureTranslogSynced(InternalEngine.java:667)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.index.shard.IndexShard.syncAfterWrite(IndexShard.java:3665)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.run(TransportWriteAction.java:518)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportWriteAction$WritePrimaryResult.runPostReplicationActions(TransportWriteAction.java:318)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:173)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:305)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:273)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:235)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:286)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:137)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:215)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
   │            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
   │            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
   │            at java.base/java.lang.Thread.run(Thread.java:1623)
   │ info [o.e.r.a.RestResponseListener] [Darios-MacBook-Pro-2.local] failed to send failure response java.lang.IllegalStateException: Channel is already closed
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.rest.RestController$ResourceHandlingHttpChannel.close(RestController.java:775)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.rest.RestController$ResourceHandlingHttpChannel.sendResponse(RestController.java:762)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:55)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:40)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onResponse(NodeClient.java:161)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:205)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:199)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:182)
   │            at org.elasticsearch.ml@8.10.0-SNAPSHOT/org.elasticsearch.xpack.ml.action.TransportPutTrainedModelAction.lambda$masterOperation$0(TransportPutTrainedModelAction.java:248)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177)
   │            at org.elasticsearch.ml@8.10.0-SNAPSHOT/org.elasticsearch.xpack.ml.action.TransportPutTrainedModelAction.triggerModelFetchIfNecessary(TransportPutTrainedModelAction.java:401)
   │            at org.elasticsearch.ml@8.10.0-SNAPSHOT/org.elasticsearch.xpack.ml.action.TransportPutTrainedModelAction.lambda$masterOperation$1(TransportPutTrainedModelAction.java:243)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177)
   │            at org.elasticsearch.ml@8.10.0-SNAPSHOT/org.elasticsearch.xpack.ml.inference.persistence.TrainedModelProvider.lambda$storeTrainedModelConfig$0(TrainedModelProvider.java:211)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onResponse(NodeClient.java:161)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:205)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:199)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportBulkAction.lambda$unwrappingSingleItemBulkResponse$0(TransportBulkAction.java:184)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:236)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:298)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:630)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:611)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:600)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onResponse(NodeClient.java:161)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:205)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:199)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishOnSuccess(TransportReplicationAction.java:1041)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:953)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:944)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1411)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1510)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1481)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:42)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.lambda$onResponse$0(ChannelActionListener.java:31)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:31)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:19)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:298)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$runWithPrimaryShardReference$3(TransportReplicationAction.java:489)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.ReplicationOperation.finish(ReplicationOperation.java:474)
   │            at org.elasticsearch.server@8.1
   │ info 0.0-SNAPSHOT/org.elasticsearch.action.support.replication.ReplicationOperation.decPendingAndFinishIfNeeded(ReplicationOperation.java:460)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.ReplicationOperation$1.lambda$onResponse$0(ReplicationOperation.java:182)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.ReplicationOperation.updateCheckPoints(ReplicationOperation.java:361)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.ReplicationOperation$1.onResponse(ReplicationOperation.java:178)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.ReplicationOperation$1.onResponse(ReplicationOperation.java:173)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportWriteAction$WritePrimaryResult$1.onSuccess(TransportWriteAction.java:311)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.maybeFinish(TransportWriteAction.java:468)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.lambda$run$0(TransportWriteAction.java:520)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AsyncIOProcessor.notifyList(AsyncIOProcessor.java:111)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:89)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:73)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.index.engine.InternalEngine.asyncEnsureTranslogSynced(InternalEngine.java:667)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.index.shard.IndexShard.syncAfterWrite(IndexShard.java:3665)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.run(TransportWriteAction.java:518)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportWriteAction$WritePrimaryResult.runPostReplicationActions(TransportWriteAction.java:318)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:173)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:305)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:273)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:235)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:286)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:137)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:215)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
   │            at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
   │            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
   │            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
   │            at java.base/java.lang.Thread.run(Thread.java:1623)
   │            Suppressed: java.lang.IllegalStateException: Channel is already closed
   │                    at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.rest.RestController$ResourceHandlingHttpChannel.close(RestController.java:775)
   │                    at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.rest.RestController$ResourceHandlingHttpChannel.sendResponse(RestController.java:762)
   │                    at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:26)
   │                    at org.elasticsearch.server@8.10.0-SNAPSHOT/org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:38)
   │                    ... 80 more
```
</details>
x-pack/plugin/ml-package-loader/src/main/java/org/elasticsearch/xpack/ml/packageloader/action/TransportLoadTrainedModelPackage.java
request.getModelId(),
                request.getModelPackageConfig(),
                downloadTask
            );
            threadPool.executor(MachineLearningPackageLoader.UTILITY_THREAD_POOL_NAME)
                .execute(() -> importModel(client, taskManager, request, modelImporter, listener, downloadTask));
        } catch (Exception e) {
            taskManager.unregister(downloadTask);
            throw e;
        }
        if (request.isWaitForCompletion() == false) {
            listener.onResponse(AcknowledgedResponse.TRUE);
        }
    }
    private ParentTaskAssigningClient getParentTaskAssigningClient(Task originTask) {
        var parentTaskId = new TaskId(clusterService.localNode().getId(), originTask.getId());
        return new ParentTaskAssigningClient(client, parentTaskId);
    }
    /**
     * This is package scope so that we can test the logic directly.
     * This should only be called from the masterOperation method and the tests
     *
     * @param auditClient a client which should only be used to send audit notifications. This client cannot be associated with the passed
     *                    in task, that way when the task is cancelled the notification requests can
     *                    still be performed. If it is associated with the task (i.e. via ParentTaskAssigningClient),
     *                    then the requests will throw a TaskCancelledException.
     */
    static void importModel(
        Client auditClient,
        TaskManager taskManager,