### Elasticsearch Version

8.11

### Installed Plugins

_No response_

### Java Version

_bundled_

### OS Version

mac

### Problem Description

with apm tracing enabled elasticsearch is throwing assertion error when running a `gradlew run` cluster

```» [2023-09-29T16:43:32,698][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [runTask-0] fatal error in thread [elasticsearch[runTask-0][search][T#2]], exiting java.lang.AssertionError
»       at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:588)
»       at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:557)
»       at co.elastic.apm.agent.opentelemetry.tracing.OTelSpan.end(OTelSpan.java:122)
»       at org.elasticsearch.telemetry.apm@8.11.0-SNAPSHOT/org.elasticsearch.telemetry.apm.internal.tracing.APMTracer.stopTrace(APMTracer.java:385)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.telemetry.tracing.Tracer.stopTrace(Tracer.java:78)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager.unregister(TaskManager.java:336)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.transport.RequestHandlerRegistry.lambda$processMessageReceived$0(RequestHandlerRegistry.java:67)
»       at org.elasticsearch.base@8.11.0-SNAPSHOT/org.elasticsearch.core.Releasables$3.close(Releasables.java:142)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:44)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.lambda$onResponse$0(ChannelActionListener.java:31)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:31)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:19)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:51)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:48)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
»       at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
»       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
»       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
»       at java.base/java.lang.Thread.run(Thread.java:1583)
```


APM logs
```
2023-09-29 16:43:32,684 [elasticsearch[runTask-0][search][T#2]] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to 'indices:data/read/search[phase/query]' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (84003e0) (4)
2023-09-29 16:43:32,684 [elasticsearch[runTask-0][search][T#2]] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - endSpan 'indices:data/read/search[phase/query]' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (84003e0)
2023-09-29 16:43:32,684 [elasticsearch[runTask-0][search][T#2]] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'GET /{index}/_search' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (139632d) (11)
2023-09-29 16:43:32,684 [elasticsearch[runTask-0][search][T#2]] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'indices:data/read/search[phase/query]' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (84003e0) (3)
2023-09-29 16:43:32,684 [elasticsearch[runTask-0][search][T#2]] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'GET /{index}/_search' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (139632d) (10)
2023-09-29 16:43:32,684 [elasticsearch[runTask-0][search][T#2]] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'indices:data/read/search' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (bca2bb6) (2)
2023-09-29 16:43:32,684 [elasticsearch[runTask-0][search][T#2]] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'GET /{index}/_search' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (139632d) (9)
2023-09-29 16:43:32,685 [elasticsearch[runTask-0][search][T#2]] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'indices:data/read/search[phase/query]' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (84003e0) (2)
2023-09-29 16:43:32,693 [elasticsearch[runTask-0][search][T#2]] DEBUG co.elastic.apm.agent.impl.ActiveStack - Deactivating OTelBridgeContext[{opentelemetry-trace-span-key=OtelSpan['indices:data/read/search[phase/query]' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (84003e0)]}] on thread 112
2023-09-29 16:43:32,696 [elasticsearch[runTask-0][search][T#3]] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to 'GET /{index}/_search' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (139632d) (10)
2023-09-29 16:43:32,696 [elasticsearch[runTask-0][search][T#3]] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to 'indices:data/read/search' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (bca2bb6) (3)
2023-09-29 16:43:32,696 [elasticsearch[runTask-0][search][T#3]] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - endSpan 'indices:data/read/search' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (bca2bb6)
2023-09-29 16:43:32,696 [elasticsearch[runTask-0][search][T#3]] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'GET /{index}/_search' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (139632d) (9)
2023-09-29 16:43:32,696 [elasticsearch[runTask-0][search][T#3]] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'indices:data/read/search' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (bca2bb6) (2)
2023-09-29 16:43:32,696 [elasticsearch[runTask-0][search][T#3]] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'GET /{index}/_search' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (139632d) (8)
2023-09-29 16:43:32,696 [elasticsearch[runTask-0][search][T#3]] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'GET /{index}/_search' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (139632d) (7)
2023-09-29 16:43:32,696 [elasticsearch[runTask-0][search][T#3]] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'indices:data/read/search' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (bca2bb6) (1)
2023-09-29 16:43:32,695 [elasticsearch[runTask-0][search][T#2]] WARN  co.elastic.apm.agent.impl.transaction.AbstractSpan - End has already been called: 'indices:data/read/search[phase/query]' 00-d5cbef085caa775a48c22d8f6b572d75-de0e50d04211fefd-00 (84003e0)
```


when enabeling trace APM logs we get to see the 2 different stacktracing calling endSpan for the same span id
```
2023-09-29 16:54:31,657 [elasticsearch[runTask-0][search][T#1]] WARN  co.elastic.apm.agent.impl.transaction.AbstractSpan - End has already been called: 'indices:data/read/search[phase/query]' 00-c0c729ba17b1f5c9071e27913d3d3e04-e0d2887e8d30da7e-01 (19fd5734)
java.lang.Throwable: null
        at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:583) ~[elastic-apm-agent-1.43.0.jar:1.43.0]
        at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:557) ~[elastic-apm-agent-1.43.0.jar:1.43.0]
        at co.elastic.apm.agent.opentelemetry.tracing.OTelSpan.end(OTelSpan.java:122) ~[elastic-apm-agent-1.43.0.jar:1.43.0]
        at org.elasticsearch.telemetry.apm.internal.tracing.APMTracer.stopTrace(APMTracer.java:385) ~[?:?]
        at org.elasticsearch.telemetry.tracing.Tracer.stopTrace(Tracer.java:78) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.tasks.TaskManager.unregister(TaskManager.java:336) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.lambda$processMessageReceived$0(RequestHandlerRegistry.java:67) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.core.Releasables$3.close(Releasables.java:142) ~[elasticsearch-core-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:44) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.action.support.ChannelActionListener.lambda$onResponse$0(ChannelActionListener.java:31) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.action.ActionListener.run(ActionListener.java:368) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:31) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:19) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:51) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:48) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) [?:?]

```

```
2023-09-29 16:54:31,645 [elasticsearch[runTask-0][search][T#1]] TRACE co.elastic.apm.agent.impl.ElasticApmTracer - ending span at
java.lang.RuntimeException: this exception is just used to record where the span has been ended from
at co.elastic.apm.agent.impl.ElasticApmTracer.endSpan(ElasticApmTracer.java:528) ~[elastic-apm-agent-1.43.0.jar:1.43.0]
at co.elastic.apm.agent.impl.transaction.Span.afterEnd(Span.java:288) ~[elastic-apm-agent-1.43.0.jar:1.43.0]
at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:580) ~[elastic-apm-agent-1.43.0.jar:1.43.0]
at co.elastic.apm.agent.impl.transaction.AbstractSpan.end(AbstractSpan.java:557) ~[elastic-apm-agent-1.43.0.jar:1.43.0]
at co.elastic.apm.agent.opentelemetry.tracing.OTelSpan.end(OTelSpan.java:122) ~[elastic-apm-agent-1.43.0.jar:1.43.0]
at org.elasticsearch.telemetry.apm.internal.tracing.APMTracer.stopTrace(APMTracer.java:394) ~[?:?]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:675) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:542) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:51) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:48) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:73) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.11.0-SNAPSHOT.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) [?:?]

```




### Steps to Reproduce

start ES with tracing enabled

```
cat ../apm_enable_statefull.gradle 
rootProject {
  if (project.name == 'elasticsearch') {
    afterEvaluate {
      testClusters.matching { it.name == "runTask" }.configureEach {
        setting 'xpack.security.audit.enabled', 'true'
        keystore 'tracing.apm.secret_token', 'YOURPASSWORD'
        setting 'telemetry.metrics.enabled', 'true'
        setting 'tracing.apm.enabled', 'true'

      setting 'tracing.apm.agent.server_url', 'YOUR server' 
      }
    }
  }
}

```


start ES
` ./gradlew run -I ../apm_enable_statefull.gradle `

post a doc

curl --request POST \
  --url http://localhost:9200/xxx2232/_doc \
  --header 'Accept: application/vnd.elasticsearch+json;compatible-with=8' \
  --header 'Authorization: Basic ZWxhc3RpYy1hZG1pbjplbGFzdGljLXBhc3N3b3Jk' \
  --header 'Content-Type: application/vnd.elasticsearch+json;compatible-with=8' \
  --data '{
	"x":123
}'



call a search api
curl --request GET \
  --url http://localhost:9200/xxx2232/_search \
  --header 'Accept: application/vnd.elasticsearch+json;compatible-with=8' \
  --header 'Authorization: Basic ZWxhc3RpYzpwYXNzd29yZA==' \
  --header 'Content-Type: application/vnd.elasticsearch+json;compatible-with=8'

### Logs (if relevant)

_No response_
server/src/main/java/org/elasticsearch/search/SearchService.java
Settings settings = clusterService.getSettings();
        this.threadPool = threadPool;
        this.clusterService = clusterService;
        this.indicesService = indicesService;
        this.scriptService = scriptService;
        this.responseCollectorService = responseCollectorService;
        this.bigArrays = bigArrays;
        this.fetchPhase = fetchPhase;
        this.multiBucketConsumerService = new MultiBucketConsumerService(
            clusterService,
            settings,
            circuitBreakerService.getBreaker(CircuitBreaker.REQUEST)
        );
        this.executorSelector = executorSelector;
        this.tracer = tracer;
        TimeValue keepAliveInterval = KEEPALIVE_INTERVAL_SETTING.get(settings);
        setKeepAlives(DEFAULT_KEEPALIVE_SETTING.get(settings), MAX_KEEPALIVE_SETTING.get(settings));
        clusterService.getClusterSettings()
            .addSettingsUpdateConsumer(
                DEFAULT_KEEPALIVE_SETTING,
                MAX_KEEPALIVE_SETTING,
                this::setKeepAlives,
                SearchService::validateKeepAlives
            );
        this.keepAliveReaper = threadPool.scheduleWithFixedDelay(new Reaper(), keepAliveInterval, threadPool.executor(Names.SEARCH));
        defaultSearchTimeout = DEFAULT_SEARCH_TIMEOUT_SETTING.get(settings);
        clusterService.getClusterSettings().addSettingsUpdateConsumer(DEFAULT_SEARCH_TIMEOUT_SETTING, this::setDefaultSearchTimeout);
        minimumDocsPerSlice = MINIMUM_DOCS_PER_SLICE.get(settings);
        defaultAllowPartialSearchResults = DEFAULT_ALLOW_PARTIAL_SEARCH_RESULTS.get(settings);
        clusterService.getClusterSettings()