`org.elasticsearch.xpack.security.SecurityUsageTransportAction#masterOperation` can leak memory on the REST layer by trying to send a failure response twice.

This shows as logging:

```
[2023-10-30T17:40:28,267][ERROR][org.elasticsearch.rest.action.RestResponseListener] [instance-0000000007] failed to send failure response
java.lang.IllegalStateException: Channel is already closed
	at org.elasticsearch.rest.RestController$ResourceHandlingHttpChannel.close(RestController.java:772) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.rest.RestController$ResourceHandlingHttpChannel.sendResponse(RestController.java:759) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:55) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.rest.action.RestCancellableNodeClient$1.onFailure(RestCancellableNodeClient.java:96) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:170) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:38) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:92) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:231) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:218) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:92) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:170) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:38) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:92) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:231) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:218) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:170) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:92) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.xpack.security.profile.ProfileService.tryFreezeAndCheckIndex(ProfileService.java:1015) ~[?:?]
	at org.elasticsearch.xpack.security.profile.ProfileService.usageStats(ProfileService.java:333) ~[?:?]
	at org.elasticsearch.xpack.security.SecurityUsageTransportAction.masterOperation(SecurityUsageTransportAction.java:180) ~[?:?]
	at org.elasticsearch.xpack.security.SecurityUsageTransportAction.masterOperation(SecurityUsageTransportAction.java:50) ~[?:?]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.executeMasterOperation(TransportMasterNodeAction.java:124) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:235) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.6.1.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
	Suppressed: org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-profile-8] is unavailable
		at org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:138) ~[?:?]
		at org.elasticsearch.xpack.security.profile.ProfileService.tryFreezeAndCheckIndex(ProfileService.java:1015) ~[?:?]
		at org.elasticsearch.xpack.security.profile.ProfileService.usageStats(ProfileService.java:333) ~[?:?]
		at org.elasticsearch.xpack.security.SecurityUsageTransportAction.masterOperation(SecurityUsageTransportAction.java:180) ~[?:?]
		at org.elasticsearch.xpack.security.SecurityUsageTransportAction.masterOperation(SecurityUsageTransportAction.java:50) ~[?:?]
		at org.elasticsearch.action.support.master.TransportMasterNodeAction.executeMasterOperation(TransportMasterNodeAction.java:124) ~[elasticsearch-8.6.1.jar:?]
		at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:235) ~[elasticsearch-8.6.1.jar:?]
		at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72) ~[elasticsearch-8.6.1.jar:?]
		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917) ~[elasticsearch-8.6.1.jar:?]
		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.6.1.jar:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
		at java.lang.Thread.run(Thread.java:1589) ~[?:?]
```

which leaks on the transport layer as such:

```
[instance-0000000007] An exception was thrown by org.elasticsearch.transport.netty4.Netty4TcpChannel$$Lambda$6602/0x00000008023d2000.operationComplete()
java.util.ConcurrentModificationException: null
	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1013) ~[?:?]
	at java.util.ArrayList$Itr.next(ArrayList.java:967) ~[?:?]
	at org.elasticsearch.core.IOUtils.close(IOUtils.java:117) ~[elasticsearch-core-8.6.1.jar:?]
	at org.elasticsearch.core.Releasables.close(Releasables.java:23) ~[elasticsearch-core-8.6.1.jar:?]
	at org.elasticsearch.common.io.stream.RecyclerBytesStreamOutput.close(RecyclerBytesStreamOutput.java:172) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.rest.AbstractRestChannel.releaseOutputBuffer(AbstractRestChannel.java:193) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.core.IOUtils.close(IOUtils.java:71) ~[elasticsearch-core-8.6.1.jar:?]
	at org.elasticsearch.core.IOUtils.close(IOUtils.java:119) ~[elasticsearch-core-8.6.1.jar:?]
	at org.elasticsearch.core.Releasables.close(Releasables.java:23) ~[elasticsearch-core-8.6.1.jar:?]
	at org.elasticsearch.http.DefaultRestChannel.lambda$sendResponse$3(DefaultRestChannel.java:154) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:267) ~[elasticsearch-8.6.1.jar:?]
	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addPromise$1(Netty4TcpChannel.java:79) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[?:?]
	at io.netty.util.concurrent.PromiseCombiner.tryPromise(PromiseCombiner.java:170) ~[?:?]
	at io.netty.util.concurrent.PromiseCombiner.access$600(PromiseCombiner.java:35) ~[?:?]
	at io.netty.util.concurrent.PromiseCombiner$1.operationComplete0(PromiseCombiner.java:62) ~[?:?]
	at io.netty.util.concurrent.PromiseCombiner$1.operationComplete(PromiseCombiner.java:44) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[?:?]
```


The bug is relatively obvious:

We create a couple of listeners like so:

```
        final ActionListener<Map<String, Object>> roleMappingStoreUsageListener = ActionListener.wrap(nativeRoleMappingStoreUsage -> {
            Map<String, Object> usage = singletonMap("native", nativeRoleMappingStoreUsage);
            roleMappingUsageRef.set(usage);
            doCountDown.run();
        }, listener::onFailure);

        final ActionListener<Map<String, Object>> realmsUsageListener = ActionListener.wrap(realmsUsage -> {
            realmsUsageRef.set(realmsUsage);
            doCountDown.run();
        }, listener::onFailure);
```

all of these listeners have to come together to form the final response and wait for each other in the success case. In the failure case though, they all individually and immediately resolve the main transport listener (`listener` in this example). So if two of the sub-requests of this action fail at the same time, we try to double send a failure response which leads to #102101 and a memory leak.
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/SecurityUsageTransportAction.java
/*
 * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
 * or more contributor license agreements. Licensed under the Elastic License
 * 2.0; you may not use this file except in compliance with the Elastic License
 * 2.0.
 */
package org.elasticsearch.xpack.security;
import org.elasticsearch.action.ActionListener;
import org.elasticsearch.action.support.ActionFilters;
import org.elasticsearch.cluster.ClusterState;
import org.elasticsearch.cluster.metadata.IndexNameExpressionResolver;
import org.elasticsearch.cluster.service.ClusterService;
import org.elasticsearch.common.inject.Inject;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.util.Maps;
import org.elasticsearch.common.util.concurrent.CountDown;
import org.elasticsearch.core.Nullable;
import org.elasticsearch.license.XPackLicenseState;
import org.elasticsearch.protocol.xpack.XPackUsageRequest;
import org.elasticsearch.tasks.Task;
import org.elasticsearch.threadpool.ThreadPool;