### Elasticsearch Version

8.7.0 (snapshot archives/20230103-151926_4d37feea)

### Installed Plugins

_No response_

### Java Version

_bundled_

### OS Version

Darwin 21.6.0 Darwin Kernel Version 21.6.0: Sun Nov  6 23:31:13 PST 2022; root:xnu-8020.240.14~1/RELEASE_ARM64_T6000 arm64

### Problem Description

First seen since `f332fc2e3625af8e7e4acb03f1aea673cf00e00a`, `c40615c3155a90bd54f68254d183c54df2752367` looks OK.

---

In Kibana we use the `updateAliases` API as an atomic way to:

1. remove an alias from the old index
2. add aliases to the new index
3. remove a temporary index

We're observing tests failing where we run 3 Kibana instances in parallel, processing steps more or less at the same time (a few milliseconds apart).

### High level flow

1. We send the following request from each Kibana to update aliases and indices:

```
POST /_aliases?timeout=60s
{
  "actions": [
    {
      "remove": {
        "index": ".kibana_7.13.0_001",
        "alias": ".kibana",
        "must_exist": true
      }
    },
    {
      "add": {
        "index": ".kibana_8.7.0_001",
        "alias": ".kibana"
      }
    },
    {
      "add": {
        "index": ".kibana_8.7.0_001",
        "alias": ".kibana_8.7.0"
      }
    },
    {
      "remove_index": {
        "index": ".kibana_8.7.0_reindex_temp"
      }
    }
  ]
}
```

2. Kibana 1 succeeds, Kibana 2 + 3 error with `[index_not_found_exception]: no such index [.kibana_8.7.0_reindex_temp]`. 

3. In Kibana 2 + 3 we assume that another Kibana has already successfully deleted the temp index. We proceed to `list indices` based on our new aliases (`.kibana`, `.kibana_8.7.0`):

`GET /.kibana%2C.kibana_8.7.0?ignore_unavailable=true`

We then see a response like:

```json
{
  ".kibana_7.13.0_001": {
    "aliases": { ".kibana": {}, ".kibana_7.13.0": {} }
    ...
  }
}
```

This is unexpected because we assume the `.kibana` alias should be pointing to our new index `.kibana_8.7.0_001` NOT `.kibana_7.13.0_001`, additionally the alias `.kibana_8.7.0` should be pointing to `.kibana_8.7.0_001` as well.

4. It looks like there is a small amount of lag, when issuing the same list indices request (`GET /.kibana%2C.kibana_8.7.0?ignore_unavailable=true`) a bit later we see:

```json
{
  ".kibana_8.7.0_001": {
    "aliases": {
      ".kibana": {
        "is_hidden": true
      },
      ".kibana_8.7.0": {
        "is_hidden": true
      }
    ....
}
```

---

#### Description of requests in cURL

Note: this script does not seem to reproduce the issue, see below.

```bash
#! /bin/bash

ORIGIN=http://localhost:9200
U=elastic:changeme
SLEEP=0.1

## Setup of our partially migrated environment ##
# Create kibana0, our "old" index
curl -su$U -XPUT $ORIGIN/kibana0 > /dev/null
# Create temp, our temporary bulk index from our old index
curl -su$U -XPUT $ORIGIN/temp > /dev/null
# Set write block so we can clone
curl -su$U -XPUT $ORIGIN/temp/_block/write > /dev/null
# Clone temp to our final index kibana1
curl -su$U -XPUT $ORIGIN/temp/_clone/kibana1 > /dev/null
# Point our "current" alias to old index kibana0
curl -su$U -XPOST $ORIGIN/_aliases?timeout=60s -H 'content-type: application/json' -d '{"actions":[{"add":{"index":"kibana0","alias":"current"}},{"add":{"index":"kibana0","alias":"previous"}}]}' > /dev/null

update_aliases() {
  echo ""
 curl -su$U -XPOST $ORIGIN/_aliases?timeout=60s -H 'content-type: application/json' -d '{"actions":[{"remove":{"index":"kibana0","alias":"current","must_exist":true}},{"add":{"index":"kibana1","alias":"current"}},{"add":{"index":"kibana1","alias":"next"}},{"remove_index":{"index":"temp"}}]}'
 # List current indices, this is where we are seeing unexpected results 👇🏻 
 curl -su$U $ORIGIN/current,next?ignore_unavailable=true -H 'content-type: application/json'
 echo ""
}

# Simulate 3 Kibana's sending update alias and list index requests at "the same time"
update_aliases &
sleep $SLEEP
update_aliases &
sleep $SLEEP
update_aliases
```

### Steps to Reproduce

Running a test case from the Kibana repo that starts Elastic and multiple Kibana instances:

https://github.com/elastic/kibana/blob/f2e3c5d480892e87b945f7c189a7abe5830c03fc/src/core/server/integration_tests/saved_objects/migrations/multiple_kibana_nodes.test.ts#L204

1. Clone the Kibana repo and checkout `f2e3c5d480892e87b945f7c189a7abe5830c03fc`
2. Apply the diff below to unskip the test
3. Run `yarn kbn bootstrap` and `yarn test:jest_integration ./src/core/server/integration_tests/saved_objects/migrations/multiple_kibana_nodes.test.ts`

You may need to run the test 1-5 times before you hit a failure.


<details>

<summary>diff</summary>

```diff
diff --git i/src/core/server/integration_tests/saved_objects/migrations/multiple_kibana_nodes.test.ts w/src/core/server/integration_tests/saved_objects/migrations/multiple_kibana_nodes.test.ts
index 34df1d484b9..9985525b725 100644
--- i/src/core/server/integration_tests/saved_objects/migrations/multiple_kibana_nodes.test.ts
+++ w/src/core/server/integration_tests/saved_objects/migrations/multiple_kibana_nodes.test.ts
@@ -107,7 +107,7 @@ async function createRoot({ logFileName }: CreateRootConfig) {
 jest.setTimeout(15 * 60 * 1000);
 
 // FLAKY: https://github.com/elastic/kibana/issues/148263
-describe.skip('migration v2', () => {
+describe('migration v2', () => {
   let esServer: TestElasticsearchUtils;
   let rootA: Root;
   let rootB: Root;
@@ -149,6 +149,7 @@ describe.skip('migration v2', () => {
       settings: {
         es: {
           license: 'basic',
+          esArgs:['logger.org.elasticsearch.discovery=TRACE'],
           // original SOs: 5k of `foo` docs with this structure:
           // [
           //   { id: 'foo:1', type: 'foo', foo: { status: 'unmigrated' }, migrationVersion: { foo: '7.13.0' } },
@@ -201,7 +202,7 @@ describe.skip('migration v2', () => {
     });
   });
 
-  it('migrates saved objects normally when multiple Kibana instances are started with a small interval', async () => {
+  it.only('migrates saved objects normally when multiple Kibana instances are started with a small interval', async () => {
     const setupContracts = await Promise.all([rootA.setup(), rootB.setup(), rootC.setup()]);
 
     setupContracts.forEach((setup) => setup.savedObjects.registerType(fooType));
```
</details>

### Logs (if relevant)

[ES logs at TRACE level](https://github.com/elastic/elasticsearch/files/10343649/temp.log)

cc @rudolf 
server/src/main/java/org/elasticsearch/cluster/service/MasterService.java
this.threadPoolExecutorSupplier = threadPoolExecutorSupplier;
            this.warnThreshold = warnThreshold;
        }
        @Override
        public synchronized void onEmptyQueue() {
            isEmpty = true;
        }
        @Override
        public void onNonemptyQueue() {
            final long nowMillis = nowMillisSupplier.getAsLong();
            final long nonemptyDurationMillis;
            synchronized (this) {
                if (isEmpty) {
                    isEmpty = false;
                    nonemptySinceMillis = nowMillis;
                    lastLogMillis = nowMillis;
                    return;
                }
                if (nowMillis - lastLogMillis < warnThreshold) {
                    return;
                }
                lastLogMillis = nowMillis;
                nonemptyDurationMillis = nowMillis - nonemptySinceMillis;
            }
            final PrioritizedEsThreadPoolExecutor threadPoolExecutor = threadPoolExecutorSupplier.get();
            final TimeValue maxTaskWaitTime = threadPoolExecutor.getMaxTaskWaitTime();
            logger.warn(
                "pending task queue has been nonempty for [{}/{}ms] which is longer than the warn threshold of [{}ms];"
                    + " there are currently [{}] pending tasks, the oldest of which has age [{}/{}ms]",
                TimeValue.timeValueMillis(nonemptyDurationMillis),