Failure seen on a PR branch but this reproduces on current `main` (254e6bcabd18c26338e075287bd4d74b8b0e2257) too. Not every time but it doesn't take many iterations.

**Build scan:**
https://gradle-enterprise.elastic.co/s/3oce3rf4gjyio/tests/:x-pack:plugin:security:internalClusterTest/org.elasticsearch.xpack.security.authc.ApiKeyIntegTests/testUpdateApiKeysForSingleKey

**Reproduction line:**
`./gradlew ':x-pack:plugin:security:internalClusterTest' --tests "org.elasticsearch.xpack.security.authc.ApiKeyIntegTests.testUpdateApiKeysForSingleKey" -Dtests.seed=B36D864BB8DCE44B -Dtests.locale=ar-KW -Dtests.timezone=Europe/Belfast -Druntime.java=17`

**Applicable branches:**
main

**Reproduces locally?:**
Yes

**Failure history:**
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.security.authc.ApiKeyIntegTests&tests.test=testUpdateApiKeysForSingleKey

**Failure excerpt:**
```
java.lang.AssertionError: for api key doc {creation_time=1660033360930, api_key_invalidated=false, metadata_flattened=null, creator={principal=test_user, metadata={}, realm_type=file, full_name=null, realm=file, email=null}, limited_by_role_descriptors={user={cluster=[ALL], indices=[{privileges=[ALL], names=[*], allow_restricted_indices=true}], metadata={}, run_as=[], type=role, applications=[]}}, expiration_time=null, role_descriptors={role={cluster=[monitor], indices=[], metadata={}, run_as=[], type=role, applications=[]}}, api_key_hash={PBKDF2}10000$4ujcIQSpiY8A8CH+ueFWazxqRBkoRPuch5FbcK6Vdh4=$oEzJjucin31uSe+osf67BHCEiKO5cau4UU/ka2sQGNg=, name=test-key-cBtQrBBV0, doc_type=api_key, version=8050099}
Expected: <{}>
     but: was null

  at __randomizedtesting.SeedInfo.seed([B36D864BB8DCE44B:AF9F821EACBB4A86]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.junit.Assert.assertThat(Assert.java:956)
  at org.elasticsearch.xpack.security.authc.ApiKeyIntegTests.expectMetadataForApiKey(ApiKeyIntegTests.java:2346)
  at org.elasticsearch.xpack.security.authc.ApiKeyIntegTests.expectAttributesForApiKey(ApiKeyIntegTests.java:2319)
  at org.elasticsearch.xpack.security.authc.ApiKeyIntegTests.testUpdateApiKeysForSingleKey(ApiKeyIntegTests.java:1548)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:833)

```
x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/authc/ApiKeyIntegTests.java
final RealmConfig.RealmIdentifier creatorRealmOnCreatedApiKey = new RealmConfig.RealmIdentifier(FileRealmSettings.TYPE, "file");
        final RealmConfig.RealmIdentifier otherRealmInDomain = AuthenticationTestHelper.randomRealmIdentifier(true);
        final var realmDomain = new RealmDomain(
            ESTestCase.randomAlphaOfLengthBetween(3, 8),
            Set.of(creatorRealmOnCreatedApiKey, otherRealmInDomain)
        );
        
        final var authenticatingRealm = randomFrom(creatorRealmOnCreatedApiKey, otherRealmInDomain);
        final var authentication = randomValueOtherThanMany(
            Authentication::isApiKey,
            () -> AuthenticationTestHelper.builder()
                .user(new User(TEST_USER_NAME, TEST_ROLE))
                .realmRef(
                    new Authentication.RealmRef(
                        authenticatingRealm.getName(),
                        authenticatingRealm.getType(),
                        serviceWithNodeName.nodeName(),
                        realmDomain
                    )
                )
                .build()
        );
        final BulkUpdateApiKeyResponse response = updateApiKeys(
            serviceWithNodeName.service(),
            authentication,
            BulkUpdateApiKeyRequest.usingApiKeyIds(apiKeyId),
            Set.of()
        );
        assertSingleUpdate(apiKeyId, response);
        final Map<String, Object> expectedCreator = new HashMap<>();