We observed a node (running a somewhat nonstandard build of ES in a fairly unusual environment) which was deadlocked while invalidating a JWT credentials cache entry while rotation credentials. It looked like it might be possible to experience the same deadlock in regular ES too.

```
Found one Java-level deadlock:
=============================
"elasticsearch[es-es-search-5577dfb9fd-fcjtw][transport_worker][T#12]":
  waiting for ownable synchronizer 0x000000070826d178, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "elasticsearch[es-es-search-5577dfb9fd-fcjtw][transport_worker][T#20]"

"elasticsearch[es-es-search-5577dfb9fd-fcjtw][transport_worker][T#20]":
  waiting for ownable synchronizer 0x0000000708ff86b8, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "elasticsearch[es-es-search-5577dfb9fd-fcjtw][system_critical_read][T#1]"

"elasticsearch[es-es-search-5577dfb9fd-fcjtw][system_critical_read][T#1]":
  waiting for ownable synchronizer 0x000000070826d178, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "elasticsearch[es-es-search-5577dfb9fd-fcjtw][transport_worker][T#20]"
```

<details>
<summary>Expand for more complete stack traces</summary>

```
"elasticsearch[es-es-search-5577dfb9fd-fcjtw][transport_worker][T#12]":
    at jdk.internal.misc.Unsafe.park(java.base@20.0.2/Native Method)
    - parking to wait for  <0x000000070826d178> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(java.base@20.0.2/LockSupport.java:221)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@20.0.2/AbstractQueuedSynchronizer.java:754)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@20.0.2/AbstractQueuedSynchronizer.java:1079)
    at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@20.0.2/ReentrantReadWriteLock.java:738)
    at org.elasticsearch.common.util.concurrent.ReleasableLock.acquire(org.elasticsearch.server@8.11.0-SNAPSHOT/ReleasableLock.java:43)
    at org.elasticsearch.common.cache.Cache$CacheSegment.get(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:199)
    at org.elasticsearch.common.cache.Cache.get(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:350)
    at org.elasticsearch.common.cache.Cache.get(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:345)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm.tryAuthenticateWithCache(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtRealm.java:348)


"elasticsearch[es-es-search-5577dfb9fd-fcjtw][transport_worker][T#20]":
    at jdk.internal.misc.Unsafe.park(java.base@20.0.2/Native Method)
    - parking to wait for  <0x0000000708ff86b8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(java.base@20.0.2/LockSupport.java:221)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@20.0.2/AbstractQueuedSynchronizer.java:754)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@20.0.2/AbstractQueuedSynchronizer.java:990)
    at java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@20.0.2/ReentrantLock.java:153)
    at java.util.concurrent.locks.ReentrantLock.lock(java.base@20.0.2/ReentrantLock.java:322)
    at org.elasticsearch.common.util.concurrent.ReleasableLock.acquire(org.elasticsearch.server@8.11.0-SNAPSHOT/ReleasableLock.java:43)
    at org.elasticsearch.common.cache.Cache.invalidateAll(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:529)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm.invalidateJwtCache(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtRealm.java:453)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm.expireAll(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtRealm.java:181)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm$$Lambda$4053/0x0000000101c181f0.reloaded(org.elasticsearch.security@8.11.0-SNAPSHOT/Unknown Source)
    at org.elasticsearch.xpack.security.authc.jwt.JwtSignatureValidator$PkcJwtSignatureValidator.lambda$validate$0(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtSignatureValidator.java:311)
    at org.elasticsearch.xpack.security.authc.jwt.JwtSignatureValidator$PkcJwtSignatureValidator$$Lambda$8830/0x00000001025cdb48.accept(org.elasticsearch.security@8.11.0-SNAPSHOT/Unknown Source)
    at org.elasticsearch.action.ActionListener$2.onResponse(org.elasticsearch.server@8.11.0-SNAPSHOT/ActionListener.java:177)
    at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(org.elasticsearch.server@8.11.0-SNAPSHOT/SubscribableListener.java:273)
    at org.elasticsearch.action.support.SubscribableListener.tryComplete(org.elasticsearch.server@8.11.0-SNAPSHOT/SubscribableListener.java:193)
    at org.elasticsearch.action.support.SubscribableListener.addListener(org.elasticsearch.server@8.11.0-SNAPSHOT/SubscribableListener.java:96)
    at org.elasticsearch.action.support.SubscribableListener.addListener(org.elasticsearch.server@8.11.0-SNAPSHOT/SubscribableListener.java:71)
    at org.elasticsearch.xpack.security.authc.jwt.JwkSetLoader.reload(org.elasticsearch.security@8.11.0-SNAPSHOT/JwkSetLoader.java:90)
    at org.elasticsearch.xpack.security.authc.jwt.JwtSignatureValidator$PkcJwtSignatureValidator.validate(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtSignatureValidator.java:292)
    at org.elasticsearch.xpack.security.authc.jwt.JwtSignatureValidator$DelegatingJwtSignatureValidator.validate(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtSignatureValidator.java:144)
    at org.elasticsearch.xpack.security.authc.jwt.JwtAuthenticator.validateSignature(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtAuthenticator.java:101)
    at org.elasticsearch.xpack.security.authc.jwt.JwtAuthenticator.authenticate(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtAuthenticator.java:93)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm.authenticate(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtRealm.java:291)
    at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(org.elasticsearch.security@8.11.0-SNAPSHOT/RealmsAuthenticator.java:147)

"elasticsearch[es-es-search-5577dfb9fd-fcjtw][system_critical_read][T#1]":
    at jdk.internal.misc.Unsafe.park(java.base@20.0.2/Native Method)
    - parking to wait for  <0x000000070826d178> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(java.base@20.0.2/LockSupport.java:221)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@20.0.2/AbstractQueuedSynchronizer.java:754)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@20.0.2/AbstractQueuedSynchronizer.java:990)
    at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(java.base@20.0.2/ReentrantReadWriteLock.java:959)
    at org.elasticsearch.common.util.concurrent.ReleasableLock.acquire(org.elasticsearch.server@8.11.0-SNAPSHOT/ReleasableLock.java:43)
    at org.elasticsearch.common.cache.Cache$CacheSegment.remove(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:293)
    at org.elasticsearch.common.cache.Cache.evictEntry(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:764)
    at org.elasticsearch.common.cache.Cache.evict(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:755)
    at org.elasticsearch.common.cache.Cache.promote(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:746)
    at org.elasticsearch.common.cache.Cache.put(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:470)
    at org.elasticsearch.common.cache.Cache.put(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:457)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm.lambda$processValidatedJwt$4(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtRealm.java:408)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm$$Lambda$7938/0x000000010251aaa8.accept(org.elasticsearch.security@8.11.0-SNAPSHOT/Unknown Source)
    at org.elasticsearch.action.ActionListener$2.onResponse(org.elasticsearch.server@8.11.0-SNAPSHOT/ActionListener.java:177)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm.lambda$processValidatedJwt$5(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtRealm.java:432)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm$$Lambda$7957/0x000000010251cfa0.accept(org.elasticsearch.security@8.11.0-SNAPSHOT/Unknown Source)
    at org.elasticsearch.action.ActionListener$2.onResponse(org.elasticsearch.server@8.11.0-SNAPSHOT/ActionListener.java:177)
    at org.elasticsearch.xpack.security.authc.support.mapper.NativeRoleMappingStore.lambda$resolveRoles$18(org.elasticsearch.security@8.11.0-SNAPSHOT/NativeRoleMappingStore.java:381)
    at org.elasticsearch.xpack.security.authc.support.mapper.NativeRoleMappingStore$$Lambda$7969/0x000000010251e920.accept(org.elasticsearch.security@8.11.0-SNAPSHOT/Unknown Source)
    at org.elasticsearch.action.ActionListener$2.onResponse(org.elasticsearch.server@8.11.0-SNAPSHOT/ActionListener.java:177)
    at org.elasticsearch.xpack.security.authc.support.mapper.NativeRoleMappingStore.lambda$loadMappings$0(org.elasticsearch.security@8.11.0-SNAPSHOT/NativeRoleMappingStore.java:140)
    at org.elasticsearch.xpack.security.authc.support.mapper.NativeRoleMappingStore$$Lambda$7989/0x0000000102521370.accept(org.elasticsearch.security@8.11.0-SNAPSHOT/Unknown Source)
    at org.elasticsearch.action.ActionListener$2.onResponse(org.elasticsearch.server@8.11.0-SNAPSHOT/ActionListener.java:177)
    at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(org.elasticsearch.server@8.11.0-SNAPSHOT/ContextPreservingActionListener.java:32)
    at org.elasticsearch.xpack.core.security.ScrollHelper$1.onResponse(org.elasticsearch.xcore@8.11.0-SNAPSHOT/ScrollHelper.java:97)
```

</details>

---

### Symptoms

In this case the node left the cluster and could not rejoin, reporting timeouts. In other cases it might remain a member of the cluster, but some requests to the node will hang.

To diagnose the problem, use `jstack` to take a stack dump of the problematic node, look for reports of a deadlock, and compare the stack traces to the ones listed above.

---

### Workaround

Disable caching for all JWT-type realms:

```
xpack.security.authc.realms.jwt.<realmname>.jwt.cache.ttl: "0m"
```

or 

```
xpack.security.authc.realms.jwt.<realmname>.jwt.cache.size: 0
```

Note that this may have a performance impact.
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authc/jwt/JwtRealmAuthenticateTests.java
final JwtIssuer jwtIssuer = createJwtIssuer(0, 12, 1, 1, 1, createHttpsServer);
        assertThat(jwtIssuer.httpsServer, notNullValue());
        try {
            final JwtRealmSettingsBuilder jwtRealmSettingsBuilder = createJwtRealmSettingsBuilder(jwtIssuer, 0, 0);
            final String configKey = RealmSettings.getFullSettingKey(jwtRealmSettingsBuilder.name(), JwtRealmSettings.PKC_JWKSET_PATH);
            final String configValue = jwtIssuer.httpsServer.url.replace("/valid/", "/invalid"); 
            jwtRealmSettingsBuilder.settingsBuilder().put(configKey, configValue);
            final Exception exception = expectThrows(
                SettingsException.class,
                () -> createJwtRealm(allRealms, jwtIssuer, jwtRealmSettingsBuilder)
            );
            assertThat(exception.getMessage(), equalTo("Can't get contents for setting [" + configKey + "] value [" + configValue + "]."));
            assertThat(exception.getCause().getMessage(), equalTo("Get [" + configValue + "] failed, status [404], reason [Not Found]."));
        } finally {
            jwtIssuer.close();
        }
    }
    /**
     * Test token parse failures and authentication failures.
     * @throws Exception Unexpected test failure
     */
    public void testJwtValidationFailures() throws Exception {
        jwtIssuerAndRealms = generateJwtIssuerRealmPairs(
            randomIntBetween(1, 1), 
            randomIntBetween(0, 0), 
            randomIntBetween(1, JwtRealmSettings.SUPPORTED_SIGNATURE_ALGORITHMS.size()), 
            randomIntBetween(1, 1), 
            randomIntBetween(1, 1), 
            randomIntBetween(1, 1), 
            randomIntBetween(0, 1),