https://github.com/elastic/elasticsearch/issues/99317 documents a deadlock in the JWT realm. The underlying deadlock happens in the Cache implementation under a very specific situation. Copying the stack trace from that issue here illustrates the issue. 

```
Found one Java-level deadlock:
=============================
"elasticsearch[es-es-search-5577dfb9fd-fcjtw][transport_worker][T#12]":
  waiting for ownable synchronizer 0x000000070826d178, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "elasticsearch[es-es-search-5577dfb9fd-fcjtw][transport_worker][T#20]"

"elasticsearch[es-es-search-5577dfb9fd-fcjtw][transport_worker][T#20]":
  waiting for ownable synchronizer 0x0000000708ff86b8, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "elasticsearch[es-es-search-5577dfb9fd-fcjtw][system_critical_read][T#1]"

"elasticsearch[es-es-search-5577dfb9fd-fcjtw][system_critical_read][T#1]":
  waiting for ownable synchronizer 0x000000070826d178, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
  which is held by "elasticsearch[es-es-search-5577dfb9fd-fcjtw][transport_worker][T#20]"
```

<details>
<summary>Expand for more complete stack traces</summary>

```
"elasticsearch[es-es-search-5577dfb9fd-fcjtw][transport_worker][T#12]":
    at jdk.internal.misc.Unsafe.park(java.base@20.0.2/Native Method)
    - parking to wait for  <0x000000070826d178> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(java.base@20.0.2/LockSupport.java:221)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@20.0.2/AbstractQueuedSynchronizer.java:754)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@20.0.2/AbstractQueuedSynchronizer.java:1079)
    at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@20.0.2/ReentrantReadWriteLock.java:738)
    at org.elasticsearch.common.util.concurrent.ReleasableLock.acquire(org.elasticsearch.server@8.11.0-SNAPSHOT/ReleasableLock.java:43)
    at org.elasticsearch.common.cache.Cache$CacheSegment.get(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:199)
    at org.elasticsearch.common.cache.Cache.get(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:350)
    at org.elasticsearch.common.cache.Cache.get(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:345)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm.tryAuthenticateWithCache(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtRealm.java:348)


"elasticsearch[es-es-search-5577dfb9fd-fcjtw][transport_worker][T#20]":
    at jdk.internal.misc.Unsafe.park(java.base@20.0.2/Native Method)
    - parking to wait for  <0x0000000708ff86b8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(java.base@20.0.2/LockSupport.java:221)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@20.0.2/AbstractQueuedSynchronizer.java:754)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@20.0.2/AbstractQueuedSynchronizer.java:990)
    at java.util.concurrent.locks.ReentrantLock$Sync.lock(java.base@20.0.2/ReentrantLock.java:153)
    at java.util.concurrent.locks.ReentrantLock.lock(java.base@20.0.2/ReentrantLock.java:322)
    at org.elasticsearch.common.util.concurrent.ReleasableLock.acquire(org.elasticsearch.server@8.11.0-SNAPSHOT/ReleasableLock.java:43)
    at org.elasticsearch.common.cache.Cache.invalidateAll(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:529)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm.invalidateJwtCache(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtRealm.java:453)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm.expireAll(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtRealm.java:181)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm$$Lambda$4053/0x0000000101c181f0.reloaded(org.elasticsearch.security@8.11.0-SNAPSHOT/Unknown Source)
    at org.elasticsearch.xpack.security.authc.jwt.JwtSignatureValidator$PkcJwtSignatureValidator.lambda$validate$0(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtSignatureValidator.java:311)
    at org.elasticsearch.xpack.security.authc.jwt.JwtSignatureValidator$PkcJwtSignatureValidator$$Lambda$8830/0x00000001025cdb48.accept(org.elasticsearch.security@8.11.0-SNAPSHOT/Unknown Source)
    at org.elasticsearch.action.ActionListener$2.onResponse(org.elasticsearch.server@8.11.0-SNAPSHOT/ActionListener.java:177)
    at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(org.elasticsearch.server@8.11.0-SNAPSHOT/SubscribableListener.java:273)
    at org.elasticsearch.action.support.SubscribableListener.tryComplete(org.elasticsearch.server@8.11.0-SNAPSHOT/SubscribableListener.java:193)
    at org.elasticsearch.action.support.SubscribableListener.addListener(org.elasticsearch.server@8.11.0-SNAPSHOT/SubscribableListener.java:96)
    at org.elasticsearch.action.support.SubscribableListener.addListener(org.elasticsearch.server@8.11.0-SNAPSHOT/SubscribableListener.java:71)
    at org.elasticsearch.xpack.security.authc.jwt.JwkSetLoader.reload(org.elasticsearch.security@8.11.0-SNAPSHOT/JwkSetLoader.java:90)
    at org.elasticsearch.xpack.security.authc.jwt.JwtSignatureValidator$PkcJwtSignatureValidator.validate(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtSignatureValidator.java:292)
    at org.elasticsearch.xpack.security.authc.jwt.JwtSignatureValidator$DelegatingJwtSignatureValidator.validate(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtSignatureValidator.java:144)
    at org.elasticsearch.xpack.security.authc.jwt.JwtAuthenticator.validateSignature(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtAuthenticator.java:101)
    at org.elasticsearch.xpack.security.authc.jwt.JwtAuthenticator.authenticate(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtAuthenticator.java:93)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm.authenticate(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtRealm.java:291)
    at org.elasticsearch.xpack.security.authc.RealmsAuthenticator.lambda$consumeToken$4(org.elasticsearch.security@8.11.0-SNAPSHOT/RealmsAuthenticator.java:147)

"elasticsearch[es-es-search-5577dfb9fd-fcjtw][system_critical_read][T#1]":
    at jdk.internal.misc.Unsafe.park(java.base@20.0.2/Native Method)
    - parking to wait for  <0x000000070826d178> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
    at java.util.concurrent.locks.LockSupport.park(java.base@20.0.2/LockSupport.java:221)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@20.0.2/AbstractQueuedSynchronizer.java:754)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@20.0.2/AbstractQueuedSynchronizer.java:990)
    at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(java.base@20.0.2/ReentrantReadWriteLock.java:959)
    at org.elasticsearch.common.util.concurrent.ReleasableLock.acquire(org.elasticsearch.server@8.11.0-SNAPSHOT/ReleasableLock.java:43)
    at org.elasticsearch.common.cache.Cache$CacheSegment.remove(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:293)
    at org.elasticsearch.common.cache.Cache.evictEntry(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:764)
    at org.elasticsearch.common.cache.Cache.evict(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:755)
    at org.elasticsearch.common.cache.Cache.promote(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:746)
    at org.elasticsearch.common.cache.Cache.put(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:470)
    at org.elasticsearch.common.cache.Cache.put(org.elasticsearch.server@8.11.0-SNAPSHOT/Cache.java:457)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm.lambda$processValidatedJwt$4(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtRealm.java:408)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm$$Lambda$7938/0x000000010251aaa8.accept(org.elasticsearch.security@8.11.0-SNAPSHOT/Unknown Source)
    at org.elasticsearch.action.ActionListener$2.onResponse(org.elasticsearch.server@8.11.0-SNAPSHOT/ActionListener.java:177)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm.lambda$processValidatedJwt$5(org.elasticsearch.security@8.11.0-SNAPSHOT/JwtRealm.java:432)
    at org.elasticsearch.xpack.security.authc.jwt.JwtRealm$$Lambda$7957/0x000000010251cfa0.accept(org.elasticsearch.security@8.11.0-SNAPSHOT/Unknown Source)
    at org.elasticsearch.action.ActionListener$2.onResponse(org.elasticsearch.server@8.11.0-SNAPSHOT/ActionListener.java:177)
    at org.elasticsearch.xpack.security.authc.support.mapper.NativeRoleMappingStore.lambda$resolveRoles$18(org.elasticsearch.security@8.11.0-SNAPSHOT/NativeRoleMappingStore.java:381)
    at org.elasticsearch.xpack.security.authc.support.mapper.NativeRoleMappingStore$$Lambda$7969/0x000000010251e920.accept(org.elasticsearch.security@8.11.0-SNAPSHOT/Unknown Source)
    at org.elasticsearch.action.ActionListener$2.onResponse(org.elasticsearch.server@8.11.0-SNAPSHOT/ActionListener.java:177)
    at org.elasticsearch.xpack.security.authc.support.mapper.NativeRoleMappingStore.lambda$loadMappings$0(org.elasticsearch.security@8.11.0-SNAPSHOT/NativeRoleMappingStore.java:140)
    at org.elasticsearch.xpack.security.authc.support.mapper.NativeRoleMappingStore$$Lambda$7989/0x0000000102521370.accept(org.elasticsearch.security@8.11.0-SNAPSHOT/Unknown Source)
    at org.elasticsearch.action.ActionListener$2.onResponse(org.elasticsearch.server@8.11.0-SNAPSHOT/ActionListener.java:177)
    at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(org.elasticsearch.server@8.11.0-SNAPSHOT/ContextPreservingActionListener.java:32)
    at org.elasticsearch.xpack.core.security.ScrollHelper$1.onResponse(org.elasticsearch.xcore@8.11.0-SNAPSHOT/ScrollHelper.java:97)
```

</details>

Looking at the stack trace, the relevant threads are T#1 which is trying to do the Cache#put(k,v) , and T#20 which is trying to do the Cache#invalidateAll(). T#12 (and 16 other threads) are blocked due to the deadlock between T#1 and T#20.  In summary here is the relevant work the cache does for each workflow: 
Line numbers are relevant to commit 2461cb152369d6ef92ce0203d97f70596a1e4ecb (HEAD of main at time of investigation). 


T#20 invalidateAll()
```
- locks segement write lock on 526 (reentrent r/w lock)  
- locks lru lock on 529 (reentrant lock)
- unlocks lru lock on 542
- unlocks segment write lock on 546
```

T#1 put(k,v)
```
- locks lru lock on 464 (reentrant lock)
- (re)locks lru lock on 739
- (evict/evictEntry/remove) locks segment write lock on 293 (reentrent r/w lock) 
- (evict/evictEntry/remove) unlocks segment write lock on 310
- (re)unlocks lru lock on 748
- unlocks lru lock on 471
```
To put it another way:
```
T#1  put(k,v)        -> lock lru -> lock swl -> unlock swl -> unlock lru
T#20 invalidateAll() -> lock swl -> lock lru -> unlock lru -> unlock swl
```
So the sequence of events for a dead lock must be 

1) T#1 grabs the lru lock
2) T#20 grabs segment write lock
3) T#1 tries to grab the segment write lock but is blocked by T#20
4) T#20 tries to grab the lru lock but is blocked by T#1

or some variant of that. The important bit is that put(k,v) and invalidateAll() must be happening at the same time across different threads and the attempts to grab the different 2 locks must be intertwined between the 2 different threads. 

server/src/main/java/org/elasticsearch/common/cache/Cache.java
}
            if (future != null) {
                evictions.increment();
                notifyWithInvalidated(future);
            }
        }
        /**
         * remove an entry from the segment iff the future is done and the value is equal to the
         * expected value
         *
         * @param key the key of the entry to remove from the cache
         * @param value the value expected to be associated with the key
         * @param notify whether to trigger a removal notification if the entry has been removed
         */
        void remove(K key, V value, boolean notify) {
            CompletableFuture<Entry<K, V>> future;
            boolean removed = false;
            try (ReleasableLock ignored = writeLock.acquire()) {
                future = map == null ? null : map.get(key);
                try {
                    if (future != null) {
                        if (future.isDone()) {
                            Entry<K, V> entry = future.get();
                            if (Objects.equals(value, entry.value)) {
                                removed = map.remove(key, future);
                                if (map.isEmpty()) {
                                    map = null;
                                }
                            }
                        }