**Build scan:**
https://gradle-enterprise.elastic.co/s/lv6phjx26lutw/tests/:server:internalClusterTest/org.elasticsearch.index.store.CorruptedTranslogIT/testCorruptTranslogFiles

**Reproduction line:**
```
./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.index.store.CorruptedTranslogIT.testCorruptTranslogFiles" -Dtests.seed=212515C1E2B002FA -Dtests.locale=tr -Dtests.timezone=Asia/Harbin -Druntime.java=20
```

**Applicable branches:**
main

**Reproduces locally?:**
No

**Failure history:**
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.index.store.CorruptedTranslogIT&tests.test=testCorruptTranslogFiles

**Failure excerpt:**
```
java.lang.AssertionError: {"index":"test","shard":0,"primary":true,"current_state":"unassigned","unassigned_info":{"reason":"ALLOCATION_FAILED","at":"2023-04-27T12:44:16.620Z","failed_allocation_attempts":5,"details":"failed shard on node [TD4beuiGSFakg3XYvQ8xvg]: shard failure, reason [failed to recover from translog], failure [test/QpKaQjntQvO60IE0nV-IhQ][[test][0]] org.elasticsearch.index.engine.EngineException: failed to recover from translog\n\tat org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:544)\n\tat org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:517)\n\tat org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:124)\n\tat org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1933)\n\tat org.elasticsearch.index.shard.StoreRecovery.lambda$internalRecoverFromStore$9(StoreRecovery.java:479)\n\tat org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.index.shard.StoreRecovery.lambda$internalRecoverFromStore$10(StoreRecovery.java:488)\n\tat org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:151)\n\tat org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:122)\n\tat org.elasticsearch.index.CompositeIndexEventListener.iterateBeforeIndexShardRecovery(CompositeIndexEventListener.java:280)\n\tat org.elasticsearch.index.CompositeIndexEventListener.beforeIndexShardRecovery(CompositeIndexEventListener.java:289)\n\tat org.elasticsearch.index.shard.IndexShard.preRecovery(IndexShard.java:1658)\n\tat org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:411)\n\tat org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93)\n\tat org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2284)\n\tat org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\nCaused by: [test/QpKaQjntQvO60IE0nV-IhQ][[test][0]] org.elasticsearch.index.shard.IllegalIndexShardStateException: CurrentState[CLOSED] operation only allowed when recovering, origin [LOCAL_TRANSLOG_RECOVERY]\n\tat org.elasticsearch.index.shard.IndexShard.ensureWriteAllowed(IndexShard.java:2132)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:962)\n\tat org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1824)\n\tat org.elasticsearch.index.shard.IndexShard.runTranslogRecovery(IndexShard.java:1872)\n\tat org.elasticsearch.index.shard.IndexShard.lambda$openEngineAndRecoverFromTranslog$12(IndexShard.java:1924)\n\tat org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:542)\n\t... 21 more\n","last_allocation_status":"no"},"can_allocate":"no","allocate_explanation":"Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster that hold an in-sync copy of its data. Choose a node to which you expect this shard to be allocated, find this node in the node-by-node explanation, and address the reasons which prevent Elasticsearch from allocating this shard there.","node_allocation_decisions":[{"node_id":"TD4beuiGSFakg3XYvQ8xvg","node_name":"node_s0","transport_address":"127.0.0.1:26561","node_decision":"no","store":{"in_sync":true,"allocation_id":"1SRkOaRIRCW8laMa0YLlEw"},"deciders":[{"decider":"max_retry","decision":"NO","explanation":"shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [POST /_cluster/reroute?retry_failed&metric=none] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2023-04-27T12:44:16.620Z], failed_attempts[5], failed_nodes[[TD4beuiGSFakg3XYvQ8xvg]], delayed=false, last_node[TD4beuiGSFakg3XYvQ8xvg], details[failed shard on node [TD4beuiGSFakg3XYvQ8xvg]: shard failure, reason [failed to recover from translog], failure [test/QpKaQjntQvO60IE0nV-IhQ][[test][0]] org.elasticsearch.index.engine.EngineException: failed to recover from translog\n\tat org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:544)\n\tat org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:517)\n\tat org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:124)\n\tat org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1933)\n\tat org.elasticsearch.index.shard.StoreRecovery.lambda$internalRecoverFromStore$9(StoreRecovery.java:479)\n\tat org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.index.shard.StoreRecovery.lambda$internalRecoverFromStore$10(StoreRecovery.java:488)\n\tat org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:151)\n\tat org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:122)\n\tat org.elasticsearch.index.CompositeIndexEventListener.iterateBeforeIndexShardRecovery(CompositeIndexEventListener.java:280)\n\tat org.elasticsearch.index.CompositeIndexEventListener.beforeIndexShardRecovery(CompositeIndexEventListener.java:289)\n\tat org.elasticsearch.index.shard.IndexShard.preRecovery(IndexShard.java:1658)\n\tat org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:411)\n\tat org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93)\n\tat org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2284)\n\tat org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\nCaused by: [test/QpKaQjntQvO60IE0nV-IhQ][[test][0]] org.elasticsearch.index.shard.IllegalIndexShardStateException: CurrentState[CLOSED] operation only allowed when recovering, origin [LOCAL_TRANSLOG_RECOVERY]\n\tat org.elasticsearch.index.shard.IndexShard.ensureWriteAllowed(IndexShard.java:2132)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:962)\n\tat org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1824)\n\tat org.elasticsearch.index.shard.IndexShard.runTranslogRecovery(IndexShard.java:1872)\n\tat org.elasticsearch.index.shard.IndexShard.lambda$openEngineAndRecoverFromTranslog$12(IndexShard.java:1924)\n\tat org.elasticsearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:542)\n\t... 21 more\n], allocation_status[deciders_no]]]"}]}]}
Expected: not null
     but: was null

  at __randomizedtesting.SeedInfo.seed([212515C1E2B002FA:2D144873BB752E69]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.junit.Assert.assertThat(Assert.java:956)
  at org.elasticsearch.index.store.CorruptedTranslogIT.lambda$testCorruptTranslogFiles$0(CorruptedTranslogIT.java:97)
  at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1158)
  at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1131)
  at org.elasticsearch.index.store.CorruptedTranslogIT.testCorruptTranslogFiles(CorruptedTranslogIT.java:83)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.lang.reflect.Method.invoke(Method.java:578)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1623)

```
server/src/main/java/org/elasticsearch/indices/cluster/IndicesClusterStateService.java
public void onRecoveryFailure(RecoveryFailedException e, boolean sendShardFailure) {
            handleRecoveryFailure(shardRouting, sendShardFailure, e);
        }
    }
    
    synchronized void handleRecoveryFailure(ShardRouting shardRouting, boolean sendShardFailure, Exception failure) {
        failAndRemoveShard(shardRouting, sendShardFailure, "failed recovery", failure, clusterService.state());
    }
    private void failAndRemoveShard(
        ShardRouting shardRouting,
        boolean sendShardFailure,
        String message,
        @Nullable Exception failure,
        ClusterState state
    ) {
        try {
            AllocatedIndex<? extends Shard> indexService = indicesService.indexService(shardRouting.shardId().getIndex());
            if (indexService != null) {
                Shard shard = indexService.getShardOrNull(shardRouting.shardId().id());
                if (shard != null && shard.routingEntry().isSameAllocation(shardRouting)) {
                    indexService.removeShard(shardRouting.shardId().id(), message);
                }
            }
        } catch (ShardNotFoundException e) {
            
        } catch (Exception inner) {
            inner.addSuppressed(failure);
            logger.warn(
                () -> format(
                    "[%s][%s] failed to remove shard after failure ([%s])",