### CI Link

https://gradle-enterprise.elastic.co/s/deqtxspnhxyva

### Repro line

./gradlew ':x-pack:plugin:ilm:qa:multi-node:javaRestTest' --tests "org.elasticsearch.xpack.ilm.TimeseriesMoveToStepIT.testMoveToRolloverStep" -Dtests.seed=81FF6263145E4101 -Dtests.locale=es-ES -Dtests.timezone=America/Godthab -Druntime.java=8

### Does it reproduce?

No

### Applicable branches

7.17

### Failure history

Failed two times in the last 30 days, all very recent: https://gradle-enterprise.elastic.co/s/yp56iltyyinha

### Failure excerpt

The tests `TimeseriesMoveToStepIT.testMoveToRolloverStep` and `TimeseriesMoveToStepIT.testMoveToAllocateStep` both failed. 

I'm not sure what is the root cause but this is maybe interesting:

```
» [2022-03-16T10:55:00,346][ERROR][o.e.ExceptionsHelper     ] [javaRestTest-0] fatal error
»  	at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$4(ExceptionsHelper.java:287)
»  	at java.util.Optional.ifPresent(Optional.java:159)
»  	at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:277)
»  	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:229)
»  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
»  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
»  	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
»  	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
»  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
»  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
»  	at java.lang.Thread.run(Thread.java:748)
» [2022-03-16T10:55:00,347][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [javaRestTest-0] fatal error in thread [Thread-4], exiting
»  java.lang.AssertionError: policy step registry cache failed sanity check
»  	at org.elasticsearch.xpack.ilm.PolicyStepsRegistry.getStep(PolicyStepsRegistry.java:378) ~[?:?]
»  	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.getCurrentStep(IndexLifecycleRunner.java:108) ~[?:?]
»  	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173) ~[?:?]
»  	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:418) ~[?:?]
»  	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:349) ~[?:?]
»  	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:186) ~[?:?]
»  	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:220) ~[?:?]
»  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_301]
»  	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_301]
»  	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_301]
»  	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_301]
»  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_301]
»  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_301]
»  	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_301]
»   ↓ last 40 non err
```

As well as :

```
"action":"init","step":"ERROR","step_time":"1647428098374"},"error_details":"{\"type\":\"initialize_policy_exception\",\"reason\":\"unable to initialize policy [policy-zZOnT] for index [retryinit-20xx-01-10]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"index name [retryinit-20xx-01-10] does not match pattern 
```

The test `ShrinkActionIT.testShrinkDuringSnapshot` also failed but I suspect it is more a consequence of the other failures.
x-pack/plugin/ilm/src/test/java/org/elasticsearch/xpack/ilm/PolicyStepsRegistryTests.java
.put("index.version.created", Version.CURRENT)
                    .put(LifecycleSettings.LIFECYCLE_NAME, "policy")
                    .build()
            )
            .putCustom(ILM_CUSTOM_METADATA_KEY, lifecycleState.build().asMap())
            .build();
        SortedMap<String, LifecyclePolicyMetadata> metas = new TreeMap<>();
        metas.put("policy", policyMetadata);
        PolicyStepsRegistry registry = new PolicyStepsRegistry(metas, null, null, REGISTRY, client, null);
        Step.StepKey badStepKey = new Step.StepKey(step.getKey().getPhase(), step.getKey().getAction(), step.getKey().getName() + "-bad");
        assertNull(registry.getStep(indexMetadata, badStepKey));
        
        assertNull(registry.getStep(indexMetadata, badStepKey));
    }
    public void testUpdateFromNothingToSomethingToNothing() throws Exception {
        Index index = new Index("test", "uuid");
        Client client = mock(Client.class);
        Mockito.when(client.settings()).thenReturn(Settings.EMPTY);
        String policyName = randomAlphaOfLength(5);
        LifecyclePolicy newPolicy = LifecyclePolicyTests.randomTestLifecyclePolicy(policyName);
        logger.info("--> policy: {}", newPolicy);
        List<Step> policySteps = newPolicy.toSteps(client, null);
        Map<String, String> headers = new HashMap<>();
        if (randomBoolean()) {
            headers.put(randomAlphaOfLength(10), randomAlphaOfLength(10));
            headers.put(randomAlphaOfLength(10), randomAlphaOfLength(10));
        }
        Map<String, LifecyclePolicyMetadata> policyMap = Collections.singletonMap(
            newPolicy.getName(),
            new LifecyclePolicyMetadata(newPolicy, headers, randomNonNegativeLong(), randomNonNegativeLong())