### Elasticsearch Version

8.0.1

### Installed Plugins

_No response_

### Java Version

_bundled_

### OS Version

Elastic Cloud

### Problem Description

After upgrading from 7.17 to 8.0.1 we noticed that some watches would not write their history.
The ES logs has many log lines similar to
`[instance-0000000115] watch history could not be written [mywatch_5a0585ef-b8b1-4526-8b24-9fb9bf58981d-2022-03-15T11:29:32.801579308Z], failure [org.elasticsearch.index.mapper.MapperParsingException: failed to parse]`

I noticed that `.watcher-history-16` has the wrong mapping as everything just has default `.text`+`.keyword`, and the index template has a lot more specific fields. But the index template applies to a data stream, but `.watcher-history-16` got created as an index, so the template didn't apply. It should have been created as a data stream after https://github.com/elastic/elasticsearch/pull/64252, but somehow it didn't.




### Steps to Reproduce

(unconfirmed)
Create a 7.17 cluster with multiple nodes
Create multiple watches that fire rapidly.
Upgrade ES to 8.x
`.watcher-history-16` should be created as data stream, but got created as an index.

I tried it with a simple cluster and one watch firing every second and it upgraded to datastream, so it might be a race condition or depend on how many watches/how many nodes (we have ~80 watches/12 nodes)

### Logs (if relevant)

_No response_

This looks a bit similar to https://github.com/elastic/elasticsearch/issues/56732 except now that it's a data stream it won't be "fixed" by waiting until roll over.
x-pack/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/support/WatcherIndexTemplateRegistry.java
Client client,
        NamedXContentRegistry xContentRegistry
    ) {
        super(nodeSettings, clusterService, threadPool, client, xContentRegistry);
        ilmManagementEnabled = Watcher.USE_ILM_INDEX_MANAGEMENT.get(nodeSettings);
    }
    private static final Map<String, ComposableIndexTemplate> TEMPLATES_WATCH_HISTORY = parseComposableTemplates(
        new IndexTemplateConfig(
            WatcherIndexTemplateRegistryField.HISTORY_TEMPLATE_NAME,
            "/watch-history.json",
            WatcherIndexTemplateRegistryField.INDEX_TEMPLATE_VERSION,
            WATCHER_TEMPLATE_VERSION_VARIABLE
        )
    );
    private static final Map<String, ComposableIndexTemplate> TEMPLATES_WATCH_HISTORY_NO_ILM = parseComposableTemplates(
        new IndexTemplateConfig(
            WatcherIndexTemplateRegistryField.HISTORY_TEMPLATE_NAME_NO_ILM,
            "/watch-history-no-ilm.json",
            WatcherIndexTemplateRegistryField.INDEX_TEMPLATE_VERSION,
            WATCHER_TEMPLATE_VERSION_VARIABLE
        )
    );
    @Override
    protected Map<String, ComposableIndexTemplate> getComposableTemplateConfigs() {
        return ilmManagementEnabled ? TEMPLATES_WATCH_HISTORY : TEMPLATES_WATCH_HISTORY_NO_ILM;
    }
    private static final List<LifecyclePolicy> LIFECYCLE_POLICIES = List.of(
        new LifecyclePolicyConfig("watch-history-ilm-policy-16", "/watch-history-ilm-policy.json").load(
            LifecyclePolicyConfig.DEFAULT_X_CONTENT_REGISTRY
        )