### Elasticsearch Version

8.0.1

### Installed Plugins

_No response_

### Java Version

_bundled_

### OS Version

Elastic Cloud

### Problem Description

After upgrading from 7.17 to 8.0.1 we noticed that some watches would not write their history.
The ES logs has many log lines similar to
`[instance-0000000115] watch history could not be written [mywatch_5a0585ef-b8b1-4526-8b24-9fb9bf58981d-2022-03-15T11:29:32.801579308Z], failure [org.elasticsearch.index.mapper.MapperParsingException: failed to parse]`

I noticed that `.watcher-history-16` has the wrong mapping as everything just has default `.text`+`.keyword`, and the index template has a lot more specific fields. But the index template applies to a data stream, but `.watcher-history-16` got created as an index, so the template didn't apply. It should have been created as a data stream after https://github.com/elastic/elasticsearch/pull/64252, but somehow it didn't.




### Steps to Reproduce

(unconfirmed)
Create a 7.17 cluster with multiple nodes
Create multiple watches that fire rapidly.
Upgrade ES to 8.x
`.watcher-history-16` should be created as data stream, but got created as an index.

I tried it with a simple cluster and one watch firing every second and it upgraded to datastream, so it might be a race condition or depend on how many watches/how many nodes (we have ~80 watches/12 nodes)

### Logs (if relevant)

_No response_

This looks a bit similar to https://github.com/elastic/elasticsearch/issues/56732 except now that it's a data stream it won't be "fixed" by waiting until roll over.
x-pack/plugin/watcher/src/main/java/org/elasticsearch/xpack/watcher/support/WatcherIndexTemplateRegistry.java
public static final String WATCHER_TEMPLATE_VERSION_VARIABLE = "xpack.watcher.template.version";
    private final boolean ilmManagementEnabled;
    public WatcherIndexTemplateRegistry(
        Settings nodeSettings,
        ClusterService clusterService,
        ThreadPool threadPool,