Today we close the election scheduler when the coordinator leaves mode `CANDIDATE`, before even starting the publication that establishes the election winner as the cluster master. If this publication subsequently fails then we start a new election scheduler with the original, short, timeout, and do not back off. With very high numbers of master-eligible nodes this can lead to constant election clashes that never resolve. We must count such failed publications as failed election attempts for election scheduling and backoff purposes.

More precisely, I think a node in mode `LEADER` should not count an election as truly successful until we've received join votes from all nodes in the cluster (see `org.elasticsearch.cluster.coordination.CoordinationState#containsJoinVoteFor`) at the end of a fully-acked publication. That should be equivalent to `currentTerm() == maxTermSeen` at the end of a fully-acked publication, since we increase `maxTermSeen` on a missing join vote.

I'm not sure how a node in mode `FOLLOWER` should detect that the cluster is completely stable. The master could broadcast another message when it decides things are stable perhaps? Or maybe it would be good enough to base it on elapsed time (i.e. if we've been `FOLLOWER` in the same term for 60s)?

---

### Workaround

The simplest workaround is not to have so many master-eligible nodes. See [these docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability-cluster-small-clusters.html#high-availability-cluster-design-three-plus-nodes) for more information:

> However, it is good practice to limit the number of master-eligible nodes in the cluster to three. Master nodes do not scale like other node types since the cluster always elects just one of them as the master of the cluster. If there are too many master-eligible nodes then master elections may take a longer time to complete.
server/src/test/java/org/elasticsearch/cluster/coordination/CoordinatorTests.java
cluster.stabilise();
            assertTrue(cluster.getAnyNodeExcept(cluster.getAnyLeader()).disconnect()); // 
            cluster.stabilise();
        }
    }
    @TestLogging(
        reason = "testing LagDetector and CoordinatorPublication logging",
        value = "org.elasticsearch.cluster.coordination.LagDetector:DEBUG,"
            + "org.elasticsearch.cluster.coordination.Coordinator.CoordinatorPublication:INFO"
    )
    public void testLogsMessagesIfPublicationDelayed() {
        try (Cluster cluster = new Cluster(between(3, 5))) {
            cluster.runRandomly();
            cluster.stabilise();
            final ClusterNode brokenNode = cluster.getAnyNodeExcept(cluster.getAnyLeader());
            final MockLogAppender mockLogAppender = new MockLogAppender();
            try (var ignored = mockLogAppender.capturing(Coordinator.CoordinatorPublication.class, LagDetector.class)) {
                mockLogAppender.addExpectation(
                    new MockLogAppender.SeenEventExpectation(
                        "publication info message",
                        Coordinator.CoordinatorPublication.class.getCanonicalName(),
                        Level.INFO,
                        "after [*] publication of cluster state version [*] is still waiting for "
                            + brokenNode.getLocalNode()
                            + " ["
                            + Publication.PublicationTargetState.SENT_PUBLISH_REQUEST
                            + ']'
                    )
                );
                mockLogAppender.addExpectation(