### Elasticsearch Version

many

### Installed Plugins

none

### Java Version

_bundled_

### OS Version

all

### Problem Description

The `filter_by_filter` optimization should kick in when it makes aggs faster. We have a switch to turn it off, but we advise folks not to because it usually does a good job. Well, we've found a case where the heuristics do a bad job of picking the implementation - specifically I've seen a case when the `_doc_count` field exists and we're slower. I'm not really sure if this is universal, but it could be. I'm trying to figure out if this is an all the time thing or a sometimes thing.

### Steps to Reproduce

index docs with `_doc_count`. run a terms agg on a field with a cardinality in the ball park of 500.

### Logs (if relevant)

_No response_
server/src/main/java/org/elasticsearch/search/aggregations/bucket/filter/FilterByFilterAggregator.java
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.function.BiConsumer;
/**
 * Collects results by running each filter against the searcher and doesn't
 * build any {@link LeafBucketCollector}s which is generally faster than
 * {@link Compatible} but doesn't support when there is a parent aggregator
 * or any child aggregators.
 */
public class FilterByFilterAggregator extends FiltersAggregator {
    /**
     * Builds {@link FilterByFilterAggregator} when the filters are valid and
     * it would be faster than a "native" aggregation implementation. The
     * interface is designed to allow easy construction of
     * {@link AdaptingAggregator}.
     */
    public abstract static class AdapterBuilder<T> {
        private final String name;
        private final List<QueryToFilterAdapter<?>> filters = new ArrayList<>();
        private final boolean keyed;
        private final AggregationContext context;
        private final Aggregator parent;
        private final CardinalityUpperBound cardinality;
        private final Map<String, Object> metadata;
        private final Query rewrittenTopLevelQuery;
        private boolean valid = true;
        public AdapterBuilder(
            String name,
            boolean keyed,