### Elasticsearch Version

many

### Installed Plugins

none

### Java Version

_bundled_

### OS Version

all

### Problem Description

The `filter_by_filter` optimization should kick in when it makes aggs faster. We have a switch to turn it off, but we advise folks not to because it usually does a good job. Well, we've found a case where the heuristics do a bad job of picking the implementation - specifically I've seen a case when the `_doc_count` field exists and we're slower. I'm not really sure if this is universal, but it could be. I'm trying to figure out if this is an all the time thing or a sometimes thing.

### Steps to Reproduce

index docs with `_doc_count`. run a terms agg on a field with a cardinality in the ball park of 500.

### Logs (if relevant)

_No response_
server/src/main/java/org/elasticsearch/search/aggregations/support/AggregationContext.java
public abstract Query filterQuery(Query query);
    /**
     * The settings for the index against which this search is running.
     */
    public abstract IndexSettings getIndexSettings();
    /**
     * Compile a sort.
     */
    public abstract Optional<SortAndFormats> buildSort(List<SortBuilder<?>> sortBuilders) throws IOException;
    /**
     * Get the {@link NestedLookup} of this index
     */
    public abstract NestedLookup nestedLookup();
    /**
     * Access the nested scope. Stay away from this unless you are dealing with nested.
     */
    public abstract NestedScope nestedScope();
    /**
     * Build a {@linkplain SubSearchContext} to power an aggregation fetching top hits.
     * Try to avoid using this because it pulls in a ton of dependencies.
     */
    public abstract SubSearchContext subSearchContext();
    /**
     * Cause this aggregation to be released when the search is finished.
     */
    public abstract void addReleasable(Aggregator aggregator);
    public abstract MultiBucketConsumer multiBucketConsumer();
    /**
     * Get the filter cache.
     */