### Elasticsearch Version

many

### Installed Plugins

none

### Java Version

_bundled_

### OS Version

all

### Problem Description

The `filter_by_filter` optimization should kick in when it makes aggs faster. We have a switch to turn it off, but we advise folks not to because it usually does a good job. Well, we've found a case where the heuristics do a bad job of picking the implementation - specifically I've seen a case when the `_doc_count` field exists and we're slower. I'm not really sure if this is universal, but it could be. I'm trying to figure out if this is an all the time thing or a sometimes thing.

### Steps to Reproduce

index docs with `_doc_count`. run a terms agg on a field with a cardinality in the ball park of 500.

### Logs (if relevant)

_No response_
server/src/main/java/org/elasticsearch/search/aggregations/support/AggregationContext.java
* {@link ValuesSource data sources} that power them.
 * <p>
 * In production we always use the {@link ProductionAggregationContext} but
 * this is {@code abstract} so that tests can build it without creating the
 * massing {@link SearchExecutionContext}.
 * <p>
 * {@linkplain AggregationContext}s are {@link Releasable} because they track
 * the {@link Aggregator}s they build and {@link Aggregator#close} them when
 * the request is done. {@linkplain AggregationContext} may also preallocate
 * bytes on the "REQUEST" breaker and is responsible for releasing those bytes.
 */
public abstract class AggregationContext implements Releasable {
    /**
     * The query at the top level of the search in which these aggregations are running.
     */
    public abstract Query query();
    /**
     * Wrap the aggregator for profiling if profiling is enabled.
     */
    public abstract Aggregator profileIfEnabled(Aggregator agg) throws IOException;
    /**
     * Are we profiling the aggregation?
     */
    public abstract boolean profiling();
    /**
     * The time in milliseconds that is shared across all resources involved. Even across shards and nodes.
     */
    public abstract long nowInMillis();
    /**
     * Lookup the context for a field.