### Elasticsearch Version

many

### Installed Plugins

none

### Java Version

_bundled_

### OS Version

all

### Problem Description

The `filter_by_filter` optimization should kick in when it makes aggs faster. We have a switch to turn it off, but we advise folks not to because it usually does a good job. Well, we've found a case where the heuristics do a bad job of picking the implementation - specifically I've seen a case when the `_doc_count` field exists and we're slower. I'm not really sure if this is universal, but it could be. I'm trying to figure out if this is an all the time thing or a sometimes thing.

### Steps to Reproduce

index docs with `_doc_count`. run a terms agg on a field with a cardinality in the ball park of 500.

### Logs (if relevant)

_No response_
server/src/main/java/org/elasticsearch/search/aggregations/support/AggregationContext.java
) throws IOException;
    /**
     * Lookup the context for an already resolved field type.
     */
    public final FieldContext buildFieldContext(MappedFieldType ft) {
        return new FieldContext(ft.name(), buildFieldData(ft), ft);
    }
    /**
     * Build field data.
     */
    protected abstract IndexFieldData<?> buildFieldData(MappedFieldType ft);
    /**
     * Lookup a {@link MappedFieldType} by path.
     */
    public abstract MappedFieldType getFieldType(String path);
    /**
     * Returns a set of field names that match a regex-like pattern
     * All field names in the returned set are guaranteed to resolve to a field
     */
    public abstract Set<String> getMatchingFieldNames(String pattern);
    /**
     * Returns true if the field identified by the provided name is mapped, false otherwise
     */
    public abstract boolean isFieldMapped(String field);
    /**
     * Compile a script.
     */
    public abstract <FactoryType> FactoryType compile(Script script, ScriptContext<FactoryType> context);
    /**
     * Fetch the shared {@link SearchLookup}.