Searching a large number of unavailable shards through e.g. the `*` pattern while a large cluster is recovering from a full restart or so, leads to extremely large responses containing an exception for each shard.


![](https://user-images.githubusercontent.com/6490959/193701098-868888bc-cdc7-4104-a32a-d9c84d74aef2.png)

This example shows a 375M on heap response for ~25k unavailable shards being returned. Over the wire, this serialises with a similar size and we'd have 700M+ peak heap usage for a search request when the valid search response in a green cluster might be much smaller than this.
It seems to me that we could mostly resolve this by not returning the stack trace in this specific case of the shard not available exception (it doesn't seem valuable for users and we made a similar fix around unavailable shards in snapshots state responses)? 
server/src/main/java/org/elasticsearch/action/search/AbstractSearchAsyncAction.java
final int totalOps = this.totalOps.incrementAndGet();
        if (totalOps == expectedTotalOps) {
            onPhaseDone();
        } else if (totalOps > expectedTotalOps) {
            throw new AssertionError(
                "unexpected higher total ops [" + totalOps + "] compared to expected [" + expectedTotalOps + "]",
                new SearchPhaseExecutionException(getName(), "Shard failures", null, buildShardFailures())
            );
        } else {
            if (lastShard == false) {
                performPhaseOnShard(shardIndex, shardIt, nextShard);
            }
        }
    }
    /**
     * Executed once for every {@link ShardId} that failed on all available shard routing.
     *
     * @param shardIndex the shard index that failed
     * @param shardTarget the last shard target for this failure
     * @param exc the last failure reason
     */
    protected void onShardGroupFailure(int shardIndex, SearchShardTarget shardTarget, Exception exc) {}
    /**
     * Executed once for every failed shard level request. This method is invoked before the next replica is tried for the given
     * shard target.
     * @param shardIndex the internal index for this shard. Each shard has an index / ordinal assigned that is used to reference
     *                   it's results
     * @param shardTarget the shard target for this failure
     * @param e the failure reason
     */