The `S3RetryingInputStream` hides cases where S3 closes a connection partway through downloading a blob. By default it retries 3 times before failing the download. However, the number of failures tends to increase on larger blobs and often 3 failures is not enough to complete a multi-GB download if S3 is suffering from a cluster of failures as sometimes happens. Typically we make 10s-to-100s of MBs of progress between each failure even in this state.

I think we should increase the tenacity of `S3RetryingInputStream` when downloading larger blobs. For instance, we could not count a partial download towards the retry limit if it makes significant progress before failing.
modules/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3RetryingInputStream.java
import java.nio.file.NoSuchFileException;
import java.util.ArrayList;
import java.util.List;
import static org.elasticsearch.core.Strings.format;
/**
 * Wrapper around an S3 object that will retry the {@link GetObjectRequest} if the download fails part-way through, resuming from where
 * the failure occurred. This should be handled by the SDK but it isn't today. This should be revisited in the future (e.g. before removing
 * the {@link Version#V_7_0_0} version constant) and removed when the SDK handles retries itself.
 *
 * See https://github.com/aws/aws-sdk-java/issues/856 for the related SDK issue
 */
class S3RetryingInputStream extends InputStream {
    private static final Logger logger = LogManager.getLogger(S3RetryingInputStream.class);
    static final int MAX_SUPPRESSED_EXCEPTIONS = 10;
    private final S3BlobStore blobStore;
    private final String blobKey;
    private final long start;
    private final long end;
    private final int maxAttempts;
    private final List<IOException> failures;
    private S3ObjectInputStream currentStream;
    private long currentStreamLastOffset;
    private int attempt = 1;
    private long currentOffset;
    private boolean closed;
    private boolean eof;
    S3RetryingInputStream(S3BlobStore blobStore, String blobKey) throws IOException {
        this(blobStore, blobKey, 0, Long.MAX_VALUE - 1);
    }