The `S3RetryingInputStream` hides cases where S3 closes a connection partway through downloading a blob. By default it retries 3 times before failing the download. However, the number of failures tends to increase on larger blobs and often 3 failures is not enough to complete a multi-GB download if S3 is suffering from a cluster of failures as sometimes happens. Typically we make 10s-to-100s of MBs of progress between each failure even in this state.

I think we should increase the tenacity of `S3RetryingInputStream` when downloading larger blobs. For instance, we could not count a partial download towards the retry limit if it makes significant progress before failing.
test/framework/src/main/java/org/elasticsearch/repositories/blobstore/AbstractBlobContainerRetriesTestCase.java
public abstract class AbstractBlobContainerRetriesTestCase extends ESTestCase {
    private static final long MAX_RANGE_VAL = Long.MAX_VALUE - 1;
    protected HttpServer httpServer;
    @Before
    public void setUp() throws Exception {
        httpServer = MockHttpServer.createHttp(new InetSocketAddress(InetAddress.getLoopbackAddress(), 0), 0);
        httpServer.start();
        super.setUp();
    }
    @After
    public void tearDown() throws Exception {
        httpServer.stop(0);
        super.tearDown();
    }
    protected abstract String downloadStorageEndpoint(BlobContainer container, String blob);
    protected abstract String bytesContentType();
    protected abstract Class<? extends Exception> unresponsiveExceptionType();