The `S3RetryingInputStream` hides cases where S3 closes a connection partway through downloading a blob. By default it retries 3 times before failing the download. However, the number of failures tends to increase on larger blobs and often 3 failures is not enough to complete a multi-GB download if S3 is suffering from a cluster of failures as sometimes happens. Typically we make 10s-to-100s of MBs of progress between each failure even in this state.

I think we should increase the tenacity of `S3RetryingInputStream` when downloading larger blobs. For instance, we could not count a partial download towards the retry limit if it makes significant progress before failing.
test/framework/src/main/java/org/elasticsearch/repositories/blobstore/AbstractBlobContainerRetriesTestCase.java
);
    }
    public void testReadBlobWithPrematureConnectionClose() {
        final int maxRetries = randomInt(20);
        final BlobContainer blobContainer = createBlobContainer(maxRetries, null, null, null);
        
        final byte[] bytes = randomBlobContent(1);
        httpServer.createContext(downloadStorageEndpoint(blobContainer, "read_blob_incomplete"), exchange -> {
            sendIncompleteContent(exchange, bytes);
            exchange.close();
        });
        final Exception exception = expectThrows(Exception.class, () -> {
            try (
                InputStream stream = randomBoolean()
                    ? blobContainer.readBlob("read_blob_incomplete", 0, 1)
                    : blobContainer.readBlob("read_blob_incomplete")
            ) {
                Streams.readFully(stream);
            }
        });
        assertThat(
            exception.getMessage().toLowerCase(Locale.ROOT),
            either(containsString("premature end of chunk coded message body: closing chunk expected")).or(
                containsString("premature end of content-length delimited message body")
            ).or(containsString("connection closed prematurely"))
        );
        assertThat(exception.getSuppressed().length, equalTo(Math.min(10, maxRetries)));
    }
    protected static byte[] randomBlobContent() {
        return randomBlobContent(1);