We limit the number of concurrent blob downloads per node and per recovery, but if a user configures too many concurrent recoveries they may run out of per-node download permits which yields the following message:

> Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]

The `[25]` here is wrong, the 25 permits are shared across all inbound recoveries so we should divide the suggested value by `cluster.routing.allocation.node_concurrent_incoming_recoveries`. In extreme cases the user may have configured so many concurrent recoveries that the only reasonable limit is zero, and in that case we should suggest setting `indices.recovery.use_snapshots: false` instead.


server/src/test/java/org/elasticsearch/indices/recovery/RecoverySettingsTests.java
.build()
                .getMaxBytesPerSec(),
            equalTo(DEFAULT_MAX_BYTES_PER_SEC)
        );
    }
    public void testMaxBytesPerSecOnDataNodeWithIndicesRecoveryMaxBytesPerSec() {
        final ByteSizeValue random = randomByteSizeValue();
        assertThat(
            "Data nodes that are not dedicated to cold/frozen should use the defined rate limit when set",
            nodeRecoverySettings().withIndicesRecoveryMaxBytesPerSec(random)
                .withRoles(randomDataNodeRoles())
                .withRandomMemory()
                .build()
                .getMaxBytesPerSec(),
            equalTo(random)
        );
    }
    public void testMaxBytesPerSecOnDataNodeWithIndicesRecoveryMaxBytesPerSecAndOvercommit() {
        final Double maxOvercommitFactor = randomBoolean() ? randomDoubleBetween(1.0d, 100.0d, true) : null;
        final ByteSizeValue indicesRecoveryMaxBytesPerSec = switch (randomInt(2)) {
            case 0 -> ByteSizeValue.MINUS_ONE;
            case 1 -> ByteSizeValue.ZERO;
            case 2 -> ByteSizeValue.ofGb(between(100, 1000));
            default -> throw new AssertionError();
        };
        RecoverySettings recoverySettings = nodeRecoverySettings().withIndicesRecoveryMaxBytesPerSec(indicesRecoveryMaxBytesPerSec)
            .withNetworkBandwidth(ByteSizeValue.ofGb(1))
            .withDiskReadBandwidth(ByteSizeValue.ofMb(500))
            .withDiskWriteBandwidth(ByteSizeValue.ofMb(250))
            .withMaxOvercommitFactor(maxOvercommitFactor)