We limit the number of concurrent blob downloads per node and per recovery, but if a user configures too many concurrent recoveries they may run out of per-node download permits which yields the following message:

> Unable to acquire permit to use snapshot files during recovery, this recovery will recover index files from the source node. Ensure snapshot files can be used during recovery by setting [indices.recovery.max_concurrent_snapshot_file_downloads] to be no greater than [25]

The `[25]` here is wrong, the 25 permits are shared across all inbound recoveries so we should divide the suggested value by `cluster.routing.allocation.node_concurrent_incoming_recoveries`. In extreme cases the user may have configured so many concurrent recoveries that the only reasonable limit is zero, and in that case we should suggest setting `indices.recovery.use_snapshots: false` instead.


server/src/test/java/org/elasticsearch/indices/recovery/RecoverySettingsTests.java
equalTo(new ByteSizeValue(90, ByteSizeUnit.MB))
            );
        }
        {
            assertThat(
                "Dedicated cold/frozen data nodes with 16GB < RAM <= 32GB have a default 90mb rate limit",
                nodeRecoverySettings().withRoles(dataRoles)
                    .withMemory(ByteSizeValue.ofBytes(randomLongBetween(ByteSizeUnit.GB.toBytes(16L) + 1L, ByteSizeUnit.GB.toBytes(32L))))
                    .build()
                    .getMaxBytesPerSec(),
                equalTo(new ByteSizeValue(125, ByteSizeUnit.MB))
            );
        }
        {
            assertThat(
                "Dedicated cold/frozen data nodes with RAM > 32GB have a default 250mb rate limit",
                nodeRecoverySettings().withRoles(dataRoles)
                    .withMemory(ByteSizeValue.ofBytes(randomLongBetween(ByteSizeUnit.GB.toBytes(32L) + 1L, ByteSizeUnit.TB.toBytes(4L))))
                    .build()
                    .getMaxBytesPerSec(),
                equalTo(new ByteSizeValue(250, ByteSizeUnit.MB))
            );
        }
    }
    public void testMaxBytesPerSecOnColdOrFrozenNodeWithIndicesRecoveryMaxBytesPerSec() {
        final ByteSizeValue random = randomByteSizeValue();
        assertThat(
            "Dedicated cold/frozen data nodes should use the defined rate limit when set",
            nodeRecoverySettings().withRoles(randomFrom(Set.of("data_cold"), Set.of("data_frozen"), Set.of("data_cold", "data_frozen")))
                .withMemory(ByteSizeValue.ofBytes(randomLongBetween(1L, ByteSizeUnit.TB.toBytes(4L))))