### Elasticsearch Version

8.6.2

### Installed Plugins

_No response_

### Java Version

_bundled_

### OS Version

Elasticsearch Service (ESS)

### Problem Description

Rate limit for snapshot speed is not tied to the recovery rate limit computed from `node.bandwidth.recovery*` settings.

This happens irregardless of the data tiers where the data is located. 

This is likely a bug introduced in https://github.com/elastic/elasticsearch/pull/91021 . **Issue has been observed by users and it is reproducible.**


Tagging @tlrx @kingherc as we have internally discussed about this.

### Steps to Reproduce

1) Create a deployment in Elasticsearch Service (ESS):
- Hardware profile: Storage optimized (dense)
- Hot tier: 60GB RAM x 1 availability zone (AZ)
- Warm tier: 60GB RAM x 1 availability zone (AZ)

Once the deployment is created, check the `node.bandwidth.recovery*` settings and check the Elasticsearch logs for the computed rate limit.

From the [nodes info API](https://www.elastic.co/guide/en/elasticsearch/reference/8.6/cluster-nodes-info.html) output, extract the `node.bandwidth.recovery` setting using `jq`:

```
jq '.nodes[]|{name:.name, instance_configuration:.attributes.instance_configuration, bandwidth:.settings.node.bandwidth}' nodes.json
```

```
{
  "name": "instance-0000000000",
  "instance_configuration": "aws.es.datahot.i3en",
  "bandwidth": {
    "recovery": {
      "disk": {
        "write": "636200Kb",
        "read": "1249280Kb"
      },
      "network": "1018060Kb"
    }
  }
}
{
  "name": "instance-0000000001",
  "instance_configuration": "aws.es.datawarm.d3",
  "bandwidth": {
    "recovery": {
      "disk": {
        "write": "954255Kb",
        "read": "293795Kb"
      },
      "network": "727541Kb"
    }
  }
}
```

In the Elasticsearch logs, we can find the computed rate limit for the two nodes:

```
[2023-04-26T00:14:00,061][INFO ][org.elasticsearch.indices.recovery.RecoverySettings] [instance-0000000000] using rate limit [248.5mb] with [default=40mb, read=397.6mb, write=248.5mb, max=60.6gb]
[2023-04-26T00:14:00,866][INFO ][org.elasticsearch.indices.recovery.RecoverySettings] [instance-0000000001] using rate limit [114.7mb] with [default=40mb, read=114.7mb, write=284.1mb, max=28gb]
```

2) Stop SLM (`POST _slm/stop`)

3) Using [rally](https://github.com/elastic/rally), run the `nyc_taxis` track with the `append-no-conflicts-index-only` challenge to create the `nyc_taxis` index, which should be about 20-25GB in size.

4) Execute the SLM policy to take a snapshot:

```
PUT _slm/policy/cloud-snapshot-policy/_execute
```

5) Wait for the snapshot to complete:

```
[2023-04-26T01:13:46,029][INFO ][org.elasticsearch.xpack.slm.SnapshotLifecycleTask] [instance-0000000000] snapshot lifecycle policy [cloud-snapshot-policy] issuing create snapshot [cloud-snapshot-2023.04.26-4olr-h_lq_63ljveqiwr2q]
[2023-04-26T01:13:46,055][INFO ][org.elasticsearch.snapshots.SnapshotsService] [instance-0000000000] snapshot [found-snapshots:cloud-snapshot-2023.04.26-4olr-h_lq_63ljveqiwr2q/VLurh60aT_qeeSlsEGcWZg] started
[2023-04-26T01:24:14,821][INFO ][org.elasticsearch.snapshots.SnapshotsService] [instance-0000000000] snapshot [found-snapshots:cloud-snapshot-2023.04.26-4olr-h_lq_63ljveqiwr2q/VLurh60aT_qeeSlsEGcWZg] completed with state [SUCCESS]
```

Observe the host network inbound/outbound utilisation. While the snapshot was taken, we can see that the snapshot speed was limited to ~ 40mb/s:

<img width="2526" alt="snapshot-in-hot-tier-default-settings" src="https://user-images.githubusercontent.com/51113389/234481515-32b1c9a3-23b5-4e34-8d2b-4d8d23b7c702.png">

**This is not the expected behaviour.**

6) Delete the snapshot

7) Update the snapshot repository `found-snapshots` with `max_snapshot_bytes_per_sec` to `248mb` (to roughly match the expected rate limit in the hot data tier)

8) Execute the SLM policy to take a snapshot

9) Wait for the snapshot to complete:

```
[2023-04-26T01:46:44,787][INFO ][org.elasticsearch.xpack.slm.SnapshotLifecycleTask] [instance-0000000000] snapshot lifecycle policy [cloud-snapshot-policy] issuing create snapshot [cloud-snapshot-2023.04.26-mwgisfissvq1uvorxigoyg]
[2023-04-26T01:46:44,797][INFO ][org.elasticsearch.snapshots.SnapshotsService] [instance-0000000000] snapshot [found-snapshots:cloud-snapshot-2023.04.26-mwgisfissvq1uvorxigoyg/3rjPoMI-Ti6pXJStt24GNg] started
[2023-04-26T01:48:28,709][INFO ][org.elasticsearch.snapshots.SnapshotsService] [instance-0000000000] snapshot [found-snapshots:cloud-snapshot-2023.04.26-mwgisfissvq1uvorxigoyg/3rjPoMI-Ti6pXJStt24GNg] completed with state [SUCCESS]
```

Observe the host network inbound/outbound utilisation. While the snapshot was taken, we can see that the snapshot speed was up to ~ 250mb/s:

<img width="2533" alt="snapshot-in-hot-tier-with-custom-snapshot-speed" src="https://user-images.githubusercontent.com/51113389/234487404-1054056a-7d3e-4816-8b38-6e2bec67ed3d.png">

10) Delete the snapshot and remove the `max_snapshot_bytes_per_sec` setting from the snapshot repository configuration

11) Update the `_tier_preference` for `nyc_taxis` index so that the index moves to the warm data tier:

```
PUT nyc_taxis/_settings
{
  "index": {
    "routing": {
      "allocation": {
        "include": {
          "_tier_preference": "data_warm,data_hot"
        }
      }
    }
  }
}
```

12) Wait for the primary shard to be moved to the warm data tier

13) Execute the SLM policy to take a snapshot

14) Wait for the snapshot to complete:

```
[2023-04-26T02:19:25,635][INFO ][org.elasticsearch.xpack.slm.SnapshotLifecycleTask] [instance-0000000000] snapshot lifecycle policy [cloud-snapshot-policy] issuing create snapshot [cloud-snapshot-2023.04.26-8qo6hzwksmqi_6dnd7m_hq]
[2023-04-26T02:19:25,650][INFO ][org.elasticsearch.snapshots.SnapshotsService] [instance-0000000000] snapshot [found-snapshots:cloud-snapshot-2023.04.26-8qo6hzwksmqi_6dnd7m_hq/kioCVM4YSpii7lK3iRVC6g] started
[2023-04-26T02:29:53,481][INFO ][org.elasticsearch.snapshots.SnapshotsService] [instance-0000000000] snapshot [found-snapshots:cloud-snapshot-2023.04.26-8qo6hzwksmqi_6dnd7m_hq/kioCVM4YSpii7lK3iRVC6g] completed with state [SUCCESS]
```

Observe the host network inbound/outbound utilisation. While the snapshot was taken, we can see that the snapshot speed was limited to ~ 40mb/s:

<img width="2537" alt="snapshot-in-warm-tier-default-settings" src="https://user-images.githubusercontent.com/51113389/234488629-777a377a-fd99-4464-aae8-89e13cbf4e47.png">

**This is not the expected behaviour**

### Logs (if relevant)

_No response_
server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java
changedSettingNames.addAll(updatedSettings.keySet());
        changedSettingNames.removeAll(ignoredSettings);
        changedSettingNames.removeIf(setting -> Objects.equals(.get(setting), updatedSettings.get(setting)));
        changedSettingNames.removeAll(DYNAMIC_SETTING_NAMES);
        return changedSettingNames.isEmpty();
    }
    // 
    // 
    @Override
    public void updateState(ClusterState state) {
        final Settings previousSettings = metadata.settings();
        metadata = getRepoMetadata(state);
        final Settings updatedSettings = metadata.settings();
        if (updatedSettings.equals(previousSettings) == false) {
            snapshotRateLimiter = getSnapshotRateLimiter();
            restoreRateLimiter = getRestoreRateLimiter();
        }
        uncleanStart = uncleanStart && metadata.generation() != metadata.pendingGeneration();
        final boolean wasBestEftConsistency = bestEftConsistency;
        bestEftConsistency = uncleanStart || isReadOnly() || metadata.generation() == RepositoryData.UNKNOWN_REPO_GEN;
        if (isReadOnly()) {
            // 
            return;
        }
        if (bestEftConsistency) {
            final SnapshotsInProgress snapshotsInProgress = state.custom(SnapshotsInProgress.TYPE, SnapshotsInProgress.EMPTY);
            long bestGenerationFromCS = bestGeneration(snapshotsInProgress.Repo(this.metadata.name()));
            // 
            // 
            //