The `recoverLocallyUpToGlobalCheckpoint` step of peer recovery may take an extended period of time, but today we do not disable the recovery monitor during this step so if it takes more than 30 minutes (by default) then the recovery will time out, fail, and retry repeatedly. We should disable the recovery monitor during this step.
server/src/main/java/org/elasticsearch/indices/recovery/RecoveryTarget.java
public void setLastAccessTime() {
        lastAccessTime = System.nanoTime();
    }
    /**
     * Set flag to signal to {@link org.elasticsearch.indices.recovery.RecoveriesCollection.RecoveryMonitor} that it must not cancel this
     * recovery temporarily. This is used by the recovery clean files step to avoid recovery failure in case a long running condition was
     * added to the shard via {@link IndexShard#addCleanFilesDependency()}.
     *
     * @return releasable that once closed will re-enable liveness checks by the recovery monitor
     */
    public Releasable disableRecoveryMonitor() {
        assert recoveryMonitorEnabled : "recovery monitor already disabled";
        recoveryMonitorEnabled = false;
        return () -> {
            setLastAccessTime();
            recoveryMonitorEnabled = true;
        };
    }
    public Store store() {
        ensureRefCount();
        return store;
    }
    /**
     * Closes the current recovery target and waits up to a certain timeout for resources to be freed.
     * Returns true if resetting the recovery was successful, false if the recovery target is already cancelled / failed or marked as done.
     */
    boolean resetRecovery(CancellableThreads newTargetCancellableThreads) throws IOException {
        if (finished.compareAndSet(false, true)) {
            try {
                logger.debug("reset of recovery with shard {} and id [{}]", shardId, recoveryId);