<!-- Describe the issue -->

Hi everyone,
We use bitcoin core with hundreds of wallets. The only efficient way we have found to do so is :
1. wait for any bitcoin-cli command to terminate before issuing another one
2. never issue any bitcoin-cli command unless tip update is 1
3. always unload a wallet after loading and using it

Now, we observ that bitcoin core systematically crashes every 20-30 minutes, then automatically restarts. And the problem is, it has to re-sync from about 0.997, which easily takes 15-20 minutes. During that time, the daemon cannot be used.

If we don't load/unload wallets, the daemon is stable and keeps to 1, no issue.

**System information**
Bitcoin Core RPC client version v0.20.0.0-ga62f0ed64f8bbbdfe6467ac5ce92ef5b5222d1bd
Compiled and run on Debian 9, on a VM with 16 mo memory. Launched that way : /bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1

**bitcoin.conf :**
server=1
dbcache=16384
par=1
blocksonly=1
maxuploadtarget=137
maxconnections=16
rpcuser=bitcoinrpc
rpcpassword=xxxxxx
rpcallowip=127.0.0.1
testnet=0
txindex=1
walletbroadcast=1
txconfirmtarget=1

Please note that the same behaviour was happening with older releases (from 0.17.x).

Thanks for any help. Please ask for any needed information / testing.
Regards,
{
    int64_t nNow = GetTime();
    int64_t start_time = GetTimeMillis();
    assert(reserver.isReserved());
    uint256 block_hash = start_block;
    ScanResult result;
    WalletLogPrintf("Rescan started from block %s...\n", start_block.ToString());
    fAbortRescan = false;
    ShowProgress(strprintf("%s " + _("Rescanning...").translated, GetDisplayName()), 0); 
    uint256 tip_hash = WITH_LOCK(cs_wallet, return GetLastBlockHash());
    uint256 end_hash = tip_hash;
    if (max_height) chain().findAncestorByHeight(tip_hash, *max_height, FoundBlock().hash(end_hash));
    double progress_begin = chain().guessVerificationProgress(block_hash);
    double progress_end = chain().guessVerificationProgress(end_hash);
    double progress_current = progress_begin;
    int block_height = start_height;
    while (!fAbortRescan && !chain().shutdownRequested()) {
        m_scanning_progress = (progress_current - progress_begin) / (progress_end - progress_begin);
        if (block_height % 100 == 0 && progress_end - progress_begin > 0.0) {
            ShowProgress(strprintf("%s " + _("Rescanning...").translated, GetDisplayName()), std::max(1, std::min(99, (int)(m_scanning_progress * 100))));
        }
        if (GetTime() >= nNow + 60) {
            nNow = GetTime();
            WalletLogPrintf("Still rescanning. At block %d. Progress=%f\n", block_height, progress_current);
        }
        CBlock block;
        bool next_block;
        uint256 next_block_hash;