<!-- Describe the issue -->

Hi everyone,
We use bitcoin core with hundreds of wallets. The only efficient way we have found to do so is :
1. wait for any bitcoin-cli command to terminate before issuing another one
2. never issue any bitcoin-cli command unless tip update is 1
3. always unload a wallet after loading and using it

Now, we observ that bitcoin core systematically crashes every 20-30 minutes, then automatically restarts. And the problem is, it has to re-sync from about 0.997, which easily takes 15-20 minutes. During that time, the daemon cannot be used.

If we don't load/unload wallets, the daemon is stable and keeps to 1, no issue.

**System information**
Bitcoin Core RPC client version v0.20.0.0-ga62f0ed64f8bbbdfe6467ac5ce92ef5b5222d1bd
Compiled and run on Debian 9, on a VM with 16 mo memory. Launched that way : /bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1

**bitcoin.conf :**
server=1
dbcache=16384
par=1
blocksonly=1
maxuploadtarget=137
maxconnections=16
rpcuser=bitcoinrpc
rpcpassword=xxxxxx
rpcallowip=127.0.0.1
testnet=0
txindex=1
walletbroadcast=1
txconfirmtarget=1

Please note that the same behaviour was happening with older releases (from 0.17.x).

Thanks for any help. Please ask for any needed information / testing.
Regards,
PKHash pkh_dest(keyid);
                if (GetDestData(pkh_dest, "used", nullptr)) {
                    return true;
                }
            }
        }
    }
    return false;
}
CWalletTx* CWallet::AddToWallet(CTransactionRef tx, const CWalletTx::Confirmation& confirm, const UpdateWalletTxFn& update_wtx, bool fFlushOnClose)
{
    LOCK(cs_wallet);
    WalletBatch batch(*database, "r+", fFlushOnClose);
    uint256 hash = tx->GetHash();
    if (IsWalletFlagSet(WALLET_FLAG_AVOID_REUSE)) {
        
        std::set<CTxDestination> tx_destinations;
        for (const CTxIn& txin : tx->vin) {
            const COutPoint& op = txin.prevout;
            SetSpentKeyState(batch, op.hash, op.n, true, tx_destinations);
        }
        MarkDestinationsDirty(tx_destinations);
    }
    
    auto ret = mapWallet.emplace(std::piecewise_construct, std::forward_as_tuple(hash), std::forward_as_tuple(this, tx));
    CWalletTx& wtx = (*ret.first).second;
    bool fInsertedNew = ret.second;
    bool fUpdated = update_wtx && update_wtx(wtx, fInsertedNew);
    if (fInsertedNew) {
        wtx.m_confirm = confirm;