<!-- Describe the issue -->

Hi everyone,
We use bitcoin core with hundreds of wallets. The only efficient way we have found to do so is :
1. wait for any bitcoin-cli command to terminate before issuing another one
2. never issue any bitcoin-cli command unless tip update is 1
3. always unload a wallet after loading and using it

Now, we observ that bitcoin core systematically crashes every 20-30 minutes, then automatically restarts. And the problem is, it has to re-sync from about 0.997, which easily takes 15-20 minutes. During that time, the daemon cannot be used.

If we don't load/unload wallets, the daemon is stable and keeps to 1, no issue.

**System information**
Bitcoin Core RPC client version v0.20.0.0-ga62f0ed64f8bbbdfe6467ac5ce92ef5b5222d1bd
Compiled and run on Debian 9, on a VM with 16 mo memory. Launched that way : /bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1

**bitcoin.conf :**
server=1
dbcache=16384
par=1
blocksonly=1
maxuploadtarget=137
maxconnections=16
rpcuser=bitcoinrpc
rpcpassword=xxxxxx
rpcallowip=127.0.0.1
testnet=0
txindex=1
walletbroadcast=1
txconfirmtarget=1

Please note that the same behaviour was happening with older releases (from 0.17.x).

Thanks for any help. Please ask for any needed information / testing.
Regards,
#include <util/string.h>
#include <util/translation.h>
#include <wallet/coincontrol.h>
#include <wallet/fees.h>
#include <algorithm>
#include <assert.h>
#include <boost/algorithm/string/replace.hpp>
using interfaces::FoundBlock;
const std::map<uint64_t,std::string> WALLET_FLAG_CAVEATS{
    {WALLET_FLAG_AVOID_REUSE,
        "You need to rescan the blockchain in order to correctly mark used "
        "destinations in the past. Until this is done, some destinations may "
        "be considered unused, even if the opposite is the case."
    },
};
static const size_t OUTPUT_GROUP_MAX_ENTRIES = 10;
static RecursiveMutex cs_wallets;
static std::vector<std::shared_ptr<CWallet>> vpwallets GUARDED_BY(cs_wallets);
static std::list<LoadWalletFn> g_load_wallet_fns GUARDED_BY(cs_wallets);
bool AddWallet(const std::shared_ptr<CWallet>& wallet)
{
    LOCK(cs_wallets);
    assert(wallet);
    std::vector<std::shared_ptr<CWallet>>::const_iterator i = std::find(vpwallets.begin(), vpwallets.end(), wallet);
    if (i != vpwallets.end()) return false;
    vpwallets.push_back(wallet);
    wallet->ConnectScriptPubKeyManNotifiers();
    return true;
}
bool RemoveWallet(const std::shared_ptr<CWallet>& wallet)