<!-- Describe the issue -->

Hi everyone,
We use bitcoin core with hundreds of wallets. The only efficient way we have found to do so is :
1. wait for any bitcoin-cli command to terminate before issuing another one
2. never issue any bitcoin-cli command unless tip update is 1
3. always unload a wallet after loading and using it

Now, we observ that bitcoin core systematically crashes every 20-30 minutes, then automatically restarts. And the problem is, it has to re-sync from about 0.997, which easily takes 15-20 minutes. During that time, the daemon cannot be used.

If we don't load/unload wallets, the daemon is stable and keeps to 1, no issue.

**System information**
Bitcoin Core RPC client version v0.20.0.0-ga62f0ed64f8bbbdfe6467ac5ce92ef5b5222d1bd
Compiled and run on Debian 9, on a VM with 16 mo memory. Launched that way : /bitcoind -daemon -walletrbf -conf=/root/.bitcoin/bitcoin.conf -fallbackfee=0.00001 -rpcworkqueue=128 -txindex=1

**bitcoin.conf :**
server=1
dbcache=16384
par=1
blocksonly=1
maxuploadtarget=137
maxconnections=16
rpcuser=bitcoinrpc
rpcpassword=xxxxxx
rpcallowip=127.0.0.1
testnet=0
txindex=1
walletbroadcast=1
txconfirmtarget=1

Please note that the same behaviour was happening with older releases (from 0.17.x).

Thanks for any help. Please ask for any needed information / testing.
Regards,
if (effective_value > 0) {
                    group.fee += coin.m_input_bytes < 0 ? 0 : coin_selection_params.effective_fee.GetFee(coin.m_input_bytes);
                    group.long_term_fee += coin.m_input_bytes < 0 ? 0 : long_term_feerate.GetFee(coin.m_input_bytes);
                    if (coin_selection_params.m_subtract_fee_outputs) {
                        group.effective_value += coin.txout.nValue;
                    } else {
                        group.effective_value += effective_value;
                    }
                    ++it;
                } else {
                    it = group.Discard(coin);
                }
            }
            if (group.effective_value > 0) utxo_pool.push_back(group);
        }
        
        CAmount not_input_fees = coin_selection_params.effective_fee.GetFee(coin_selection_params.tx_noinputs_size);
        bnb_used = true;
        return SelectCoinsBnB(utxo_pool, nTargetValue, cost_of_change, setCoinsRet, nValueRet, not_input_fees);
    } else {
        
        for (const OutputGroup& group : groups) {
            if (!group.EligibleForSpending(eligibility_filter)) continue;
            utxo_pool.push_back(group);
        }
        bnb_used = false;
        return KnapsackSolver(nTargetValue, utxo_pool, setCoinsRet, nValueRet);
    }
}
bool CWallet::SelectCoins(const std::vector<COutput>& vAvailableCoins, const CAmount& nTargetValue, std::set<CInputCoin>& setCoinsRet, CAmount& nValueRet, const CCoinControl& coin_control, CoinSelectionParams& coin_selection_params, bool& bnb_used) const