<!-- This issue tracker is only for technical issues related to Bitcoin Core.

General bitcoin questions and/or support requests are best directed to the Bitcoin StackExchange at https://bitcoin.stackexchange.com.

For reporting security issues, please read instructions at https://bitcoincore.org/en/contact/.

If the node is "stuck" during sync or giving "block checksum mismatch" errors, please ensure your hardware is stable by running memtest and observe CPU temperature with a load-test tool such as linpack before creating an issue! -->

<!-- Describe the issue -->
Have two wallets used by this instance of bitcoind. The default wallet has balance 0.02861762.
Initially reported balance is correct but after unloading the wallet and then loading it again, the balance is wrong. Neither of the wallets have the reported balance.
<!--- What behavior did you expect? -->
I expected the balance to be the same as initially reported.
```
btc@ubuntu:~$ bitcoin-cli  -getinfo
{
  "version": 170100,
  "protocolversion": 70015,
  "walletversion": 159900,
  "balance": 0.02861762,
...

btc@ubuntu:~$ bitcoin-cli  loadwallet wallet2
btc@ubuntu:~$ bitcoin-cli unloadwallet ""
btc@ubuntu:~$ bitcoin-cli loadwallet ""
btc@ubuntu:~$ bitcoin-cli unloadwallet wallet2
btc@ubuntu:~$ bitcoin-cli  -getinfo
{
  "version": 170100,
  "protocolversion": 70015,
  "walletversion": 159900,
  "balance": 0.01012607,
...
```
The balance 0.01012607 is wrong and I have no idea where that number came from.
Restarting the process results in correct balance again.
<!--- What was the actual behavior (provide screenshots if the issue is GUI-related)? -->

<!--- How reliably can you reproduce the issue, what are the steps to do so? -->

<!-- What version of Bitcoin Core are you using, where did you get it (website, self-compiled, etc)? -->
ubuntu package for bionic
<!-- What type of machine are you observing the error on (OS/CPU and disk type)? -->

<!-- For the GUI-related issue on Linux provide names and versions of a distro, a desktop environment and a graphical shell (if relevant). -->

<!-- Any extra information that might be useful in the debugging process. -->
<!--- This is normally the contents of a `debug.log` or `config.log` file. Raw text or a link to a pastebin type site are preferred. -->

* notifications happen and cached balances are marked dirty.
     *
     * If fUpdate is true, existing transactions will be updated.
     * TODO: One exception to this is that the abandoned state is cleared under the
     * assumption that any further notification of a transaction that was considered
     * abandoned is an indication that it is not safe to be considered abandoned.
     * Abandoned state should probably be more carefully tracked via different
     * posInBlock signals or by checking mempool presence when necessary.
     */
    bool AddToWalletIfInvolvingMe(const CTransactionRef& tx, const uint256& block_hash, int posInBlock, bool fUpdate) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);
    
    void MarkConflicted(const uint256& hashBlock, const uint256& hashTx);
    
    void MarkInputsDirty(const CTransactionRef& tx) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);
    void SyncMetaData(std::pair<TxSpends::iterator, TxSpends::iterator>) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);
    /* Used by TransactionAddedToMemorypool/BlockConnected/Disconnected/ScanForWalletTransactions.
     * Should be called with non-zero block_hash and posInBlock if this is for a transaction that is included in a block. */
    void SyncTransaction(const CTransactionRef& tx, const uint256& block_hash, int posInBlock = 0, bool update_tx = true) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);
    
    CHDChain hdChain;
    
    void DeriveNewChildKey(WalletBatch& batch, CKeyMetadata& metadata, CKey& secret, bool internal = false) EXCLUSIVE_LOCKS_REQUIRED(cs_wallet);
    std::set<int64_t> setInternalKeyPool GUARDED_BY(cs_wallet);
    std::set<int64_t> setExternalKeyPool GUARDED_BY(cs_wallet);
    std::set<int64_t> set_pre_split_keypool GUARDED_BY(cs_wallet);
    int64_t m_max_keypool_index GUARDED_BY(cs_wallet) = 0;
    std::map<CKeyID, int64_t> m_pool_key_to_index;
    std::atomic<uint64_t> m_wallet_flags{0};
    int64_t nTimeFirstKey GUARDED_BY(cs_wallet) = 0;
    /**