\section{Introduction}
Reverse engineering (RE) the brain plays an important role in neuroscience. To do so, neuroscientists record the activity of neurons and try to reverse engineer their relationship using machine learning (ML) and statistical methods (e.g. see \citet{pillow2008spatio,o2015computational}).
However, current technology allows us to record only a small subset of the neurons that participate in the solution of even simple tasks. Neuroscience, at least in the near future, needs to rely on RE the brain using partial recordings.

Even though it is not possible to simultaneously record all neurons involved in a task, it is possible to make multiple partial recordings potentially with different observed neurons in each recording. For example, multi-electrode measurements \cite{ballini20131024}, 2-photon calcium imaging \cite{kerr2008imaging,grienberger2012imaging} and optogenetics \cite{deisseroth2011optogenetics}, allow us to simultaneously observe a subsets of neurons. The optical methods even allow us to select which exact set of neurons we want to record. Therefore, it is natural to question if and how much integrating multiple partial recordings can compensate for the partial observation.

Based on these technological advances, a few studies have recently focused on dealing with sub-sampled observations of neural activity.
\citet{pillow2007neural} extend the linear-nonlinear Poisson (LNP) framework to include the activity of unmeasured (hidden) neurons to estimate connectivity patterns among observed and unobserved neurons. However, the method is limited to small numbers of unobserved neurons.
\citet{wohrer2010linear} propose recovering the full noise correlation matrix from partial electrophysiology recordings, based on fully observed signal correlations.
\citet{turaga2013inferring} use a latent dynamical system model to combine two non-simultaneously recorded but strongly interacting populations of neurons into one model, and show that combining partial observations improves the prediction of neural activity.
These studies show that there is a growing interest in combining partial observations to improve our understanding of the brain.

Quantifying the added benefit of combining partial information to elucidate neural function is hard, particularly in the absence of known ground truth. Recently, \citet{jonas2015automatic,jonas2017could} advocate using man-made systems, where the ground truth is fully accessible, to evaluate methods used for studying biological systems and identify their strengths and caveats. Based on this idea, in this paper, we explore how well we can reverse engineer the neural activation in an artificial neural network (ANN) based on partial observations of its hidden neurons. We see this as a step forward in answering the following questions: Is it meaningful to try and reconstruct a complex neural circuit when we only partially observe it? What is the best way of dealing with the missing information in partial recordings? Which of the ML methods is more successful for the task? And finally, what is the best strategy for combining partial observations, given that the sampling capacity is technically limited?

In addition, in light of the recent successes of ANNs in a wide range of ML problems, reverse-engineering an ANN has become an interesting problem in its own right. In particular, reconstructing part of a networks' input-output functions based on partial observations is related to the redundancy of the specific network. Redundancy in ANNs has already been discussed from different angles: for example, inspired by biological networks, \citet{Izui1990} show that redundant neural networks are more accurate, faster and more stable. \citet{denil2013predicting} show that there is significant redundancy in the parameterization of several deep learning models and that based on a few parameters of the model it is possible to accurately predict the remaining values. From an ML perspective, this redundancy is interesting because it allows the compression of the network. \citet{Cheng2015ICCV} exploit the redundancy in ANNs to reduce memory footprint. Techniques based on knowledge distillation \cite{hinton2014} compress the knowledge of a network into a more compact model, which is trained to predict the soft outputs of the larger model. Being able to understand neural networks based on "recordings" is important both in biology and in machine learning.

There are many definitions of "understanding" but one near universally accepted aspect of understanding is the ability to make predictions. One simple way of quantifying this is the root mean squared error (RMSE) of the input/output (I/O) prediction. For the approximation of the I/O function of a system, ML provides a good number of standard approaches to supervised learning. Popular techniques include XGBoost and ANNs. In this view, being able to fit an ML algorithm to a system's I/O function that allows low RMSE is the definition of understanding the system.

In this work, we train ANNs on the MNIST dataset. We use these ANNs as the ground-truth networks, and try to estimate their intermediate I/O functions using partial observations of the hidden layers as input. This resembles reverse engineering in neuroscience, where subsets of neurons in different parts of the brain are measured, and ML methods are used to estimate the I/O relationship between them.

\section{Problem Formulation}
Reverse engineering in neuroscience often aims to understand the functionality or I/O function of areas in the brain. However, brain measurements are noisy and limited by the recording technologies, and the ground truth is not known. Here, we use standard ANNs trained on a classical hand-written digit dataset (MNIST) as the ground truth systems. We use 3000 samples from the MNIST training data to train our ANNs, while the rest is used to produce partial recordings data, described below. Assuming a network of $L+1$ layers (with $l=0$ being the input layer and $l=L$ the output layer), with $N_l$ neurons on each layer, the activity of the neurons in the output layer $L$ is a function of the activity of any previous layer $l$:
\begin{equation}\{O_i^L\}_{i=1}^{N_L} = f(\{O_j^l\}_{j=1}^{N_l}),\end{equation}
where $O_n^l$ is used to denote the output of neuron $n$ in layer $l$. In this work, we train two different ANNs on the MNIST data. Details on network architectures and their performance are shown in Table \ref{tab:nns}. As these ANNs are just our ground truth system, the exact details should not matter much. We now have the ground truth system which we can try to reverse engineer to check how well various methods work based on multiple partial recordings.

\begin{table}
\centering
\caption{\label{tab:nns}The ground-truth neural networks.}
\resizebox{\textwidth}{!}
{\begin{tabular}{|cccccc|} \hline
&\#Hidden&\# Neurons in&\# Neurons in&Accuracy on&Accuracy on\\
&Layers&Hidden Layers& Output Layer& Training Data& Test Data\\\hline
NN&4&128, 128, 128, 128&10&99.62&97.36\\
DNN&7&512, 256, 256, 128, 128, 128, 128&10&99.55&97.78\\ \hline
\end{tabular}}
\end{table}

We want to reverse engineer the activity of the output neurons based on observations of subsets of hidden units, which are a stand-in of potentially recorded neurons in the brain. We will assume that at any given point of time, we can only record a subset of neurons, and will thus have considerable missing information. We want to quantify how well RE is possible based on an ML technique, the approach to deal with the missing data (unrecorded neurons), the number of recorded neurons, and the noise level.

To address these issues, we will simulate partial recordings from the trained ANNs and use ML to reverse engineer the hidden-unit to output-unit mapping. We assume that there is a sequence of settings $k$. For each setting, only a random subset of neurons is observable. We can think of each k, as one setting of the electrical or optical recording apparatus. For each setting, we thus observe the activity of a different subset of the neurons $\{O_j^l\}_{j\in Obs_k}$, while the rest are unobserved. An example of partial data is shown in Fig. \ref{fig:nn}. This way, we produce simulated training sets that approximate what we may record partially from a real brain.

Based on the data collected from the (partial) recordings, we want to estimate the activity of the output neurons as a function of all the observed neurons in layer $l$. More specifically, we want to estimate:
\begin{equation}\label{eq:pr_model}\{\hat O_i^L\}_{i=1}^{N_L} = \hat f(\{O_j^l\}_{j\in\cup_k{Obs_k}}).\end{equation}
This problem is not trivial because the partial recordings produce many missing values corresponding to unobserved neurons in each recording. The ML methods thus need to gracefully deal with missing data. This allows us to quantify how well we can reconstruct the I/O function of the ANNs based on different numbers and nature of partial recordings.

\begin{figure}
\begin{tabular}{cccc}
\includegraphics[width=0.22\columnwidth]{figures/nn-figure.pdf}&
\includegraphics[width=0.22\columnwidth]{figures/rec1.pdf}&
\includegraphics[width=0.22\columnwidth]{figures/rec2.pdf}&
\includegraphics[width=0.22\columnwidth]{figures/rec3.pdf}\\
(a) Trained NN&(b) Subset 1&(c) Subset 2&(d) Subset 3\\
\end{tabular}
\caption{\label{fig:nn}An example of three recording subsets. (a) The ground-truth neural network. (b,c,d): An example of three ($K=3$) settings. In each setting, the output neurons ($O_1^3,O_2^3$) and two of the neurons of layer $1$ (a:$O_3^1,O_4^1$, b:$O_1^1,O_2^1$, c:$O_1^1,O_4^1$) are observed. Observed neurons are outlined in black.} \end{figure}

\section{The Effect of Partial Recordings on Reconstruction Quality}

To reverse engineer a system from partial data, we can use an ML method for regression, but this method needs to deal with massive amounts of missing data. The missing data have a specific structure: in each recording, only a subset of variables (neurons) are observed. The final model in Eq.\ref{eq:pr_model} is defined over the union of all variables that have been observed in at least one of the recordings. The ratio of missing over observed data is higher for recordings with fewer observed neurons per recording.

Missing data is a common problem in statistics, and the way it is dealt with can affect the results, depending on the missingness mechanism. In the simplest case, where data are missing independently of observed and unobserved data, the data are said to be Missing Completely At Random (MCAR) \cite{Rubin1976}. This is the case for the partial data, since the inclusion of a neuron in a recording is based on the sampling design, and does not depend on the activity of the neurons, observed or unobserved. When data are MCAR, excluding the samples with missing values do not bias the results. Arguably, neuroscience is not MCAR. For example, as small neurons are more likely to be unobserved than big neurons because both their optical and electrical signatures are harder to pick up. However, discarding every sample that includes a missing value is not possible here, since every recording has some missing values. Thus, we must select a strategy for dealing with the missing data.

When discarding missing data is not an option, the most common remedy is to impute the missing values in a preprocessing step. Mean value imputation (MVI) is common, where every missing value for a variable is replaced with its sample mean. More complex approaches in the area of multiple imputation \cite{rubin1996multiple} fill in the missing values by assuming a model for the missing values that can be estimated from observed values. The estimated model is then used to impute the missing values. For example, Soft-Impute (SI) iteratively uses a soft-threshold singular value decomposition (SVD) to replace the missing values \cite{mazumder2010spectral}.
A few approaches in statistics try to deal with combining data with overlapping observed variables (similar to partial recordings data). In the field of causal discovery, several methods can learn causal structure from multiple partial observation, based on the assumption of a single underlying causal process \cite{triantafillou2015constraint}. However, these methods are used for learning causal networks and are not applicable in this setting. The field of statistical matching \cite{dorazio2006} proposes some methods for missing data imputation or for inferring the parameters of the joint probability distribution of the union of observed variables without explicitly imputing the missing values. However, these methods require at least some variables measured in common and are therefore not directly applicable in the general cases. Thus, here we use MVI and SI to complete the missing values in our partial recordings data.

After the missing value imputation, the complete data set can be used to train an ML method to estimate Eq.\ref{tab:nns}. We use ANNs and XGBoost in this work. XGBoost \cite{chen2016xgboost} also has a built-in approach for handling missing data without explicitly imputing the missing values, by adding a default direction to each tree node. When a value is missing, the instance is classified in this default direction, which is estimated from the data. Thus, ANNs have to be coupled with an imputation strategy, while XGBoost can also be used directly on the partial data.

An alternative approach to data imputation is training a separate estimator for each recorded subset and combining their predictions. In this case, the estimator in Eq.\ref{eq:pr_model} can be computed as the average of $K$ marginal estimators:
\begin{equation}\label{eq:pr_models_mp}
\hat f(\{O_j^l\}_{j\in\cup_k{Obs_k}}) = \frac{1}{K}\sum_k \hat f_k(\{O_j^l\}_{j\in Obs_k})
\end{equation}

The advantage of marginal estimators is that each $\hat f_k$ can be identified with any appropriate ML method without preprocessing, in parallel. However, each marginal estimator is then estimated based on fewer samples.

In the next section, we study the quality of reverse engineering based on multiple partial recordings, and how it depends on different factors, using simulated data from the ground truth networks.

\section{Experiments}
Here, we use simulated experiments to examine how the quality of reconstruction based on partial recordings depends on: (a) the data imputation method, (b) the size of the ground truth network, (c) the number of different recorded subsets and (d) the noise level. We use the networks in Table \ref{tab:nns} as our ground truth networks, and simulate observations by sampling with replacement from the second half of the MNIST training data that were not used for training the ground truth networks. We typically simulate $K=10$ partial recorded subsets. This means that we consider $K$ subsets of neurons to record from, each chosen randomly (and thus often overlapping), which corresponds to switching the subset of neurons recorded $K-1$ times. In each subset, all output neurons and $N$ neurons from a given hidden layer are observed. We then train an ML method to predict the values of the output neurons based on the partial data of the given hidden layer. We calculate the average (over neurons) of the Root Mean Squared Error (denoted by \rmse) over all output neurons, then we report the mean and the standard deviation of \rmse\ over $5$ iterations as a measure of the quality of our RE. We can thus ask how all the mentioned factors relate to the quality of reverse engineering.

The two popular methods we will look at are XGBoost and ANNs as they represent probably the two most popular approaches in supervised regression. More specifically, We use them coupled with different imputation techniques (MVI and SI), as well as with combining separate marginal models (denoted by MP for mean prediction). In addition, we used XGBoost with the built-in approach for handling missing data (denoted by XBG-G). For each partial data set, we trained an independent model with XGBoost for every output neuron, and a single ANN model with 10 output neurons. XGBoost was used with the following parameters: $\eta=0.5$ and $max\_depth=6$. We used an ANN with three hidden layers ($16$ neurons in each, ReLU activation function) and applied early-stopping based on the loss value. Having implementations of all methods allows us to ask how they perform at reverse engineering the ground truth ANNs.

We should expect that having more data helps us better predict output responses. Indeed, prediction quality improves but saturates at a level that depends on the number of simultaneously recorded neurons (Fig. \ref{fig:sample_size}). The same qualitative behavior was seen for the methods we do not show in the figure which we suppressed for clarity. We may expect and observe that the saturation level that can be reached may depend strongly on the number of neurons in the system, the manifold of stimuli used, and the nonlinearity of the system.

\begin{figure} \centering
\includegraphics[trim=.5cm 0cm 1cm 1cm, clip, width=.55\columnwidth]{figures/diffSampleSize_NNMVI_RMSE_2.pdf}\\
\caption{\label{fig:sample_size}\textbf{Unsurprisingly, having more data leads to better reconstructions.}
\rmse\ over $5$ iterations as a function of sample size per setting from the first and the third hidden layer of the NN, using ANN-MVI.} \end{figure}

To make comparisons meaningful, we will for the rest of the paper, only compare situations where the same total amount of information is acquired, i.e. each training set will have the same product of the number of observed neurons and the number of observations. This is meant to mimic the constraints of real neuroscientific experiments where fixed information throughput limits the overall recorded data set size. You can either opt to record more neurons (larger subset sizes) for fewer samples, or fewer neurons for more samples.

We want to know which method is better at dealing with many partial recordings. We thus measure the quality of RE for different methods and a different number of observed neurons in each recording (Fig. \ref{fig:RMSEs}). Both layers have similar performance. The error decreases as the number of observed neurons increases, although the total number of non-missing data points remains nearly the same. For both layers, ANNs with MVI perform better for almost all settings.

\begin{figure} \centering
\includegraphics[trim=3.55cm 0cm 3.8cm 0cm, clip, width=\columnwidth]{figures/diffMethods_RMSEs.pdf}\\
\caption{\label{fig:RMSEs}\textbf{Neural networks with mean value imputation work best at reverse engineering an
ANN.}
\rmse\ as a function of the subset size from the first (left) and third (right) layer of the ground truth NN.} \end{figure}

How successfully can we reverse engineer a deeper and wider neural network? And, how many neurons must we observe to have reasonably good error rate? To address these questions, we perform the experiments using ANN-MVI with data simulated from a deeper neural network (DNN) as the ground truth network; to compare NN and DNN, see Table \ref{tab:nns}. Results in Fig. \ref{fig:NN_DNN} show similar trends in both NN and DNN data: increasing the number of observed neurons improves performance. Moreover, for wider layers, a smaller ratio of neurons is required for adequate performance. In all layers, by observing roughly twenty percent of the neurons, RE achieves nearly the same error rate as with a fully observed layer. Therefore, ANN with MVI may be a reasonable choice for deep neural networks.

\begin{figure} \centering
\includegraphics[trim=3.55cm .2cm 3.8cm .2cm, clip, width=1\columnwidth]{figures/NNMVI_RMSEs_NN_vs_DNN_2.pdf}
\caption{\label{fig:NN_DNN} \textbf{Simultaneously recording more neurons helps reverse engineering for a fixed number of samples, just as well for shallow and deep networks.}
Left: \rmse\ for RE as a function of observed subset size using ANN-MVI in layers $1, 3$ of the NN and in layers $1,3,6$ of the DNN. Right: Performance (as a percentage of maximum decrease in \rmse) vs percentage of observed neurons for each layer.} \end{figure}

Results so far show that increasing the number of the observed neurons at the expense of presenting fewer stimuli improves RE for a fixed number of selections of subsets of neurons. However, it is possible that increasing the number of selected subsets can compensate for few simultaneously recorded neurons. Choosing many different subsets is often experimentally feasible, e.g. by focusing a laser on different subsets of neurons on a different plane. To test the trade-off between the number of subsets and number of recorded neurons in each, we simulated partial data for varying $K$, and a different number of observed neurons per subset. RE with a single selection of neurons is not meaningfully possible for small numbers of recorded neurons (Fig. \ref{fig:par_rec}). However, increasing the number of selected subsets improves the quality of the reverse engineering; this improvement is more noticeable when fewer neurons are simultaneously recorded. While the changes in performance are not always monotonic, in general, combining more subsets can compensate for the lack of full observations. Thus, combining more recordings of small subsets, we can achieve similar performance to combining fewer recordings of larger subsets.

\begin{figure} \centering
\hspace*{-.55cm}
\includegraphics[trim=4.6cm 1.1cm 0cm 1.7cm, clip, width=1.19\columnwidth]{figures/diffRec_NNMVI_RMSEs_NN_vs_DNN.pdf}
\caption{\label{fig:par_rec} \textbf{Sequentially recording from many subsets of neurons can compensate for recording only small subsets at a time.}
\rmse\ for RE using ANN-MVI for a different number of sequentially recorded subsets $K$, keeping the total number observed values approximately constant. Results are shown for layers $1,3$ of the NN and $1,3,6$ of the DNN.} \end{figure}

The recordings of neural networks include several sources of noise. The effective noise in the recordings has a component that comes from measurement, e.g. Johnson noise in the electrode, and a component that comes from the brain itself (e.g. Poisson noise in spiking). To study the effect of noise on RE, we add signal-dependent noise to the partial recordings. For each sample, we add zero-mean Gaussian noise with variance that is proportional to the activity of the neuron in the given sample. We set the noise variance to $20, 50$ or $100$ percentage of the neurons' activity value. Here, $100\%$ corresponds to the level expected for a Poisson process. We add the noise after simulating the partial recordings and use ANNs to estimate the I/O function. To check which of the imputation techniques can handle the noise better, we consider both MVI and SI imputation techniques. MVI still outperforms SI for partial data with additive Gaussian noise, but both methods are robust to all levels of additive signal-dependent noise (Fig. \ref{fig:noise}).

\begin{figure} \centering
\includegraphics[trim=3.55cm .2cm 3.8cm 0cm, clip, width=1\columnwidth]{figures/Noise_RMSEs_2.pdf}
\caption{\label{fig:noise}\textbf{Noise has little effect but mean value imputation beats soft imputation.}
\rmse\ for RE as a function of the observed subset size using ANN-MVI and ANN-SI with various additive noise levels.} \end{figure}

To check if our scaling analysis is applicable to current neuroscience datasets, we used spike count data (50ms bins for 172 neurons) from cortical area PMd \citet{stevenson2011statistical} to predict hand velocity of a monkey, with the same setting as simulations. We then reverse engineer the I/O function using XGBoost with MVI. Similar to the simulations, performance improves with larger subset sizes (Fig. \ref{fig:spiking}). However, this behavior does not even get close to reaching saturation. This is expected because in this dataset we only record from order hundreds of millions of neurons involved in a hard task (life). Observing more neurons simultaneously helps, but not all that much. However, we do believe that recordings from considerably more neurons are necessary to make this analysis meaningful.

\begin{figure} \centering
\includegraphics[trim=0cm 0cm 1cm 1cm, clip, width=.55\columnwidth]{figures/XGBMVI_RMSEs_Spiking.pdf}
\caption{\label{fig:spiking}\textbf{Recording from larger subsets of neurons in PMd helps for predicting hand kinematics.}
\rmse\ for RE as a function of the observed subset size using XGBoost-MVI on spiking data.} \end{figure}

\section{Discussion}
Modern techniques in neuroscience allow us to simultaneously record from many neurons. Motivated by this, we investigated how well we can reverse engineer neural activity in ANNs from multiple partial observations. Surprisingly, MVI - the simplest and easiest imputation technique - achieves a lower error rate for RE a system with multiple partial recordings even compared to a more complex and computationally expensive method, like Soft Impute. Also, we found that increasing the number of simultaneously observed neurons increases the quality of our RE. However, many recordings from small subsets of neurons can well approximate what we get if we simultaneously record from all neurons. In real neural data, observing more neurons simultaneously also improves the quality of RE, however, since we can only observe a tiny part of the actual system, this behavior does not reach a saturated point.
Thus, combining multiple partial recordings may improve our understanding I/O functions in the brain.

Obviously, an artificial neural network is not a brain. Even though ANNs are inspired by neuroscience, brains are biophysically more complex and probabilistic. Biological neurons can perform on many different time scales and are more heterogeneous in many ways. In the brain, the communication takes place through spikes, whereas it happens via abstract rates in neural networks. Moreover, in our specific work here, we dealt with a tiny ANN. Our analysis might give categorically different answers if we could apply it to the real brain.

The architecture of our ground truth ANNs is obviously idiosyncratic: The networks we use are simple feed-forward neural networks with no lateral or recurrent connections. More complex architectures, typical in the brain, may produce different behavior. In the future, it would be interesting to extend our analysis to recurrent neural networks and to networks with lateral connections.

In all our simulations, the ground truth ANNs used were trained on the MNIST dataset, which may have an atypical structure. After all, for each of the digits, there are a small number of universally accepted generative models that are transmitted culturally at places called Kindergarten. This means that the manifold of produced digits will have a far simpler structure than typical manifolds of things in the real world. Moreover, this structure makes the MNIST problem easy from an ML perspective and may lead to low dimensional solutions that may lead our technique to produce a low estimate for the number of neurons that need to be simultaneously recorded.

This work presents a test case where artificial systems are used to evaluate and guide computational neuroscience. Using ANNs as a stand-in for an actual neural circuit, we found that reverse engineering neural function based on a small fixed subset of recorded neurons is not possible, even for a low dimensional task. Using real spiking data we found some evidence that such behavior may also be real for the brain. This may suggest that the number of neurons that need to be simultaneously recorded may be larger for more complicated tasks. We, therefore, believe that a similar scaling analysis should be standard for reverse engineering in neuroscience applications.

Modern neuroscientific techniques allow running similar analyses to the one we presented here on real brains. Using Ca2+ imaging \cite{kerr2008imaging,grienberger2012imaging} along with modern optical targeting techniques allows trading of the number of recorded neurons with the noise level. It also allows simultaneous recording from the input parts of a system and the outputs. As such, the neural network that we used here, could be readily replaced with a real sample of brain tissue. Doing scaling analyses on reverse engineering approaches is the only way of knowing how recording techniques need to be optimized.

\bibliographystyle{unsrtnat}


\section{Introduction}
\label{introduction}

Portfolio allocation plays an important role in the financial market, which is fundamental and important for investment companies and quantitative analysts. The famous economist, Harry Markowitz received the 1990 Nobel Memorial Price in Economic Sciences for his pioneering theoretical contributions to financial economics and corporate finance. His innovative work laid the foundation for Modern Portfolio Theory (MPT) \cite{sharpe1970portfolio}, i.e., construction of a portfolio to maximize the expected return while minimizing the investment risk.
Portfolio theory studies how ``rational investors'' optimize their portfolios. A rational investor may maximize the expected return at a given level of expected risk or minimize the expected risk at a given expected level of return. The essence of investment is a tradeoff between the profits and risks of uncertainty.

 The target of portfolio allocation is that either maximizing the Sharpe ratio (average return minus the risk-free return divided by the standard deviation) or minimizing the risk for a range of returns. Portfolio theory uses the mean-variance to characterize these two key factors. The mean value refers to the weighted average of expected return, and the weight is the allocated fraction of investment. The variance refers to the variance of the expected return of the portfolio. We refer to the standard deviation of expected return as the volatility, which portrays the risk of the portfolio. Traditional approach is performed in two steps as described in \cite{markowitz1952portfolio}. First, the expected returns of the stocks and the covariance matrix of the stock prices are computed. The trading strategy is then extracted by following the portfolio allocation.

 However, the challenge of the traditional portfolio allocation is that the approach can be very complicated to implement if the manager wants to revise the decisions made at each time step and take, for example, transaction cost into consideration.
The financial market is complex and being influenced by immense factors such as macroeconomic, traders' expectation and risk aversion.  Investment companies are eager to optimize allocation of capital and thus maximize performance, for example, get a higher return while the risk is as small as possible.

 Despite traditional portfolio allocation's momentous theoretical importance, many people who criticize MPT believe that the basic assumptions and models of their financial markets are inconsistent with the real world in numerous respects \cite{mangram2013simplified}.
 In general, some key criticisms include: investor ‘irrationality’, high risk is accompanied by high returns, no taxes or transaction costs, the market is efficient, investment independence, unlimited access to capital and investors may get perfect information.

In this paper, we apply the optimistic and pessimistic reinforcement learning for stock portfolio allocation. The existing approach to solve the stock trading problem is to model it as a Markov Decision Process (MDP) and use dynamic programming (DP) to obtain the optimum strategy.
But the DP algorithms have limited practical usages because they assume a perfect model and they are also compute-intensive.
The scalability of DP is limited due to the state space explosion when dealing with the stock market. Reinforcement learning has no such restrictions. Deep reinforcement learning methods use a function approximator and a stochastic approximation to calculate a relevant expectation, which can be applied to problems with a large continuous state space \cite{neuneier1998enhancing}. Then, we explore the Deep Deterministic Policy Gradient (DDPG) \cite{lillicrap2015continuous}\cite{Liu2019NIPS}\cite{Wenhang2019ICML}, to find the best trading strategy in the complex and dynamic stock market.

We adopt the DDGP algorithm that consists of three key components: (i) actor-critic framework \cite{konda2000actor} that models large state and action spaces; (ii) target network that stabilizes the training process \cite{mnih2015human}; (iii) experience replay that removes the correlations between samples and increases the usage of transition data. The efficiency of DDPG algorithm is demonstrated by achieving higher return than the traditional portfolio allocation method and the Dow Jones Industrial Average index.

Furthermore, the proposed deep reinforcement learning scheme takes into account the impact of the market index, which is very meaningful in practice. Since the machine learning based methods are more objective and more quantitative than the trader's decision based on the market. Theoretical studies of behavioral finance have shown that environment can influence investment decisions. Investors may optimistic and pessimistic, as behavioral finance has asserted \cite {li2014effect}. In general, the bear market occurred during a recession or depression, when pessimism occurred. When the price of securities rises faster than the overall average interest rate, there will be a bull market. Bull market is accompanied by economic growth and investor optimism.
We hence proposed a modified Rescorla-Wanger model~\cite{lefebvre2017behavioural}, which can learn differently from positive and negative environment, which can calculate reward of choosing different options (buy, hold and sell). The model can adjust the amplitude of changes according to the sign of prediction errors. For example, when the prediction error is positive (actual reward is better than the expected reward), then the learning rate would adjust the amplitude from one trial to the next. The model can distinguish between good and bad environment feedback. Furthermore, we propose the optimistic and pessimistic reinforcement learning model that are applicable to both the bear market and the bull market, respectively.

The reminder of this paper is organized as follows. Section 2 contains background of stock portfolio allocation. In Section 3, we drive and specify the main DDPG algorithm and  optimistic-pessimistic deep reinforcement learning. Section 4 describes data preprocessing, experimental setup and presents the performance of Adaptive DDPG model. Section 5 gives our conclusions.

\section{Problem Formualtion for Stock Portfolio Allocation}

In this section, we introduce the portfolio allocation model, the impact of the market environment on stocks, and the relevant reinforcement learning method that can be applied to the stock trading strategy.

\subsection{Portfolio Allocation}

\begin{figure*}[!tb]
	\centering

	{\includegraphics[width=3.0in]{crisis.pdf}}
	{\includegraphics[width=3.0in]{not_crisis.pdf}}

	\caption{Portfolio allocation for: (a) 01/05/2009-03/31/2009 (during financial crisis); (b) 04/01/2009-06/25/2009 (after financial crisis).}
	\label{figure:portfolio1}
\end{figure*}

\subsubsection{Portfolio Allocation}

The portfolio theory discussed in this paper is narrowly defined. In the developed securities market, Markowitz's portfolio theory has proven to be effective in practice and is widely used in portfolio selection and asset allocation.

The theory contains two important parts: 1) the mean-variance analysis method; 2) and the portfolio efficient frontier. Specifically, the optimized investment portfolio is depicted in a two-dimensional plane with the volatility as the abscissa and the return as the ordinate, forming a curve. There is a point on this curve with the lowest volatility, called {\em the min-variance point (MVP)}. The portion of this curve above the min-variance point is the well-known Markowitz portfolio effective boundary, and the corresponding portfolio is called the effective portfolio. The portfolio's efficient frontier is a monotonically increasing convex curve. However, because the traditional effective market hypothesis cannot explain market anomalies, portfolio theory is challenged by the behavioral finance theory.

\subsubsection{Traditional Methods}

 We provide two base portfolio allocation methods for investors with different risk levels. The first one is using mean-variance optimization to allocate the stocks, this method is suitable for the investors who prefer a higher sharp ratio. The second one is min-variance portfolio with the lowest possible risk for investors.

\textbf{Mean variance method:}

Markowitz's work~\cite{sharpe1970portfolio} show that it is not a security's own risk that is important to an investor, but rather the contribution the security makes to the variance of the entire portfolio. This follows from the relation between the variance of the return of a portfolio  ($\sigma_p^2$)  and the variance of the return of its constituent securities  ($\sigma_i^2$, $i=1,2,…,m$). Calculate the annualized rate of return $\mu_p$ and covariance matrix $\Sigma_p$ as follows:
\begin{align}
\label{Mean}
\mu_p&=E(r_p)= \sum_{i=1}^{m}w_i E(r_i)=W^T\mu, \\
\sigma_p^2&=\sum_{i,j=1}^{m}w_iw_j\sigma_{i,j}=W^T\Sigma_p W,
\end{align}
where $m$ is the number of stocks, $w_i$ is the weight of $i$-th stock, which is  the portfolio percentage. $\sigma_{ij}$ is the covariance of the returns of stock $i$ and $j$. Then we set constraints for the mean-variance:
\begin{align}
\label{con_mv1}
w_i \in [0,0.2],i=1,...,m;~~\sum_{i=1}^{m}w_i =1,
\end{align}
where 0 and 0.2 is the lower bound and upper bound of allocation weight. Our objective function is to find the allocation that makes the highest sharp ratio (the portfolio of red stars in Figure~\ref{figure:portfolio1}).

\textbf{Min-variance method:}

The min-variance method is similar with the mean variance method, except the objective function is replaced by finding the portfolio with the smallest variance (the portfolio of green stars in Figure~\ref{figure:portfolio1}).

Therefore, the decision to hold securities should not be made solely by comparing its expected returns and variance with other stocks, but depend on the other stocks the investor wants to hold. Stocks should be properly evaluated as a group instead of in isolation.

\subsubsection{Limitation of Modern Portfolio Theory}
As we mentioned in the Introduction, none of these MPT's assumptions are entirely true and the assumptions compromised MPT. The key limitations of the MPT are as follows:
\begin{itemize}
\item \textbf{Assumption 1:  investors are rational.}

MPT assumes that the investor is rational and seeks to maximize returns while minimizing risk. This contradicts the observations of market participants who were involved in the “herd behavior” investment activity \cite{morien12travis}. For example, investors often choose “hot” stocks, and because of speculative excessive behavior, markets often experience prosperity or depression. Large stock market trends often begin and end with periods of frenzied buying (bubbles) or selling (crashes). These herding behavior that is irrational and driven by emotion-greed in the bubbles, fear in the crashes. Even if herd behaviors might be rare, this has important consequences for a whole range of real markets.

\item \textbf{Assumption 2:  the market is efficient.}

Markowitz theoretical assumes that the market is fully valid \cite{markowitz1952portfolio}. In contrast, it does not consider potential market failures such as information asymmetry. A hundred years of  ``prosperity'', ``depression'', ``bubble'' and ``financial crisis'' indicate that the market is far from efficient. Using the market index to do the  portfolio allocation can overcome the market's ineffective problems to some extent.

\item \textbf{Assumption 3:  investment are independent.}

MPT assumes that securities whose individual performance is independent of other securities are selected.  However, during market pressures and extreme uncertainty, seemingly independent investments actually show relevance. Market history has proven that there is no such tool \cite{mcclure2010modern}. Stocks and markets cannot be separated, and the influence of market environment on stocks must be considered.
\end{itemize}

MPT seeks to maximize risk-adjusted returns while ignoring environmental, personal, strategic or social factors. The historical `expected value' assumptions often fail to consider that the updated environment does not exist during historical data.

\subsection{Market Environment}

The market price of a stock is determined by the value of the stock, but at the same time it is affected by many other factors. Generally, the factors affecting the stock market price mainly include the following two aspects:
\begin{itemize}
\item Macroeconomic factors: The impact of the macroeconomic environment and its changes on the stock market price, including the regular factors such as the cyclical fluctuations of macroeconomic operations and the policy factors such as the economic policies implemented by the government \cite{flannery2002macroeconomic}. The stock market is an important part of the entire financial market system.  Therefore, stock prices in the stock market will naturally change with the macroeconomic conditions. For example, in general, stock prices fluctuate with the rise and fall of gross national products.
\item Market factors:
Various stock market operations may affect the stock market price. For example, bullish and bearish, short and short selling, chasing and killing \cite{chang2009integrating}. Generally speaking, if in the bull market, investors intend to be more aggressive, and the stock price will rise. Conversely, if the short-selling behavior prevails and the investor is overwhelmed, the stock price tends to fall. Since various stock market operations are mainly short-term behaviors, the impact of market factors on stock market prices has a clearly short-term nature.
\end{itemize}

To illustrate the impacts of the market's overall environment on the portfolio, we compared the portfolio strategy for three months before and after the end of the financial crisis. The Stock Pool is Dow Jones 30. Figure~\ref{figure:portfolio1}(a) plots the portfolio allocation during the financial crisis and Figure~\ref{figure:portfolio1}(b) plots the portfolio allocation after the financial crisis. Comparing these two figures, we find that in the financial crisis, the investment portfolio generally shows a low return and high volatility state; after the financial crisis, the economy begins to recover, and the portfolio presents relatively low volatility and higher returns.

An emerging area of applied reinforcement learning is stock market trading, in which a trader's behavior is similar to an agent because buying and selling are particular actions. Reward is that stock changes the state of the trader by generating profit or loss.

\subsection{ Markov Decision Process Formulation}
MDP is particularly important for reinforcement learning. A particular MDP is defined by its state $s$ and action set $a$ and the one-step dynamics of the environment. Given any state and action, the probability of each possible next state $s^{\prime}$ is as follows
\begin{align}
\label{MDP}
\mathcal{P}_{s s^{\prime}}^{a}=\operatorname{Pr}\left\{s_{t+1}=s^{\prime} | s_{t}=s, a_{t}=a\right\}.
\end{align}
These quantities are called transition probabilities. Similarly, the expected value of the next reward is
\begin{align}
\label{MDP2}
\mathcal{R}_{s s^{\prime}}^{a}=\mathbb{E}\left\{r_{t+1} | s_{t}=s, a_{t}=a, s_{t+1}=s^{\prime}\right\}.
\end{align}
We assume that the environment is a finite MDP. The quantities $\mathcal{P}_{s s^{\prime}}$ and $\mathcal{R}_{s s^{\prime}}^{a}$ specify the most important aspects of limited MDP dynamics.

\subsection{Relevant Reinforcement learning method}
As mentioned in the Introduction, DDPG is mainly developed from: PG (Policy Gradient)$\rightarrow$DPG (Deterministic Policy Gradient)$\rightarrow$DDPG (Deep Deterministic Policy Gradient). Next, we will introduce this evolution process to give reasons for adopting DDGP to find the optimal trading strategy in the complex and dynamic stock market.

\section{Adaptive Deep Reinforcement Learning}

\label{reinforcement}
We model the stock trading process as a MDP. We then formulate our trading goal as a maximization problem. The portfolio allocation task is formalized as MDP under the following assumptions:

\begin{enumerate}
\item Investors can trade at each time step along a continuous infinite time horizon.
\item A single investor's trading cannot influence the market.
\item There are only two kinds of assets (stocks and risk-free asset) for investing the capital.
\item The investor invests the total amount, which means investors has no risk aversion.
\end{enumerate}

\subsection{Basic Idea}

Training intelligent agents for automatic financial assertion transactions is a long-standing topic that has been widely discussed in modern artificial intelligence \cite{saad1998comparative}. In essence, the transaction process is well described as an online decision problem, which involves two key steps: market environment and best action execution. Due to the lack of supervision information, dynamic decision making is more challenging. Therefore, it requires agents to explore the unknown stock market environment themselves and make the right decisions at the same time.

Compared to traditional reinforcement learning tasks, algorithmic trading is much more difficult due to the following challenges: 1) the challenge stems from the difficulty of summarizing and representing the financial environment; 2) financial data contains a lot of noise, jumping and moving, resulting in a very unstable time series.

In summary, the reinforcement learning trains the agent to interact with an environment to obtain the maximum total reward. This bonus value is generally associated with the mission target defined by the agent.

\subsubsection{Basic concepts related to DDPG}

In order to address the above problems and considering the stochastic and interactive nature of the trading market, we model the stock trading process as a Markov Decision Process (MDP) as shown in Figure~\ref{figure:buy}, which is specified as six parts: state, action, reward, discounted future reward, policy and action-value.
\begin{figure}[!tb]
	\centering
	{\includegraphics[width=3.0in]{buy_hold_sell3.pdf}}
	\caption{Portfolio allocation 0 is the starting state at time $t$, three actions (buy, hold and sell) lead to three possible portfolios at time $t+1$ after price changes.}
	\label{figure:buy}
\end{figure}

State $s= [p,w,b]$: $s$ is the state of the environment. These states are generated based on the agent's behavior strategy. $s$ is a set that includes the information of the prices of stocks $p\in \mathbb{R}_+^D$, the weight of holdings of stocks $w=(w_1,w_2,...,w_D)^T$; $w_i\in [0,1],~ i=1,2,...,D;~ \sum_{i=1}^{D}w_i=1$, and the remaining balance $b\in \mathbb{R}_+$, where $D$ is the number of stocks that we consider in the market and $\mathbb{Z}_+$ denotes non-negative integer numbers.

Action $a$: a set of actions on all $D$ stocks. The available actions of each stock include selling, buying, and holding, which result in decreasing, increasing, and no change of the weight of holdings $w$, respectively. DDPG is a learning method for continuous behavior. In our model, the action is continuous, because our weights are continuously changing.

Reward $r(s,a,s')$:
the change of the portfolio value when action $a$ is taken at state $s$ and arriving at the new state $s'$. The portfolio value is the sum of the equities in all held stocks $p^T w$ and balance $b$. We also called $r(s,a,s')$ as the single-step reward value, returned by the environment after the action $a$ is executed in the states. The above relationship can be represented by a state transition in Figure~\ref{figure:state}:

\begin{figure}[!htb]
	\centering
	{\includegraphics[width=2.3in]{state.pdf}}
	\caption{State transition diagram.}
	\label{figure:state}
\end{figure}

Discounted future reward $R_t$: is the weighted sum of the reward values obtained for all actions from the current state to a future state.
\begin{align}
\label{discount}
    R_{t}=\sum_{i=t}^{T} \gamma^{i-t} r\left(s_{i}, a_{i},s_{i+1}\right),
\end{align}
where $\gamma$ is called discounted rate, $\gamma\in[0,1]$, usually $\gamma=0.99$.

Policy $\pi(s)$: the trading strategy of stocks at state $s$. It is essentially the probability distribution of $a$ at state $s$. To be more specific, given a state, the decision policy will calculate the next action to take.

Action-value function $Q_\pi(s,a)$: the expected reward achieved by action $a$ at state $s$ following policy $\pi$.

 \subsubsection{Framework of stock market dynamics}

One of the solutions to the stock trading problem is to model it as a MDP and use dynamic programming (DP) to solve the optimal strategy. However,  DP only solves problems with small discrete state spaces.

Driven by these challenges, we explore the deep reinforcement learning algorithm DDPG \cite{lillicrap2015continuous}\cite{Liu2019NIPS}\cite{Wenhang2019ICML}, to find the best trading strategies in complex dynamic stock markets. Most reinforcement learning algorithms boil down to just three main steps: infer, perform, and learn. During the first step, the algorithm selects the best action $a$ given a state $s$ using the knowledge it has so far. Next, it performs the action to find out the reward $r$ as well as the next state $s$. Then, it improves its understanding of the world using the newly acquired knowledge. We would describe the framework of stock market dynamics as follows. We use subscript to denote time $t$, and the available actions on stock $d$ are:
\begin{itemize}
    \item Selling: $k$ ($k\in [0,w_d]$, where $d = 1,...,D$) weight of shares can be sold from the current holdings, where $k$ must be a weight. In this case, $w_{t+1}=w_t-k$.
    \item Holding: $k=0$ and it leads to no change in $w_t$.
    \item Buying: $k$ weight of shares can be bought and it leads to $w_{t+1}=w_t+k$. In this case $a_t[d]=-k$ is  negative weight.
\end{itemize}
It should be noted that all bought stocks should not result in a negative balance on the portfolio value. That is, without loss of generality we assume that selling orders are made on the first $d_1$ stocks and the buying orders are made on the last $d_2$ ones, and that $a_t$ should satisfy $p_t[1:d_1]^T a_t[1:d_1]+b_t+ p_t[D-d_2:D]^T a_t[D-d_2:D]\ge 0$. The remaining balance is updated as $b_{t+1}=b_t+p_t^T a_t$.
As defined above, the portfolio value consists of the balance and sum of the equities in all held stocks.
At time $t$, an action is taken, and based on the executed action and the updates of stock prices, the portfolio values change from ``portfolio value 0'' to ``portfolio value 1'', ``portfolio value 2'', or ``portfolio value 3'' at time $(t+1)$.

According to Bellman Equation, the expected reward of taking action $a_t$ is calculated by taking the expectation of the rewards $r(s_t,a_t,s_{t+1})$, plus the expected reward in the next state $s_{t+1}$. Based on the assumption that the returns are discounted by a factor of $\gamma$, we have
\begin{align}
    \label{decom}
    Q_\pi(s_{t},a_t) = &~\mathbb{E}_{s_{t+1}} \{r(s_t,a_t,s_{t+1})  \nonumber\\
    &+ \gamma \mathbb{E}_{a_{t+1} \sim \pi(s_{t+1})}[ Q_{\pi} (s_{t+1},a_{t+1})] \}.
\end{align}
The above $Q$ function is the action-value function, defined in the state $s_t$, after taking the action $a_t$.

The goal is to design a trading strategy that maximizes the investment return at a target time $t_f$ in the future, i.e., $p_{t_f}^T w_t+b_{t_f}$, which is also equivalent to $\sum_{t=1}^{t_f-1}r(s_t,a_t,s_{t+1})$. Due to the Markov property of the model, the problem can be boiled down to optimizing the policy that maximizes the function $Q_\pi (s_t,a_t)$. This problem is very hard because the action-value function is unknown to the policy maker and has to be learned via interacting with the environment. Hence in this paper, we employ the optimistic \& pessimistic deep reinforcement learning approach to solve this problem.

\subsection{Optimized Model Incorporates Market Environment}

In order to incorporate the market information into the deep reinforcement learning, we propose an effective method to quantitatively analyze the mechanism of stock information penetration.

\subsubsection{Modified Rescorla-Wanger model}
The computational part include a Rescorla-Wanger model (also called Q-learning, in the following referred as RW model). Base on the RW model, we use a modified model which learns differently from positive and negative environment emotional news. Positive and negative prediction errors representing positive and negative environment emotional news respectively (refer to as RW$\pm$). For each state, Q-values represent the expected reward by taking a specific action in a given market environment. Considering that we have three actions buying, holding and selling, the model estimates the expected values of buying, holding and selling options, on the basis of sequence actions and outcomes. The initial Q-value is set as 0 before learning. In each step $t$, the value of the options (buy, hold and sell) is updated according to the rules, as follows:
\begin{align}
\label{Qbuy}
    Q_{\pi}(s_{t+1},a_{t+1})=Q_{\pi}(s_t,a_t)+\alpha\delta(t).
\end{align}
where $\alpha$ is the learning rate, which is a scaling parameter that adjusts the magnitude of the change from one trial to the next, and $\delta(t)$ is the prediction error (we also define as environment emotional news), calculated as follows:
\begin{align}
    \delta(t)=r(s_t,a_t,s_{t+1})-Q_{\pi}(s_t,a_t),
\end{align}
which is the difference between the expected reward of $Q_{\pi}(s_t,a_t)$ and the actual reward $r(s_t,a_t,s_{t+1})$. Following this rule, the option value is increased if the result is better than expected, while the option value is decreased in the opposite case, and the amplitude of the update is similar after the positive, neutral and negative prediction errors.

\subsubsection{Update rule }
The update rule of the modified Q-learning algorithm (RW$\pm$) is given by \cite{lefebvre2017behavioural}
\begin{align}
\label{Qal_buy}
    Q_{\pi}(s_{t+1},a_{t+1})=Q_{\pi}(s_t,a_t)+ \left\{ \begin{array}{cc}{\alpha^{+} \delta(t)} & {\text { if } \delta(t)>0,} \\ {\alpha^{-} \delta(t)} & {\text { if } \delta(t)<0.} \end{array} \right.
\end{align}
When the prediction error is positive, which means the actual reward $r(s_t,a_t,s_{t+1})$ is better than the expected reward $Q_\pi(s_t,a_t)$, the learning rate $\alpha^+$ adjusts the amplitude of the change from one trial to the next, and vice versa.
Therefore, the RW$\pm$ model allows the amplitude of the update to be different, a following sequential positive (‘good  environment emotional news’) and negative (‘bad  environment emotional news’) prediction errors. The RW$\pm$ model also allows us to consider individual differences in the way which learn from positive and negative experiences.

Furthermore, given the Q-values, the associated policy of selecting each options (buy, hold and sell) was estimated by implementing the softmax rule as follows:
\begin{align}
\label{soft}
   \pi(s_t)=e^{\left(Q_{\pi}(s_t,a_t) \beta\right)} /\left(e^{\left(Q_{\pi}(s_t,a_t) \beta\right)}+e^{\left(Q_{\pi}(s_t,a_t) \beta\right)}\right),
\end{align}
where $\beta$ is a scaling parameter that adjusts the stochasticity of decision-making, which is used to control the exploration–exploitation trade-off. \eqref{soft} is a standard stochastic decision rule that calculates the probability of selecting one of a set of options based on the associated value.

\subsection{Adaptive DDPG}

We use the adaptive DDPG algorithm to maximize return on investment. DDPG is an improved version of the Deterministic Policy Gradient (DPG) algorithm and DPG is based on Policy Gradient (PG) improvements. As for the DDPG, Q-learning uses greedy action $a_{t+1}$ to maximize $Q\left(s_{t+1}, a_{t+1}\right)$ for state $s_{t+1}$ as follow
\begin{align}
    \label{greedy}
    &Q_{\pi}\left(s_{t}, a_{t}\right)  \nonumber \\
    =&~\mathbb{E}_{s_{t+1}}\left[r\left(s_{t}, a_{t}, s_{t+1}\right)+\gamma \max _{a_{t+1}} Q\left(s_{t+1}, a_{t+1}\right)\right].
\end{align}
 As shown in Figure~\ref{figure:network}, the adaptive DDPG includes an actor network and a critic network. The actor network $\mu\left(s | \theta^{\mu}\right)$ maps state to actions, and after the prediction error $\delta(t)$ is available, the critic network then updates $Q\left(s, a | \theta^{Q}\right)$ according to the prediction error $\delta(t)$ and the learning rate $\alpha^+$ (or $\alpha^-$), where $\theta^\mu$ is the set of actor network parameters and $\theta^Q$ is the set of critic network parameters. $\mathcal{N}^{+}$ and $\mathcal{N}^{-}$ are the random processes corresponding to the positive and negative environment respectively, which are used to add noise to the output of the actor network to explore actions.

 \begin{figure}[!tb]
	{\includegraphics[width=3.2in]{network2.pdf}}
	\caption{Learning network architecture.}
	\label{figure:network}
\end{figure}

 Similar to DDPG, our model uses the experience replay buffer $R$ to store transitions. The adaptive DDPG agent updates $Q(s,a)$ according to the prediction error $\delta(t)$ and takes an action $a_t$ on $s_t$, and then receives a reward based on $s_{t+1}$. Then calculate $y_i = r_i+\gamma Q' (s_{i+1}, \mu' (s_{i+1}|\theta^{\mu'},\theta^{Q'})), i=1,...,N$. The transition $(s_t,a_t,s_{t+1},r_t)$ is then stored in replay buffer $R$. After $N$ sample transitions are drawn from $R$, we update the critic network by minimizing the expected difference $L(\theta^Q)$ between outputs of the target critic network $Q'$ and the critic network $Q$.

After the critic network and the actor network are updated by the transitions from the experience buffer, the target actor network and the target critic network are updated as follows:
\begin{align}
    \theta^{Q'} &\leftarrow \tau \theta^Q + (1-\tau)\theta^{Q'},
    \\
    \theta^{\mu'} &\leftarrow \tau \theta^\mu + (1-\tau)\theta^{\mu'},
\end{align}

where $\tau$ denotes learning rate.

\section{Performance Evaluation}
We evaluate the performance of the adaptive DDPG algorithm in this section. The results show that the adaptive DDPG agent achieves higher return than the vanilla DDPG, Dow Jones Industrial Average (DJIA) and the traditional portfolio allocation strategies.
\subsection{Data Preprocessing and Comparison Methods}
We selected the constituent stocks of the Dow Jones index as our stock pool. The time span of the data (daily prices) is from 01/01/2001 to 09/30/2018. The dataset is downloaded from Compustat database accessed through Wharton Research Data Services (WRDS). The dataset from 01/01/2001 to 12/30/2013 (including 3268 trading days) is used as the training data, and the remaining dataset (from 01/02/2014 to 10/02/2018 including 1190 trading days) is used as the testing data. We train our agent on training data and test the agent's performance on testing data.

Figure~\ref{figure:market} shows the market index during the testing data and the corresponding learning rate, i.e., we set $\alpha^{+} = 1$ and $\alpha^{-} = 0$. And we set ${\mathcal{N}}^{+}$ as the normal random process and ${\mathcal{N}}^{-}$ as a random process that only generates negative values.

\begin{figure}[!tb]
	{\includegraphics[width=3.3in]{market.pdf}}
	\caption{The market index during the testing data and the corresponding learning rate.}
	\label{figure:market}
\end{figure}

We use the 30 stocks data's daily price to train the adaptive DDPG agent. Then, we run the agent on testing data and compare performance with the vanilla DDPG,  DJIA and the min-variance and mean-variance portfolio allocation strategies. We use final portfolio value, annualized return, annualized standard error and the Sharpe ratio to evaluate the proposed method. Final portfolio value reflects the overall effect of investing in a certain time range. Annualized return is the geometric average amount of money earned by an investment each year over a given time period. Annualized standard error reflects the volatility and shows the robustness of the model.  The Sharpe ratio (the return earned per unit volatility) is used to evaluate the portfolio’s performance~\cite{sharpe1994sharpe}.

\begin{figure*}[!htb]
	\centering
	{\includegraphics[width=7.0in]{result.pdf}}
	\caption{Portfolio allocation returns of the proposed adaptive DDPG and traditional methods.}
	\label{figure:result}
\end{figure*}

\begin{table*}
\centering
\begin{tabular}{cccccc}
\toprule
Method  & \textbf{Adaptive DDPG} & DDPG & DJIA & Min-variance & Mean-variance\\
\midrule
Initial value     & \textbf{10,000} & 10,000  & 10,000 & 10,000 & 10,000\\
Final value     & \textbf{21,880} & 18,156 & 16,089 & 16,333 & 19,632 \\
Annualized return      & \textbf{18.84\%} & 14.71\% & 11.36\% & 11.48\% & 15.86\% \\
Annualized Std. error       & \textbf{11.59\%}  & 14.68\% &12.43\% & 11.64\% & 12.70\% \\
Sharpe ratio  & \textbf{1.63}  & 1.01 & 0.91  & 0.99 & 1.25\\
\bottomrule
\end{tabular}
\caption{Trading Performance.}
\label{tab:results}
\end{table*}

\subsection{Performance Results}
Figure~\ref{figure:result} shows that the adaptive DDPG model is significantly better than the vanilla DDPG, the Dow Jones Industrial Average and the traditional portfolio allocation strategies. And we can see that the DDPG strategy is better than the Dow Jones Industrial Average and the traditional portfolio allocation strategies.

As can be seen from Table~\ref{tab:results}, the Adaptive DDPG achieves an annualized rate of return of 18.84\%, which is much higher than the vanilla DDPG of 14.71\%, the Dow Jones Industrial Average of 11.36\%, and the min-variance and mean-variance portfolio allocations respectively of 11.48\% and 15.86\%. The annualized Sharpe ratio of the adaptive DDPG strategy is also higher, indicating that the adaptive DDPG strategy is superior. Therefore, the results show that the adaptive DDPG strategy can effectively develop a matching strategy that is superior to the vinilla DDPG, benchmark Dow Jones industrial average and the traditional portfolio allocation methods.

\section{Conclusions}
In this paper, we propose an Adaptive Deep Deterministic Policy Gradient (Adaptive DDPG) scheme for portfolio allocation task. The Adaptive DDPG incorporates optimistic or pessimistic deep reinforcement learning, which allows the amplitude of the up-date to be different according to the positive or negative prediction errors. Experiment results based on the Dow Jones stocks show that the proposed Adaptive DDPG model can get a better portfolio allocation strategy under different market conditions. Portfolio return results show that the investment return can be significantly improved based on our Adaptive DDPG.

Future work will be interesting to explore more advanced model and deal with larger scale data \cite{burda2018large} to incorporate with price prediction and anomaly detection schemes \cite{Xinyi2019KDD}, and to improve the robustness of machine learning algorithms \cite{yang2018practical}. We also want to do some text analysis, such as extracting text information from real-time news or social networks into the model for analysis \cite{hu2018listening}.

\nocite{langley00}