You will be presented with the abstract, introduction, and discussion section from a biomedical research article. Your task is to create a summary that a layperson can understand, capturing the essence of the research article. Keep essential scientific terms, but ensure that the language remains clear and the concepts are explained in an uncomplicated manner.

Title: Multi-layer network utilizing rewarded spike time dependent plasticity to learn a foraging task}
Article:
Abstract:
Neural networks with a single plastic layer employing reward modulated spike time dependent plasticity( STDP) are capable of learning simple foraging tasks. Here we demonstrate advanced pattern discrimination and continuous learning in a network of spiking neurons with multiple plastic layers. The network utilized both reward modulated and non-reward modulated STDP and implemented multiple mechanisms for homeostatic regulation of synaptic efficacy, including heterosynaptic plasticity, gain control, output balancing, activity normalization of rewarded STDP and hard limits on synaptic strength. We found that addition of a hidden layer of neurons employing non-rewarded STDP created neurons that responded to the specific combinations of inputs and thus performed basic classification of the input patterns. When combined with a following layer of neurons implementing rewarded STDP, the network was able to learn, despite the absence of labeled training data, discrimination between rewarding patterns and the patterns designated as punishing. Synaptic noise allowed for trial-and-error learning that helped to identify the goal-oriented strategies which were effective in task solving. The study predicts a critical set of properties of the spiking neuronal network with STDP that was sufficient to solve a complex foraging task involving pattern classification and decision making.
Introduction:
Biologically inspired neural networks should be capable of performing sophisticated information processing that the human and animal brains can perform. Information processing by the brain is deeply multilayered and involves many sequential steps before sensory information can be interpreted and translated into behavior. What makes this cascade powerful and unique is its capability to learn and respond to an ever changing environment. In the most studied sensory pathway—the visual one—the sensory input gets progressively more general, though the stages at which visual learning occurs are still a matter of controversy and different plasticity mechanisms might be operating at different processing steps[1]. Eventually sensory information reaches decision centers( such as lateral intraparietal cortex) which govern behavior and those centers are under the influence of reward signals[2, 3]. It has been shown that reinforcement based on the reward can be vital to visual learning[4]. To what extent reward influences learning in the sensory cortices seems to be task dependent and different studies showed both learning with[5, 6] and without[7, 8] the presence of reward. While a great deal of research has gone into understanding the mechanisms of learning, it is still not fully known how learning at the cellular level gives rise to the learning at the level of animal behavior. Most learning models successfully employ hebbian type learning principles formulated as an abstract rule[9] without being concerned with the cellular level mechanisms. The most promising model of learning at the synaptic level is spike time dependent plasticity( STDP)[10]. In STDP, when a presynaptic cell fires shortly before a post synaptic cell, the synapse between them increases in strength. If the postsynaptic cell fires before the presynaptic cell, the synapse decreases in strength. Such basic form of STDP is only capable of unsupervised learning. By storing STDP events in the form of synaptic tags[11], a delayed reward signal can enable a network to perform reinforcement learning[12]. This mechanism relies on the dopamine modulation of synaptic tags created by STDP[13, 14, 15]. A small number of studies have attempted to perform task learning using rewarded STDP including studies that attempted to maintain the stability of a virtual robotic arm[16, 17]. In theoretical studies, STDP commonly leads to the net increase of synaptic strength across the network[18], causing large numbers of synapses to climb toward their maximum values. This would result in a highly dysfunctional brain and can eventually promote an epileptic state[19]. To prevent runaway synaptic dynamics, various regulatory synaptic mechanisms maintain the distribution of synaptic weights in biological networks. This includes homeostatic mechanisms when a group of cells that experiences high firing frequencies over a prolonged period becomes less responsive to excitatory input by effectively reducing incoming excitatory synaptic weights[20]. Another regulatory mechanism, heterosynaptic plasticity, responds to the increase of a synaptic weight by reducing all other incoming synaptic strengths of the same post synaptic neuron[21, 22, 23, 24] preventing neurons from becoming overactive[25, 26, 27, 28]. In this new work, we built a model containing two plastic layers inspired by the pathways known in mammalian brain from sensory to higher order sensory-motor areas responsible for decision making[29, 4]. The first layer of the model uses unsupervised( unrewarded) learning to classify the input while the second layer( based on rewarded STDP) is responsible for decision making. As a whole the network simulated the brain of an agent moving through an unknown environment, continuously learning distinct input patterns of food and adjusting synaptic weights controlling its movement according to the reward and punishment signals based on the shape of the different configurations of food particles that it acquires. We demonstrate that such multilayered network combining rewarded and nonrewarded STDP and synaptic regulatory mechanisms is capable of solving a more complex foraging task, involving discrimination between elementary patterns of food, than it was previously reported in the networks with only one plastic layer based on rewarded STDP[30].
Discussion:
While STDP has been previously applied in the network models to explain pattern recognition[36], and rewarded STDP has been shown to be capable of overcoming the distal reward problem[12, 37, 38, 39, 16, 17], our study combined both types of synaptic plasticity in a single model to achieve high learning performance in discrimination and decision making tasks. By punishing the network when incorrect decision( wrong type of food is acquired) was made, the network was required( a) to identify the input pattern( food type) by recognizing the arrangement of the food participles and( b) to learn to move toward the rewarded pattern( horizontal arrangement of food particles) and away from the punished pattern( vertical arrangement of food particles). It has been shown that STDP is most relevant to cells firing in the mid range frequencies, while other rules, depending primarily on the presynaptic firing rate, may become important for the networks operating in the high frequency range[40, 41, 42]. Combining both rate-based and time-based plasticities gives rise to more complex network connectivity patterns[43, 44] which could potentially be used for more complex tasks. Our model does not employ the rate-based plasticity but future extensions of our model may incorporate a combination of these complementary forms of synaptic learning. The idea of synaptic strength normalization was formulated shortly after Hebb’s seminal work[45]. It was shown that for a successful Hebbian cell assembly to develop, the total sum of input synaptic weights has to be held near constant in order to prevent runaway dynamics[46, 47]. This leads to a strong need for extensive homeostatic regulation to prevent pathological synchronized activity and indeed there is evidence that homeostatic mechanisms are deeply involved in both preventing and causing epileptic conditions[48, 49, 50, 51]. Such normalization rules became an integral part of modeling studies of supervised learning and cortical map formation[52]. In our model, maintaining the balance of synaptic strength within the network was vital to learning and to achieving high model performance. However, not all synaptic rules contributed equally to the final performance of the model. By far the most important was heterosynaptic plasticity which reduces( increases) the strength of all other synapses when one synapse increases( reduces) in strength by homosynaptic plasticity. Such a tendency towards maintaining the net input to a neuron has already been found in experimental studies[53, 54, 26, 27] and has been tested in detailed biophysical models[25, 28]. Its potential mechanism may depend on the intracellular Ca2+ dynamics which is influenced by the backpropagating action potentials[55, 56] at the synapses which do not actively participate in homosynaptic plasticity induction[24, 23, 22, 57, 21]. We also found that the model could only operate efficiently when the net network activity remained within a certain range. If the network became overactive, high levels of activity in the output layer made direction selectivity highly inconsistent. If the network activity was too low, not enough input would be received by the output layer to cause spiking, leading to poor performance as well. Control of the long term average network activity can be achieved by homeostatic scaling[58, 59], which adjusted the strength of synaptic inputs up for the neurons with low average firing rates and down for the neurons with high average activity. Whenever there was variation in food density near the agent, it was important to maintain comparable firing rates in the output neurons. Keeping a functional range of this activity from one trial to the next required that the network activation did not increase in direct proportion to the input activity. As the highly active and more consistently rewarded synapses regularly increased in strength, it was possible for certain neurons to gain a major control over the network dynamics. Although it was not necessary that all neurons be equally represented, having a small number of neurons exert much greater control over the network had a pronounced negative effect on performance. Output-side synaptic balancing allowed us to reduce inequalities in the representation of the neurons and created competition between the outputs of a neuron as was proposed in[30]. This was particularly useful when a middle-layer neuron had two( or more) strong connections to the output neurons, thus “supporting” two different directions of movement. If such middle-layer neuron received a reward, it would increase strength of both connections—despite the fact that only one of them was actually responsible for correct food acquisition. Output-side balancing allowed only synapses correlated with reward to be strengthened via rewarded STDP traces. While we do not know direct experimental evidence for this mechanism, a similar effect can be potentially achieved by more complex circuits involving reciprocal inhibition between output neurons. We observed that by normalizing the amount of potentiaton in STDP( based on the history of STDP traces occurring in the synapse past), we increased the speed of initial learning and increased the stability of efficiency gains for the network in the long term. Though not directly observed, this mechanism is consistent with experimental data[32, 33, 34] and might be of interest for additional experimental scrutiny. Interaction between synaptic excitation and inhibition is a ubiquitous phenomenon found across many different brain circuits[60, 61, 62, 63, 64, 65, 66, 67]; it can be observed during spontaneous activity of the brain[68, 69, 70] and is an integral part of the oscillatory mechanisms in the neuronal networks[71]. To simplify implementation of the inhibitory effects in our model, the neurons of the preceding layer have sent inhibitory connections to the same set of post synaptic cells they excited in the following layer. This essentially provided feedforward inhibition between layers of neurons, using a population of virtual inhibitory neurons that was not explicitly modeled. The total strength of inhibitory connections from any given cell was the same as the total of its excitatory connections. While not biologically realistic, this simplification effectively simulated a feed forward inhibition found in various brain areas[72] where neurons in one layer project to both excitatory and inhibitory neurons in the down stream layer and inhibitory neurons in turn project to excitatory neurons in the same layer[73]. As inhibitory neurons are often more electrically compact[74] and have a faster membrane time constant, this can result in the inhibition arriving to the post synaptic excitatory neurons with only a small delay compared to the excitation. No significant difference was seen in performance of this network as a result of this simplified implementation of inhibition when compared to the previous model where inhibition was implemented explicitly[30]. Noise is another ubiquitous phenomenon in the neural networks and spans multiple time scales and different physical domains[75]. In contrast with artificially crafted devices where noise is usually detrimental to functioning, biological neural networks are capable of using noise in a productive way to improve efficiency and performance[76, 77]. Examples of this include enhanced input detection by means of stochastic resonance[78, 79, 80, 81] and probabilistic inference[82, 83]; noise can help with performance of the associative memories[84], smoothing the threshold for action potentials[85, 86], allowing rapid response time for neuronal populations[87, 88] or faithful propagation of firing rates across a layered network[89]. The largest source of the noise in any biological neuronal network is synaptic noise[90]. In our model we employed noise at the level of synaptic currents that represented variability in synaptic vesicle release. Noise was not only tolerated by this model but it was required for its function. Before training there was no meaningful mapping between the input received by the network and its output; the output activity( and therefore direction of movement) was essentially driven by random initial configuration of synaptic weights. Variability( noise) in synaptic release occasionally caused an unexpected output for a given input. When the network output resulted in food acquisition, it was reinforced by a reward. If a synapse that was likely to be associated with a reward was able to surpass the strength of another synapse of the same presynaptic cell, noise in the synaptic output created instances where the “correct” output cell fired and the “incorrect” output cell did not, despite the “incorrect” output cell having a stronger incoming synaptic strength. This effectively allowed for a form of trial and error learning. Without such variety of the actions attempted, it would be difficult to find which goal-oriented strategies are effective and it can be seen as representing a strategy of associating a given stimulus with a given response. Unlike common machine learning approaches or the currently booming deep network architectures[91] which, in some cases, have already outperformed humans[92, 93, 94, 95, 96], in this study we only applied biologically plausible and experimentally identified plasticity rules. This new study, following our previous work[30], generated specific predictions regarding what biological learning mechanisms are necessary and/or sufficient to accomplish the learning task. The key components for a successfully working model were STDP, synaptic balancing processes keeping the learning stable, dopamine type reward feedback derived from successful outcome of action and noise at the synaptic level allowing trial-and-error learning. Although these components have already been studied, sometimes in great detail, our study links them together( unsupervised STDP for sensory learning and rewarded STDP for decision making in particular) in a new way and shows advantages of interaction between different plasticity mechanisms for complex task solving.
This study explores how intelligent behavior emerges from the basic principles known at the cellular level of biological neuronal network dynamics . Compared to the approaches used in the artificial intelligence community , we applied biologically realistic modeling of neuronal dynamics and plasticity . The building blocks of the model are spiking neurons , spike-time dependent plasticity ( STDP ) and homeostatic rules , known experimentally , which are shown to play a fundamental role in both keeping the network stable and capable of continous learning . Our study predicts that a combination of these principles makes possible a foraging behavior in a previously unknown environment , including pattern classification to distinct between environment shapes which are rewarded and those which are punished and decision making to select the optimal strategy to acquire the maximal number of the rewarded elements . To solve this complex task we used multi-layer neuronal processing that implemented pattern generalization by unsupervised STDP at the earlier processing step , as commonly observed in the animal and human sensory processing , followed by reinforcement learning at the later steps . In the model , the intelligent behavior emerged spontaneously due to the network organization implementing both local unsupervised plasticity and reward feedback resulting from a successful behavior in the environment .