You will be presented with the full text of a science research paper. Please write an abstract for this paper. Your response should include the abstract and no additional text.

Paper text:
understanding the mechanisms of learning and maintenance of effective behavior in stochastic environments with complex hierarchy of goals is one of key issues in the study of neural processing and for the development of bio - inspired artificial intelligence .
however , these mechanisms are better understood if history of their development is taken into account .    fields of machine learning and adaptive behavior have been rapidly developing in the recent decades .
one of the most popular approaches in these areas is reinforcement learning ( rl ) @xcite and its modifications . it can be effectively used for learning of autonomous agents in many domains .
however , problem spaces with multiple goals require execution of alternative behavioral strategies that are not possible for rl algorithms , since the behavior of the agent is determined by the single value function .
also , it is not always possible to define appropriate value function _ a priori_. to address this issue approaches for the generation of reinforcement values using evolutionary algorithms are currently under development @xcite .
other algorithms are also unable to produce effective solution for the environments with hierarchy of goals @xcite .
the various methods designed for the generation of behavioral sequences @xcite in most cases can not effectively work in situations with a large number of goals present in the environment and when an agent has to autonomously explore them .
neuroevolutionary approaches have been successfully used for the synthesis of autonomous agent s controllers in low - formalized tasks @xcite .    to solve the problem of alternative behaviors
the algorithm should implement non - markov decision process in opposite to reinforcement learning @xcite .
in other words for selection of different actions at the same state of the environment an agent should have memory of previous actions .
short - term memory ( stm ) is extensively covered in the area of recurrent neural networks research in terms of the signal reverberation @xcite and serial order recall @xcite .
theorists suggested reverberation phenomenon as significant mechanism for stm @xcite . nevertheless neural principles underlying generation of the behavior as a non - linear integration of the sensory information and internal state of the network are not yet discovered . as well
the question of the role stm plays in the adaptive behavior with alternative actions is still underexplored .    in this paper
we present a model of an agent situated in an environment with hierarchy of goals .
the behavior of the agent is controlled by recurrent neural network .
we simulated evolution of agents in the model world and studied emergence of short - term memory and its role in adaptive behavior .
in the current study a state of an environment in which an autonomous agent `` lives '' is represented by binary vector :    @xmath0    at any discrete time step the agent may change a single bit of this vector to the opposite .
thus , environmental structure is a @xmath1-dimensional hypercube .
every dimension of this hypercube might be interpreted as some feature of the real environment , for instance temperature , pressure , etc .
competing goals of different complexity are prespecified in the environment as an ordered set of single bit changes of the environmental state vector :    @xmath2    where @xmath3 is a number of target bit of the state vector , @xmath4 is target value of the bit , @xmath5 is a complexity of the goal
. goals of varying complexity are defined in the environment , together forming a branched hierarchical structure .
we have generated structure of the environment by , firstly , composing hierarchy of full goals and then each goal was divided into sequence of subgoals .
to determine the complexity of a particular environment we introduce occupancy based on a probabilistic approach :    @xmath6    where @xmath7 is the number of goals in an environment , @xmath8 is the number of unique bits of the state vector , changed during the process of achieving a goal ( single bit can be changed several times ) .
the value , which is opposite to the occupancy , we shall call the difficulty of the environment :    @xmath9    reward is associated with each goal in the environment and directly proportional to the complexity of corresponding goal . after the agent reached goal , level of accruing reward associated with this goal is reset and then linearly recovers to the original value over the time @xmath10
.    environment can be either deterministic or non - deterministic . in the later changes of the state vector
occur not only when the agent performs actions , but also stochastically according with a fixed probability .
agent s behavior in the environment is controlled by artificial neural network ( ann ) of arbitrary topology , which develops through the evolutionary process .
ann is divided into input , output and number of hidden layers .
the network consists of mcculloch - pitts neurons with nonnegative logistic activation function .
signal passes through the synapse only when output value of a presynaptic neuron is above a certain activation threshold ( was set to 0.5 in all simulations ) .
thus , we introduce two possible states for each neuron : active ( when output is above a threshold ) and inactive ( otherwise ) .
the current state vector of the environment is directly fed to the neurons of input layer of the network .
hidden layers allow existence of recurrent connections with delayed transmission .
pair of the most active neurons of output layer encodes agent s action at the current time step .
each pair stands for a particular bit of the state vector and target value .
thereby transfers of a bit from 0 to 1 and vice versa are encoded by different pairs of the output neurons .
the agent could produce ineffective actions , when attempts to change a state of a bit to one in which it already is .
population of agents evolves in the hypercube environment described in previous section .
every individual is separately placed in the environment to evaluate fitness . for a given time the agent operates in the environment , reaching goals and accumulating reward .
value of accumulated reward is inaccessible to the agent and total reward affects its reproductive success .    to evolve agents anns we used neuroevolutionary algorithm based on duplication of neurons .
this algorithm is similar to well - known neat @xcite as it makes possible evolution of neural network topology but in more natural way by duplicating neurons with attached connections .
specifically we use _
neuron s duplication _ mutation when duplicated neuron inherits from parent the whole synaptic structure instead of _ add node _ mutation of neat .
incoming connections of the descendant and parent neurons retain its previous weights , while weights of outgoing connections divided in half for both neurons .
thus , two neurons in the aggregate perform the former function , but later in evolution the descendant neuron could diverge into a separate structure . to optimize evolution by
dynamically reducing dimensionality of the search space we also introduced _ delete connection _ mutation , which performed in the same manner as _ add connection _ mutation .
neurons that lost all their connections are being deleted from the network .
we did not use any crossover algorithm , since preliminary results of comparison between two versions of evolutionary algorithm ( with crossover and without ) have not showed significantly difference .
the structure of the agents networks in initial population consists of input and output layers and one additional interneuron , which will be an ancestor for all future interneurons .
this initial interneuron is fully connected to input layer ( by incoming connections ) and to output layer , herewith there are no initial direct connections between input and output neurons .
we used the following parameters of the artificial environment in all simulations : dimensionality of the hypercube environment @xmath1  8 bits ; agent lifetime duration @xmath11  250 time steps ; reward s recovery time @xmath10  30 time steps ; probability of random change of the state vector s bit @xmath12  0.0085 ( for each bit ) .
evolutionary algorithm was running with parameters : population @xmath13  250 agents ; period of evolution @xmath14  5000 generations ; probability of the synaptic weight mutation @xmath15  0.6 ( for each synapse ) ; variance of the synaptic weight mutation @xmath16  0.08 ; probability of adding the synapse @xmath17  0.1 ( for whole network ) ; probability of deleting the synapse @xmath18  0.05 ( for whole network ) ; probability of neuron s duplication @xmath19  0.007 ( for whole network ) .
as the first step we studied how occupancy of the environment affects efficiency of evolved behavior in deterministic and stochastic environments .
as expected the simulation results show the growth of an average cumulative reward with increase of the occupancy , reflecting that a task is easier when the environment is more densely populated with goals ( fig .
[ fig : statevolution ] ) .
agents evolved in non - deterministic environments behave significantly more successfully ( fig .
[ fig : statevolution ] ) in terms of average accumulated reward and in most cases have richer repertoires of behavior .
such phenomenon can be explained by the fact that during the `` life '' in an environment which could suddenly change the agent with a wider spectrum of behavioral policies should be more successful .
thus stochasticity of the environment contributes to the selection of agents with more flexible and robust behavior , allowing them to reach greater number of goals from different initial position .
however , with increasing probability of random changes in the environment one can observe a sharp decline in the efficiency of the evolution ( data not shown ) due to the destabilization of all policies caused by excessively frequent changes in the environment .
we then analyzed evolution of behavioral sequences of the agents in the environment of intermediate goal occupancy .
typical evolved behavior usually consists of two phases : preliminary phase of converging to the main behavioral cycle and then residing on it . in a particular behavioral run
both of these phases depend on the initial state of the environment , from which the agent has been started .
we shall call the main behavioral cycle ( sequence of actions or attained goals ) a behavioral strategy
.    results of simulations demonstrate that sharp increase in the average cumulative reward among the population usually coincides with emergence of a large number of new cycles and their dynamic competition .
corresponding periods are the most interesting in the study of behavioral evolution . on the fig .
[ fig : behevolution ] we give an example of such period .
one can observe competition between different behavioral strategies during a very limited evolutionary period .
herewith new strategies are appearing as an extension of previously evolved cycles .
extension can be implemented not only as simple addition of a new part to an old one , but also as complex compilation of two strategies .
eventually one of the strategies wins the competition and dominates in the behavior of the most of the population .
then evolution occurs through increase in number of states , from which it could be implemented .
the analysis of behavioral strategies emerging in the simulated evolution shows that the agents acquire ability to store short - term memory due to reverberation in the neural network by recurrent connections .
the evidence for short - term memory comes from the fact that agents can implement policies based on alternative actions .
agents usually can perform different actions from one state of the environment depending on the previous history of behavior .
state transition diagram for the exemplar evolved behavior with three alternative actions is shown on the fig .
[ fig : alternativebehavior ] .
such phenomenon would not been possible in the case of reactivity work of the neural network .
corresponding behavior is generated by a neural network consists of 30 neurons with only 15 interneurons and 611 synaptic connections .
only a small number of neurons significantly change output and affect the decision making at states associated with alternative situations .
this group of neurons is specialized on particular alternative actions .
the majority of neurons remain at the same level of activity , while neurons which determine a behavior change their activity from zero to maximum level .
based on the various behavioral policies the deepest short - term memory we found was at least 4 past states ( lower bound ) .
such conclusion was made after considering situations in which alternative actions are performed with the same history of behavior and , thus , the lower bound can be determined as the first different action of the two behavioral sequences .
the ability to use short - term memory makes possible implementation of much more complex behavior and accumulation of more reward during the agent s `` life '' .
example of a behavioral strategy , based on the alternation of two cycles of actions , is shown on the fig .
[ fig : behstrategy ] .
since after the goal is reached it restores the reward for a fixed number of time steps , such a strategy allows goals that were reached on the first cycle to restore their value , while the agent passes the second cycle .
behavior on the fig .
[ fig : behstrategy ] can be implemented using short - term memory with depth of two states .    primitive alternative behavior in the model
could also be supported by slow neurodynamical process ( fig .
[ fig : slowprocess ] ) . in this case
the `` memory '' , required to perform alternative actions , could be up to 30 previous states .
the mechanism underling this type of behavior is oscillatory dynamic of the neuron s output .
while output of a neuron is above threshold , and therefore transmitted by outgoing connections , agent performs the first part of behavior . at some point output falls below the transmission threshold and the neuron no longer inhibits the other neuron , which activity is essential for the second branch of behavior .        in the records of neural network activity ( fig .
[ fig : rasterplot ] ) during the entire period of autonomous agent s behavior one can identify both neurons , that are active while performing most of the actions , and neurons , that are active only in a very limited number of states of the environment ( sometimes even in a single state ) .
in our study we have explored possibility of evolutionary emergence of short - term memory in recurrent neural networks controlling agents behavior in multi - goal environments . on the first stage we analyzed overall dynamics of a model . as expected , performance of agents was worse when the number of goals in environment was smaller . on the other hand it turned out surprisingly that in stochastic environments agents with more flexible and stable behavioral policies evolve and able to accumulate more reward in comparison to agents in the same environment but without noise .
this phenomenon is quite unusual for machine learning algorithms @xcite , since in the case of non - deterministic environment the performance is usually worse .
detailed investigation of evolutionary dynamics revealed that the evolution of agents consisted of two stages : rapid emergence / development of new behavioral strategies and their propagation among the population .
formation of new strategies was usually manifested as complex integration of few already existed .    on the level of individual behavior we found that evolution develops strategies , which are based on alternative actions .
that happens due to acquisition of ability to operate with short - term memory and , therefore , to select actions taking into account previous history of behavior .
evolution discovered two different neuronal mechanisms for implementation of alternative actions : first one is based on integration of sensory information and internal signal , which is reverberating through recurrent connections ; and the second is based on slow neurodynamical oscillatory processes . in fact , there is no need to recruit synaptic plasticity during the agent s `` life '' to have an effective use of short - term memory .
it is important to notice that emergence of the ability to operate with short - term memory in our model occurs without any artificial prerequisites in the structure of an evolutionary algorithm .
the next stage of presented study is introduction of learning during agent s `` life '' , since it makes significant contribution into development of adaptive behavior .
the main mechanism of learning algorithm is detection on the neuronal level of problems for the whole organism situated in environment .
possible approach to the construction of such algorithm is the use of research in the theoretical neuroscience @xcite .
singh , s. , lewis , r. , barto , a.g .
: where do rewards come from ? in : taatgen ,
n.a . , van rijn , h. ( eds . ) , proceedings of the 31st annual meeting of the cognitive science society , pp .
austin , tx : cognitive science society ( 2009 )          floreano , d. , mondana , f. : automatic creation of an autonomous agent : genetic evolution of a neural - network driven robot . in : cliff , d. , husbands , p. , meyer , j
.- a . , wilson , s.w .
proceedings of the 3rd international conference on simulation of adaptive behavior , pp . 421430 ( 1994 )        hochreiter , s. , bengio , y. , frasconi , p. , schmidhuber , j. : gradient flow in recurrent nets  the difficulty of learning long - term dependencies .
a field guide to dynamical recurrent neural networks , ieee press , pp .
237243 ( 2001 )
animals behave adaptively in the environment with multiply competing goals . 
 understanding of the mechanisms underlying such goal - directed behavior remains a challenge for neuroscience as well for adaptive system research . to address this problem we developed an evolutionary model of adaptive behavior in the multigoal stochastic environment . 
 proposed neuroevolutionary algorithm is based on neuron s duplication as a basic mechanism of agent s recurrent neural network development . 
 results of simulation demonstrate that in the course of evolution agents acquire the ability to store the short - term memory and , therefore , use it in behavioral strategies with alternative actions . 
 we found that evolution discovered two mechanisms for short - term memory . 
 the first mechanism is integration of sensory signals and ongoing internal neural activity , resulting in emergence of cell groups specialized on alternative actions . and 
 the second mechanism is slow neurodynamical processes that makes possible to code the previous behavioral choice .    
 behavior , neuroevolution , alternative behavior , recurrent neural networks , short - term memory