You are presented with multiple paper abstracts, alongside the associated literature review and chapter titles. Given the provided information, you are going to do a literature review. Please generate a coherent literature review that synthesizes the content, draws connections between different works, highlights major findings, and identifies any gaps in the existing research.

Literature Review Title: Reinforcement learning in robotics: A survey 
Chapter Title:  Hierarchical reinforcement learning 
1. Abstract of Cited Paper (BIB001):  Abstract In this paper, we propose a hierarchical reinforcement learning architecture that realizes practical learning speed in real hardware control tasks. In order to enable learning in a practical number of trials, we introduce a low-dimensional representation of the state of the robot for higher-level planning. The upper level learns a discrete sequence of sub-goals in a low-dimensional state space for achieving the main goal of the task. The lower-level modules learn local trajectories in the original high-dimensional state space to achieve the sub-goal specified by the upper level. We applied the hierarchical architecture to a three-link, two-joint robot for the task of learning to stand up by trial and error. The upper-level learning was implemented by Q-learning, while the lower-level learning was implemented by a continuous actor–critic method. The robot successfully learned to stand up within 750 trials in simulation and then in an additional 170 trials using real hardware. The effects of the setting of the search steps in the upper level and the use of a supplementary reward for achieving sub-goals are also tested in simulation. 
2. Abstract of Cited Paper (BIB002):  This paper presents an optimal controller for an Instantaneously Coupled System (ICS) which was designed by coordinating multiple lower-dimensional optimal controllers. We augmented subsystems of the ICS with coordination variables, and then used value functions to coordinate the augmented subsystems by managing tradeoffs of the coordination variables. We apply this method to humanoid walking and present a controller for a 3D simulation that uses multiple coordinated policies generated using Dynamic Programming. Additionally, we present simulated walking perturbation experiments as well as standing balance results from a force-controlled humanoid robot. 
3. Abstract of Cited Paper (BIB003):  In this paper, we present a framework for guiding autonomous learning in robot systems. The paradigm we introduce allows a robot to acquire new skills according to an intrinsic motivation function that finds behavioral affordances. Affordances-in the sense of (Gibson, Toward and Ecological Psychology, Hillsdale, NJ, 1977)-describe the latent possibilities for action in the environment and provide a direct means of organizing functional knowledge in embodied systems. We begin by showing how a robot can assemble closed-loop action primitives from its sensory and motor resources, and then show how these primitives can be sequenced into multi-objective policies. We then show how these policies can be assembled hierarchically to support incremental and cumulative learning. The main contribution of this paper demonstrates how the proposed intrinsic motivator for affordance discovery can cause a robot to both acquire such hierarchical policies using reinforcement learning and then to generalize these policies to new contexts. As the framework is described, its effectiveness and applicability is demonstrated through a longitudinal learning experiment on a bimanual robot. 
4. Abstract of Cited Paper (BIB004):  Future robots need to autonomously acquire motor skills in order to reduce their reliance on human programming. Many motor skill learning methods concentrate on learning a single solution for a given task. However, discarding information about additional solutions during learning unnecessarily limits autonomy. Such favoring of single solutions often requires re-learning of motor skills when the task, the environment or the robot's body changes in a way that renders the learned solution infeasible. Future robots need to be able to adapt to such changes and, ideally, have a large repertoire of movements to cope with such problems. In contrast to current methods, our approach simultaneously learns multiple distinct solutions for the same task, such that a partial degeneration of this solution space does not prevent the successful completion of the task. In this paper, we present a complete framework that is capable of learning different solution strategies for a real robot Tetherball task. 
5. Abstract of Cited Paper (BIB005):  Learning new motor tasks autonomously from interaction with a human being is an important goal for both robotics and machine learning. However, when moving beyond basic skills, most monolithic machine learning approaches fail to scale. In this paper, we take the task of learning table tennis as an example and present a new framework which allows a robot to learn cooperative table tennis from interaction with a human. Therefore, the robot first learns a set of elementary table tennis hitting movements from a human teacher by kinesthetic teach-in, which is compiled into a set of dynamical system motor primitives (DMPs). Subsequently, the system generalizes these movements to a wider range of situations using our mixture of motor primitives (MoMP) approach. The resulting policy enables the robot to select appropriate motor primitives as well as to generalize between them. Finally, the robot plays with a human table tennis partner and learns online to improve its behavior.

Literature Review:
A task can often be decomposed into different levels. For example when using meta-actions (Section 4.1), these meta-actions correspond to a lower level dealing with the execution of sub-tasks which are coordinated by a strategy level. Hierarchical reinforcement learning does not assume that all but one levels are fixed but rather learns all of them simultaneously. For example, hierarchical Q-learning has been used to learn different behavioral levels for a six-legged robot: moving single legs, locally moving the complete body, and globally moving the robot towards a goal BIB001 . A stand-up behavior considered as a hierarchical reinforcement learning task has been learned using Q-learning in the upper-level and a continuous actor-critic method in the lower level BIB002 . Navigation in a maze can be learned using an actor-critic architecture by tuning the influence of different control modules and learning these modules  . Huber and Grupen (1997) combined discrete event system and reinforcement learning techniques to learn turning gates for a quadruped. BIB004 learned bi-manual manipulation tasks by assembling policies hierarchically. BIB005 learned options in a tetherball scenario and BIB006 learned different strokes in a table tennis scenario. BIB003 showed that the optimal policy for some global systems (such as a walking controller) can be constructed by finding the optimal controllers for simpler subsystems and coordinating these.