You are presented with multiple paper abstracts, alongside the associated literature review and chapter titles. Given the provided information, you are going to do a literature review. Please generate a coherent literature review that synthesizes the content, draws connections between different works, highlights major findings, and identifies any gaps in the existing research.

Literature Review Title: Energy management for user’s thermal and power needs: A survey 
Chapter Title:  Approaches: Heuristic techniques 
1. Abstract of Cited Paper (BIB001):  \cal Q-learning (Watkins, 1989) is a simple way for agents to learn how to act optimally in controlled Markovian domains. It amounts to an incremental method for dynamic programming which imposes limited computational demands. It works by successively improving its evaluations of the quality of particular actions at particular states. ::: ::: This paper presents and proves in detail a convergence theorem for \cal Q-learning based on that outlined in Watkins (1989). We show that \cal Q-learning converges to the optimum action-values with probability 1 so long as all actions are repeatedly sampled in all states and the action-values are represented discretely. We also sketch extensions to the cases of non-discounted, but absorbing, Markov environments, and where many \cal Q values can be changed each iteration, rather than just one. 
2. Abstract of Cited Paper (BIB002):  Metaheuristics are general algorithmic frameworks, often nature-inspired, designed to solve complex optimization problems, and they are a growing research area since a few decades. In recent years, metaheuristics are emerging as successful alternatives to more classical approaches also for solving optimization problems that include in their mathematical formulation uncertain, stochastic, and dynamic information. In this paper metaheuristics such as Ant Colony Optimization, Evolutionary Computation, Simulated Annealing, Tabu Search and others are introduced, and their applications to the class of Stochastic Combinatorial Optimization Problems (SCOPs) is thoroughly reviewed. Issues common to all metaheuristics, open problems, and possible directions of research are proposed and discussed. In this survey, the reader familiar to metaheuristics finds also pointers to classical algorithmic approaches to optimization under uncertainty, and useful informations to start working on this problem domain, while the reader new to metaheuristics should find a good tutorial in those metaheuristics that are currently being applied to optimization under uncertainty, and motivations for interest in this field. 
3. Abstract of Cited Paper (BIB003):  Abstract Microgrid works as a local energy provider for domestic buildings to reduce energy expenses and gas emissions by utilising distributed energy resources (DERs). The rapid advances in computing and communication capabilities enable the concept smart buildings become possible. Most energy-consuming household tasks do not need to be performed at specific times but rather within a preferred time. If these types of tasks can be coordinated among multiple homes so that they do not all occur at the same time yet still satisfy customers’ requirement, the energy cost and power peak demand could be reduced. In this paper, the optimal scheduling of smart homes’ energy consumption is studied using a mixed integer linear programming (MILP) approach. In order to minimise a 1-day forecasted energy consumption cost, DER operation and electricity-consumption household tasks are scheduled based on real-time electricity pricing, electricity task time window and forecasted renewable energy output. Peak demand charge scheme is also adopted to reduce the peak demand from grid. Two numerical examples on smart buildings of 30 homes and 90 homes with their own microgrid indicate the possibility of cost savings and electricity consumption scheduling peak reduction through the energy consumption and better management of DER operation. 
4. Abstract of Cited Paper (BIB004):  Innovations in the residential sector are required to reduce environmental impacts, as the sector is a contributor to greenhouse gas emissions. The increasing demand for electricity and the emergence of smart grids have presented new opportunities for home energy management systems (HEMS) in demand response markets. HEMS are demand response tools that shift and curtail demand to improve the energy consumption and production profile of a dwelling on behalf of a consumer. HEMS usually create optimal consumption and productions schedules by considering multiple objectives such as energy costs, environmental concerns, load profiles and consumer comfort. The existing literature has presented several methods, such as mathematical optimization, model predictive control and heuristic control, for creating efficient operation schedules and for making good consumption and production decisions. However, the effectiveness of the methods in the existing literature can be difficult to compare due to diversity in modelling parameters, such as appliance models, timing parameters and objectives. The present chapter provides a comparative analysis of the literature on HEMS, with a focus on modelling approaches and their impact on HEMS operations and outcomes. In particular, we discuss a set of HEMS challenges such as forecast uncertainty, modelling device heterogeneity, multi-objective scheduling, computational limitations, timing considerations and modelling consumer well-being. The presented work is organized to allow a reader to understand and compare the important considerations, approaches, nomenclature and results in prominent and new literary works without delving deeply into each one. 
5. Abstract of Cited Paper (BIB005):  The concept of smart grid SG has been introduced to improve the operation of the power systems. In modern structures of power systems, different reasons prompt researchers to suggest integrated analysis of multi-carrier energy systems. Considering synergy effects of the couplings between different energy carriers and utilising intelligent technologies for monitoring and controlling of energy flow may change energy system management in the future. In this paper, we propose a new solution which is entitled ‘smart energy hub’ SEH that models a multi-carrier energy system in a SG. SEH solutions allow homeowners to manage their energy consumption to reduce their electricity and gas bill. We present this concept for a residential customer by an ‘energy management system’ which uses reinforcement learning algorithm and Monte Carlo estimation method for finding a near optimal solution. The simulation results show that by using this concept and then applying the algorithm for a residential customer, running costs are reduced up to 40% while keeping the household owner's desired comfort levels. 
6. Abstract of Cited Paper (BIB006):  The development of technologies such as micro turbines and gas furnaces has been the major driving force towards integration of electricity and natural gas networks in the EH (energy hubs). Besides, the existing power grids are getting smarter by implementing the new generation of information technologies and communication systems. In smart grid, the electricity suppliers can modify the customers' electrical load consumption by employing appropriate scheduling schemes such as DR (demand response) programs. In this paper, we consider the S. E. Hubs (smart energy hubs) framework, in which the customers can use EMS (energy management system) to access to the electricity and natural gas prices data and wisely manage their daily energy consumption. We extend the existing DR programs to the IDR (integrated demand response) programs with the aim of modifying both electricity and natural gas consumption on the customer side. The interaction between S.E. hubs in the IDR program is formulated as a non-cooperative game. The goal of the IDR game is to maximize the natural gas and electricity utility companies' profit and to minimize the customers' consumption cost. It is shown that the proposed game model is an ordinal potential game with unique Nash equilibrium. Simulations are performed on an energy system with 6 S. E. Hubs, one electricity utility, and one natural gas utility companies. The results confirm that the IDR program can benefit both the customer side, by reducing the electricity and gas consumption cost, and the supplier side, by reducing the peak load demand in the electricity and natural gas load profiles. 
7. Abstract of Cited Paper (BIB007):  Abstract Today, as a consequence of the growing installation of efficient technologies, e.g. micro-combined heat and power (micro-CHP), the integration of traditionally separated electricity and natural gas networks has been attracting attentions from researchers in both academia and industry. To model the interaction among electricity and natural gas networks in distribution systems, this paper models a residential customer in a multi-energy system (MES). In this paper, we propose a fully automated energy management system (EMS) based on a reinforcement learning (RL) algorithm to motivate residential customers for participating in demand side management (DSM) programs and reducing the peak load in both electricity and natural gas networks. This proposed EMS estimates the residential customers’ satisfaction function, energy prices, and efficiencies of appliances based on the residential customers’ historical actions. Simulations are performed for the sample model and results depict how much of each energy, i.e. electricity and natural gas, the residential customer should consume and how much of natural gas should be converted in order to meet electricity and heating loads. It is also shown that the proposed RL algorithm reduces residential customer energy bill and electrical peak load up to 20% and 24%, respectively. 
8. Abstract of Cited Paper (BIB008):  PV installations in buildings can utilize different on-site flexibility resources to balance mismatch in electricity production and demand. This paper studies cost-optimal and rule-based control for buildings with PV, employing a heat pump, thermal and electrical storage and shiftable loads as flexibility sources to increase the value of PV for the prosumer. The cost-optimal control minimizes variable electricity cost employing market data on electricity price and optionally constrains grid feed-in to zero; the rule-based control aims at maximizing PV self-consumption. The flexibility strategies are combined into a simulation model to analyze different system configurations over a full year. 
9. Abstract of Cited Paper (BIB009):  Along with the development of Smart Energy System (SES), the advancing popularity of hybrid energy appliances, such as micro-combined heat and power (CHP) and electric heaters, requires an overall energy management strategy to optimize the energy using while guaranteeing energy supply for both the electricity and heat demand. This paper based on the concept of Multi-Commodity Smart Energy System (MC-SES) proposes a self-sufficient hybrid energy model to minimize the energy exchange with the external electricity grid. Next to the optimization solution for this hybrid energy management problem, a greedy algorithm has been proposed to alleviate the high computational complexity of optimization searching. The optimization, direct allocation and greedy-based methods will all be analyzed, simulated and compared in this paper.

Literature Review:
Heuristic techniques are designed to find a sufficiently good solution by following prescribed rules. The main idea is to find a balance between the solution quality and the computation time BIB001 BIB003 . Although heuristics cannot guarantee an optimal solution, they usually help finding good-enough solutions, while significantly reducing the computational burden of alternative mathematical methods BIB005 . Heuristics are typically domain-specific and their performance depends on the system they are implemented in. Moreover, they require a certain experience and knowledge of the system itself BIB005 . For instance, the NEST thermostat is smart device that uses heuristics to control the indoor temperature. By collecting usage statistics and environmental data of the local space, the device ''learns'' the user's behavior and implements rules (e.g., turning on and off the gas boiler) in order to satisfy user's temperature preferences  . The reinforcement learning method is applied in BIB006 and BIB008 with the aim of minimizing the total costs of a large residential energy hub and a single residential prosumer, respectively. The EMS learns which actions yield rewards (positively reinforced) or punishments (negatively reinforced), given a stochastic Markovian user behavior. The method achieves a near-optimal solution by means of the Q-learning algorithm BIB002 . In BIB009 and BIB004 , a costoptimal scheduling is compared to a rule-based heuristic one. In BIB009 , the heuristic control aims at maximizing the PV self-generation, by following a list of priorities in case of PV power surplus. For instance, first, shiftable appliances are moved in the surplus period; then, the remaining surplus is stored in ESS, or converted to heat. As the last resort, the surplus may be curtailed. In BIB004 , the heuristic scheduling turns on appliances at their earliest starting time, irrespective of energy prices. A greedy algorithm is proposed in BIB010 , which allocates resources according to a hierarchy. The residential devices are ranked according to their flexibility, from uncontrollable devices to shiftable appliances and storage. The algorithm allocates electricity and heat starting from the less flexible devices, and set the operation conditions for generation DERs. A near-optimal solution is found, significantly reducing the computational time, compared to a mathematical method. In BIB007 , the scheduling problem is formulated as a non-cooperative game and solved with a game theoreticbased algorithm. The algorithm is proven to converge to a Nash equilibrium, which determines the energy strategy profile that minimizes the user's bills, while maximizing the energy utilities' profit.