LaRes: Evolutionary Reinforcement Learning with LLM-based Adaptive Reward Search
Pengyi Li, Jianye Hao, Hongyao Tang, Jinbin Qiao, YAN ZHENG
The integration of evolutionary algorithms (EAs) with reinforcement learning (RL) has shown superior performance compared to standalone methods. However, previous research focuses on exploration in policy parameter space, while overlooking the reward function search. To bridge this gap, we propose LaRes, a novel hybrid framework that achieves efficient policy learning through reward function search. LaRes leverages large language models (LLMs) to generate the reward function population, guiding RL in policy learning. The reward functions are evaluated by the policy performance and improved through LLMs. To improve sample efficiency, LaRes employs a shared experience buffer that collects experiences from all policies, with each experience containing rewards from all reward functions. Upon reward function updates, the rewards of experiences are relabeled, enabling efficient use of historical data. Furthermore, we introduce a Thompson sampling-based selection mechanism that enables more efficient elite interaction. To prevent policy collapse when improving reward functions, we propose a reward scaling and parameter constraint mechanism to efficiently coordinate reward search with policy learning. Across 16 robot manipulation tasks, LaRes outperforms other strong RL and ERL baselines guided by expert rewards in terms of both sample efficiency and final performance.