Exploitation Is All You Need... for Exploration
Ensuring sufficient exploration is a central challenge when training
meta-reinforcement learning (meta-RL) agents to solve novel environments.
Conventional solutions to the exploration-exploitation dilemma inject explicit
incentives such as randomization, uncertainty bonuses, or intrinsic rewards to
encourage exploration. In this work, we hypothesize that an agent trained
solely to maximize a greedy (exploitation-only) objective can nonetheless
exhibit emergent exploratory behavior, provided three conditions are met: (1)
Recurring Environmental Structure, where the environment features repeatable
regularities that allow past experience to inform future choices; (2) Agent
Memory, enabling the agent to retain and utilize historical interaction data;
and (3) Long-Horizon Credit Assignment, where learning propagates returns over
a time frame sufficient for the delayed benefits of exploration to inform
current decisions. Through experiments in stochastic multi-armed bandits and
temporally extended gridworlds, we observe that, when both structure and memory
are present, a policy trained on a strictly greedy objective exhibits
information-seeking exploratory behavior. We further demonstrate, through
controlled ablations, that emergent exploration vanishes if either
environmental structure or agent memory is absent (Conditions 1 & 2).
Surprisingly, removing long-horizon credit assignment (Condition 3) does not
always prevent emergent exploration-a result we attribute to the
pseudo-Thompson Sampling effect. These findings suggest that, under the right
prerequisites, exploration and exploitation need not be treated as orthogonal
objectives but can emerge from a unified reward-maximization process.
6891995ff01a094725f83594
Micah Rentschler
6891995ff01a094725f83595
Jesse Roberts
Exploitation Is All You Need... for Exploration
Ensuring sufficient exploration is a central challenge when training
meta-reinforcement learning (meta-RL) agents to solve novel environments.
Conventional solutions to the exploration-exploitation dilemma inject explicit
incentives such as randomization, uncertainty bonuses, or intrinsic rewards to
encourage exploration. In this work, we hypothesize that an agent trained
solely to maximize a greedy (exploitation-only) objective can nonetheless
exhibit emergent exploratory behavior, provided three conditions are met: (1)
Recurring Environmental Structure, where the environment features repeatable
regularities that allow past experience to inform future choices; (2) Agent
Memory, enabling the agent to retain and utilize historical interaction data;
and (3) Long-Horizon Credit Assignment, where learning propagates returns over
a time frame sufficient for the delayed benefits of exploration to inform
current decisions. Through experiments in stochastic multi-armed bandits and
temporally extended gridworlds, we observe that, when both structure and memory
are present, a policy trained on a strictly greedy objective exhibits
information-seeking exploratory behavior. We further demonstrate, through
controlled ablations, that emergent exploration vanishes if either
environmental structure or agent memory is absent (Conditions 1 & 2).
Surprisingly, removing long-horizon credit assignment (Condition 3) does not
always prevent emergent exploration-a result we attribute to the
pseudo-Thompson Sampling effect. These findings suggest that, under the right
prerequisites, exploration and exploitation need not be treated as orthogonal
objectives but can emerge from a unified reward-maximization process.
Meta-reinforcement learning agents can exhibit exploratory behavior when trained with a greedy objective, provided the environment has recurring structure, the agent has memory, and long-horizon credit assignment is possible.
meta-reinforcement learning meta-RL exploration-exploitation dilemma intrinsic rewards greedy objective emergent exploratory behavior Recurring Environmental Structure Agent Memory Long-Horizon Credit Assignment stochastic multi-armed bandits temporally extended gridworlds pseudo-Thompson Sampling