Dyna-Mind: Learning to Simulate from Experience for Better AI Agents
Reasoning models have recently shown remarkable progress in domains such as
math and coding. However, their expert-level abilities in math and coding
contrast sharply with their performance in long-horizon, interactive tasks such
as web navigation and computer/phone-use. Inspired by literature on human
cognition, we argue that current AI agents need ''vicarious trial and error'' -
the capacity to mentally simulate alternative futures before acting - in order
to enhance their understanding and performance in complex interactive
environments. We introduce Dyna-Mind, a two-stage training framework that
explicitly teaches (V)LM agents to integrate such simulation into their
reasoning. In stage 1, we introduce Reasoning with Simulations (ReSim), which
trains the agent to generate structured reasoning traces from expanded search
trees built from real experience gathered through environment interactions.
ReSim thus grounds the agent's reasoning in faithful world dynamics and equips
it with the ability to anticipate future states in its reasoning. In stage 2,
we propose Dyna-GRPO, an online reinforcement learning method to further
strengthen the agent's simulation and decision-making ability by using both
outcome rewards and intermediate states as feedback from real rollouts.
Experiments on two synthetic benchmarks (Sokoban and ALFWorld) and one
realistic benchmark (AndroidWorld) demonstrate that (1) ReSim effectively
infuses simulation ability into AI agents, and (2) Dyna-GRPO leverages outcome
and interaction-level signals to learn better policies for long-horizon,
planning-intensive tasks. Together, these results highlight the central role of
simulation in enabling AI agents to reason, plan, and act more effectively in
the ever more challenging environments.
68ec56edcd07fb414898c8c9
Xiao Yu
68ec56edcd07fb414898c8ca
Baolin Peng
68ec56edcd07fb414898c8cb
Michel Galley
68ec56edcd07fb414898c8cc
Hao Cheng
68ec56edcd07fb414898c8cd
Qianhui Wu
68ec56edcd07fb414898c8ce
Janardhan Kulkarni
68ec56edcd07fb414898c8cf
Suman Nath
68ec56edcd07fb414898c8d0
Zhou Yu
68ec56edcd07fb414898c8d1
Jianfeng Gao
Dyna-Mind: Learning to Simulate from Experience for Better AI Agents
Reasoning models have recently shown remarkable progress in domains such as
math and coding. However, their expert-level abilities in math and coding
contrast sharply with their performance in long-horizon, interactive tasks such
as web navigation and computer/phone-use. Inspired by literature on human
cognition, we argue that current AI agents need ''vicarious trial and error'' -
the capacity to mentally simulate alternative futures before acting - in order
to enhance their understanding and performance in complex interactive
environments. We introduce Dyna-Mind, a two-stage training framework that
explicitly teaches (V)LM agents to integrate such simulation into their
reasoning. In stage 1, we introduce Reasoning with Simulations (ReSim), which
trains the agent to generate structured reasoning traces from expanded search
trees built from real experience gathered through environment interactions.
ReSim thus grounds the agent's reasoning in faithful world dynamics and equips
it with the ability to anticipate future states in its reasoning. In stage 2,
we propose Dyna-GRPO, an online reinforcement learning method to further
strengthen the agent's simulation and decision-making ability by using both
outcome rewards and intermediate states as feedback from real rollouts.
Experiments on two synthetic benchmarks (Sokoban and ALFWorld) and one
realistic benchmark (AndroidWorld) demonstrate that (1) ReSim effectively
infuses simulation ability into AI agents, and (2) Dyna-GRPO leverages outcome
and interaction-level signals to learn better policies for long-horizon,
planning-intensive tasks. Together, these results highlight the central role of
simulation in enabling AI agents to reason, plan, and act more effectively in
the ever more challenging environments.
Introducing Dyna-Mind, a two-stage training framework that enhances AI agents' reasoning and planning abilities through simulation, leading to improved performance in complex interactive environments.
vicarious trial and error Dyna-Mind Reasoning with Simulations ReSim Dyna-GRPO online reinforcement learning simulation ability reasoning planning Sokoban ALFWorld AndroidWorld