Expanding the Action Space of LLMs to Reason Beyond Language
Large Language Models (LLMs) are powerful reasoners in natural language, but
their actions are typically confined to outputting vocabulary tokens. As a
result, interactions with external environments -- such as symbolic operators
or simulators -- must be expressed through text in predefined formats, parsed,
and routed to external interfaces. This overloads the model's language with
both reasoning and control duties, and requires a hand-crafted parser, external
to the LLM. To address this, we decouple environment interactions from language
by internalizing them in an Expanded Action space (ExpA), beyond the
vocabulary. The model starts reasoning in the default language environment, but
may trigger routing actions and switch to an external environment at any time.
From there, the model can only invoke environment-specific actions, receive
feedback from the environment, and potentially route back to language as a
result. To promote effective exploration of the expanded action space and new
environments, we introduce ExpA Reinforcement Learning (EARL) with
counterfactual policy optimization. On tasks requiring multi-turn interactions
and contingent planning, EARL outperforms strong baselines with
vocabulary-constrained actions. It performs robustly across calculator-based
multi-task learning and, in the partially observed sorting problem, achieves
perfect Sort-4 accuracy while self-discovering an efficient algorithm
competitive with classical designs.
68f47aa88589920bf4d31d7d
Zhongqi Yue
claimed_verified
2025-10-23T15:01:51.880Z
65a11f6c628cd4ce2f657448
/avatars/759e559434bb8fa2d6e12b762ef65136.svg
Yue Zhongqi
user
nickyue
68f47aa88589920bf4d31d7e
Weishi Wang
claimed_verified
2025-10-24T01:07:36.186Z
6423b122df71531a9c455318
/avatars/24cf967182b3096b584ec8e21128f535.svg
Wang Weishi
user
wwsntu
68f47aa88589920bf4d31d7f
Yundaichuan Zhan
claimed_verified
2025-10-22T15:46:21.495Z
68f7b4717714aaca65414aab
/avatars/1990d56557fbe67c3a82578079db8a62.svg
yundaichuanzhan
user
yundaichuanzhan
68f47aa88589920bf4d31d80
Juncheng Li
68f47aa88589920bf4d31d81
Daniel Dahlmeier
68f47aa88589920bf4d31d82
Fredrik D. Johansson
Expanding the Action Space of LLMs to Reason Beyond Language
Large Language Models (LLMs) are powerful reasoners in natural language, but
their actions are typically confined to outputting vocabulary tokens. As a
result, interactions with external environments -- such as symbolic operators
or simulators -- must be expressed through text in predefined formats, parsed,
and routed to external interfaces. This overloads the model's language with
both reasoning and control duties, and requires a hand-crafted parser, external
to the LLM. To address this, we decouple environment interactions from language
by internalizing them in an Expanded Action space (ExpA), beyond the
vocabulary. The model starts reasoning in the default language environment, but
may trigger routing actions and switch to an external environment at any time.
From there, the model can only invoke environment-specific actions, receive
feedback from the environment, and potentially route back to language as a
result. To promote effective exploration of the expanded action space and new
environments, we introduce ExpA Reinforcement Learning (EARL) with
counterfactual policy optimization. On tasks requiring multi-turn interactions
and contingent planning, EARL outperforms strong baselines with
vocabulary-constrained actions. It performs robustly across calculator-based
multi-task learning and, in the partially observed sorting problem, achieves
perfect Sort-4 accuracy while self-discovering an efficient algorithm
competitive with classical designs.
Expanded Action space (ExpA) with ExpA Reinforcement Learning (EARL) enhances Large Language Models (LLMs) by decoupling environment interactions from language, improving performance in multi-turn interactions and contingent planning tasks.
Expanded Action space ExpA ExpA Reinforcement Learning EARL counterfactual policy optimization multi-turn interactions contingent planning calculator-based multi-task learning partially observed sorting problem Sort-4 accuracy