In-the-Flow Agentic System Optimization for Effective Planning and Tool
  Use
Outcome-driven reinforcement learning has advanced reasoning in large
language models (LLMs), but prevailing tool-augmented approaches train a
single, monolithic policy that interleaves thoughts and tool calls under full
context; this scales poorly with long horizons and diverse tools and
generalizes weakly to new scenarios. Agentic systems offer a promising
alternative by decomposing work across specialized modules, yet most remain
training-free or rely on offline training decoupled from the live dynamics of
multi-turn interaction. We introduce AgentFlow, a trainable, in-the-flow
agentic framework that coordinates four modules (planner, executor, verifier,
generator) through an evolving memory and directly optimizes its planner inside
the multi-turn loop. To train on-policy in live environments, we propose
Flow-based Group Refined Policy Optimization (Flow-GRPO), which tackles
long-horizon, sparse-reward credit assignment by converting multi-turn
optimization into a sequence of tractable single-turn policy updates. It
broadcasts a single, verifiable trajectory-level outcome to every turn to align
local planner decisions with global success and stabilizes learning with
group-normalized advantages. Across ten benchmarks, AgentFlow with a 7B-scale
backbone outperforms top-performing baselines with average accuracy gains of
14.9% on search, 14.0% on agentic, 14.5% on mathematical, and 4.1% on
scientific tasks, even surpassing larger proprietary models like GPT-4o.
Further analyses confirm the benefits of in-the-flow optimization, showing
improved planning, enhanced tool-calling reliability, and positive scaling with
model size and reasoning turns.
68e5c595975ac4c405ef20d4
Zhuofeng Li
claimed_verified
2025-10-08T13:49:36.168Z
66349404f2c753240d02952a
https://cdn-avatars.huggingface.co/v1/production/uploads/66349404f2c753240d02952a/xKBKicwyk7BoOITQPwBJn.png
ZhuofengLi
user
ZhuofengLi
68e5c595975ac4c405ef20d5
Haoxiang Zhang
68e5c595975ac4c405ef20d6
Seungju Han
68e5c595975ac4c405ef20d7
Sheng Liu
68e5c595975ac4c405ef20d8
Jianwen Xie
68e5c595975ac4c405ef20d9
Yu Zhang
claimed_verified
2025-10-09T06:51:49.364Z
64b78f80b5d293ec95a3f694
https://cdn-avatars.huggingface.co/v1/production/uploads/64b78f80b5d293ec95a3f694/3rKg-m2i6ulj_aYhldvU8.png
Yu Zhang
user
yuz9yuz
68e5c595975ac4c405ef20da
Yejin Choi
68e5c595975ac4c405ef20db
James Zou
68e5c595975ac4c405ef20dc
Pan Lu
claimed_verified
2025-10-09T06:51:59.516Z
60f5f68fa7fd83d025749234
https://cdn-avatars.huggingface.co/v1/production/uploads/60f5f68fa7fd83d025749234/gCeJAZfzaANAcEvI6v5-P.jpeg
Pan Lu
user
lupantech
In-the-Flow Agentic System Optimization for Effective Planning and Tool
  Use
Outcome-driven reinforcement learning has advanced reasoning in large
language models (LLMs), but prevailing tool-augmented approaches train a
single, monolithic policy that interleaves thoughts and tool calls under full
context; this scales poorly with long horizons and diverse tools and
generalizes weakly to new scenarios. Agentic systems offer a promising
alternative by decomposing work across specialized modules, yet most remain
training-free or rely on offline training decoupled from the live dynamics of
multi-turn interaction. We introduce AgentFlow, a trainable, in-the-flow
agentic framework that coordinates four modules (planner, executor, verifier,
generator) through an evolving memory and directly optimizes its planner inside
the multi-turn loop. To train on-policy in live environments, we propose
Flow-based Group Refined Policy Optimization (Flow-GRPO), which tackles
long-horizon, sparse-reward credit assignment by converting multi-turn
optimization into a sequence of tractable single-turn policy updates. It
broadcasts a single, verifiable trajectory-level outcome to every turn to align
local planner decisions with global success and stabilizes learning with
group-normalized advantages. Across ten benchmarks, AgentFlow with a 7B-scale
backbone outperforms top-performing baselines with average accuracy gains of
14.9% on search, 14.0% on agentic, 14.5% on mathematical, and 4.1% on
scientific tasks, even surpassing larger proprietary models like GPT-4o.
Further analyses confirm the benefits of in-the-flow optimization, showing
improved planning, enhanced tool-calling reliability, and positive scaling with
model size and reasoning turns.
AgentFlow, a trainable agentic framework with in-the-flow optimization, enhances reasoning in large language models by coordinating specialized modules and outperforms top baselines across various tasks.
reinforcement learning large language models tool-augmented approaches monolithic policy agentic systems specialized modules trainable framework planner executor verifier generator evolving memory on-policy training multi-turn interaction Flow-based Group Refined Policy Optimization Flow-GRPO long-horizon sparse-reward credit assignment single-turn policy updates trajectory-level outcome group-normalized advantages search tasks agentic tasks mathematical tasks scientific tasks GPT-4o