TempFlow-GRPO: When Timing Matters for GRPO in Flow Models
Recent flow matching models for text-to-image generation have achieved
remarkable quality, yet their integration with reinforcement learning for human
preference alignment remains suboptimal, hindering fine-grained reward-based
optimization. We observe that the key impediment to effective GRPO training of
flow models is the temporal uniformity assumption in existing approaches:
sparse terminal rewards with uniform credit assignment fail to capture the
varying criticality of decisions across generation timesteps, resulting in
inefficient exploration and suboptimal convergence. To remedy this shortcoming,
we introduce TempFlow-GRPO (Temporal Flow GRPO), a principled GRPO
framework that captures and exploits the temporal structure inherent in
flow-based generation. TempFlow-GRPO introduces two key innovations: (i) a
trajectory branching mechanism that provides process rewards by concentrating
stochasticity at designated branching points, enabling precise credit
assignment without requiring specialized intermediate reward models; and (ii) a
noise-aware weighting scheme that modulates policy optimization according to
the intrinsic exploration potential of each timestep, prioritizing learning
during high-impact early stages while ensuring stable refinement in later
phases. These innovations endow the model with temporally-aware optimization
that respects the underlying generative dynamics, leading to state-of-the-art
performance in human preference alignment and standard text-to-image
benchmarks.
68a56e446cf0bf898542ed36
Xiaoxuan He
claimed_verified
2025-08-20T08:12:50.108Z
66974e7a57a5a55a1f4a0469
https://cdn-avatars.huggingface.co/v1/production/uploads/66974e7a57a5a55a1f4a0469/76WAtxoxJO5rjZWxEmF2t.jpeg
xxh
user
shreddedpork
68a56e446cf0bf898542ed37
Siming Fu
68a56e446cf0bf898542ed38
Yuke Zhao
claimed_verified
2025-08-22T08:23:42.710Z
664c4774d1ba9237d352b0bd
https://cdn-avatars.huggingface.co/v1/production/uploads/664c4774d1ba9237d352b0bd/0lz318nQHS_7rL2x-QDbg.jpeg
yuke zhao
user
hzeroyuke
68a56e446cf0bf898542ed39
Wanli Li
68a56e446cf0bf898542ed3a
Jian Yang
68a56e446cf0bf898542ed3b
Dacheng Yin
68a56e446cf0bf898542ed3c
Fengyun Rao
68a56e446cf0bf898542ed3d
Bo Zhang
TempFlow-GRPO: When Timing Matters for GRPO in Flow Models
Recent flow matching models for text-to-image generation have achieved
remarkable quality, yet their integration with reinforcement learning for human
preference alignment remains suboptimal, hindering fine-grained reward-based
optimization. We observe that the key impediment to effective GRPO training of
flow models is the temporal uniformity assumption in existing approaches:
sparse terminal rewards with uniform credit assignment fail to capture the
varying criticality of decisions across generation timesteps, resulting in
inefficient exploration and suboptimal convergence. To remedy this shortcoming,
we introduce TempFlow-GRPO (Temporal Flow GRPO), a principled GRPO
framework that captures and exploits the temporal structure inherent in
flow-based generation. TempFlow-GRPO introduces two key innovations: (i) a
trajectory branching mechanism that provides process rewards by concentrating
stochasticity at designated branching points, enabling precise credit
assignment without requiring specialized intermediate reward models; and (ii) a
noise-aware weighting scheme that modulates policy optimization according to
the intrinsic exploration potential of each timestep, prioritizing learning
during high-impact early stages while ensuring stable refinement in later
phases. These innovations endow the model with temporally-aware optimization
that respects the underlying generative dynamics, leading to state-of-the-art
performance in human preference alignment and standard text-to-image
benchmarks.
TempFlow-GRPO enhances text-to-image generation by addressing temporal credit assignment and noise-aware optimization in flow models, improving human preference alignment and benchmark performance.
flow matching models text-to-image generation reinforcement learning human preference alignment GRPO temporal uniformity assumption trajectory branching mechanism process rewards noise-aware weighting scheme temporally-aware optimization generative dynamics