DRIFT: Learning from Abundant User Dissatisfaction in Real-World
  Preference Learning
Real-world large language model deployments (e.g., conversational AI systems,
code generation assistants) naturally generate abundant implicit user
dissatisfaction (DSAT) signals, as users iterate toward better answers through
refinements, corrections, and expressed preferences, while explicit
satisfaction (SAT) feedback is scarce. Existing preference learning approaches
are poorly aligned with this data profile, as they rely on costly human
annotations or assume plentiful positive responses. In this paper, we introduce
DRIFT (Dissatisfaction-Refined Iterative
preFerence Training), which anchors training on real-world
DSAT signals and samples positives dynamically from the evolving policy.
Empirically, DRIFT models trained on real-world WildFeedback datasets
and synthetic UltraFeedback datasets achieve up to +6.23\% (7B) /
+7.61\% (14B) on WildBench Task Score and up to +8.95\% (7B) / +12.29\% (14B)
on AlpacaEval2 win rate over base models, outperforming strong baseline methods
such as iterative DPO and SPIN. At larger scales, the improvements are
particularly pronounced: 14B models trained with DRIFT surpass GPT-4o-mini on
WildBench. Further analysis shows that DRIFT also preserves exploratory
capacity, yielding more diverse high-reward solutions rather than collapsing to
narrow subsets. Theoretically, we demonstrate that this design preserves
preference margins and avoids the gradient degeneration. These results show
that DRIFT is an effective and scalable recipe for real-world post-training
that leverages the most abundant and informative signal. The code and data are
available at https://github.com/cacayaya/DRIFT.git.
68e5c815975ac4c405ef20df
Yifan Wang
claimed_verified
2025-10-08T05:49:24.565Z
647693afb9c742c51117e1fa
/avatars/6a7a3f0f47588b5c38881b289d042b7a.svg
Yifan Wang
user
AmberYifan
68e5c815975ac4c405ef20e0
Bolian Li
68e5c815975ac4c405ef20e1
Junlin Wu
68e5c815975ac4c405ef20e2
Zhaoxuan Tan
68e5c815975ac4c405ef20e3
Zheli Liu
68e5c815975ac4c405ef20e4
Ruqi Zhang
68e5c815975ac4c405ef20e5
Ananth Grama
68e5c815975ac4c405ef20e6
Qingkai Zeng
DRIFT: Learning from Abundant User Dissatisfaction in Real-World
  Preference Learning
Real-world large language model deployments (e.g., conversational AI systems,
code generation assistants) naturally generate abundant implicit user
dissatisfaction (DSAT) signals, as users iterate toward better answers through
refinements, corrections, and expressed preferences, while explicit
satisfaction (SAT) feedback is scarce. Existing preference learning approaches
are poorly aligned with this data profile, as they rely on costly human
annotations or assume plentiful positive responses. In this paper, we introduce
DRIFT (Dissatisfaction-Refined Iterative
preFerence Training), which anchors training on real-world
DSAT signals and samples positives dynamically from the evolving policy.
Empirically, DRIFT models trained on real-world WildFeedback datasets
and synthetic UltraFeedback datasets achieve up to +6.23\% (7B) /
+7.61\% (14B) on WildBench Task Score and up to +8.95\% (7B) / +12.29\% (14B)
on AlpacaEval2 win rate over base models, outperforming strong baseline methods
such as iterative DPO and SPIN. At larger scales, the improvements are
particularly pronounced: 14B models trained with DRIFT surpass GPT-4o-mini on
WildBench. Further analysis shows that DRIFT also preserves exploratory
capacity, yielding more diverse high-reward solutions rather than collapsing to
narrow subsets. Theoretically, we demonstrate that this design preserves
preference margins and avoids the gradient degeneration. These results show
that DRIFT is an effective and scalable recipe for real-world post-training
that leverages the most abundant and informative signal. The code and data are
available at https://github.com/cacayaya/DRIFT.git.
DRIFT, a dissatisfaction-refined iterative preference training method, improves large language models using implicit user dissatisfaction signals, achieving better performance than existing methods on real-world datasets.
DRIFT Dissatisfaction-Refined Iterative Preference Training WildFeedback UltraFeedback WildBench Task Score AlpacaEval2 win rate iterative DPO SPIN gradient degeneration preference margins exploratory capacity