Clean First, Align Later: Benchmarking Preference Data Cleaning for
  Reliable LLM Alignment
Human feedback plays a pivotal role in aligning large language models (LLMs)
with human preferences. However, such feedback is often noisy or inconsistent,
which can degrade the quality of reward models and hinder alignment. While
various automated data cleaning methods have been proposed to mitigate this
issue, a systematic evaluation of their effectiveness and generalizability
remains lacking. To bridge this gap, we introduce the first comprehensive
benchmark for evaluating 13 preference data cleaning methods in the context of
LLM alignment. PrefCleanBench offers a standardized protocol to assess cleaning
strategies in terms of alignment performance and generalizability across
diverse datasets, model architectures, and optimization algorithms. By unifying
disparate methods and rigorously comparing them, we uncover key factors that
determine the success of data cleaning in alignment tasks. This benchmark lays
the groundwork for principled and reproducible approaches to improving LLM
alignment through better data quality-highlighting the crucial but
underexplored role of data preprocessing in responsible AI development. We
release modular implementations of all methods to catalyze further research:
https://github.com/deeplearning-wisc/PrefCleanBench.
68dbdc134159d1f2418f9769
Min-Hsuan Yeh
68dbdc134159d1f2418f976a
Yixuan Li
Clean First, Align Later: Benchmarking Preference Data Cleaning for
  Reliable LLM Alignment
Human feedback plays a pivotal role in aligning large language models (LLMs)
with human preferences. However, such feedback is often noisy or inconsistent,
which can degrade the quality of reward models and hinder alignment. While
various automated data cleaning methods have been proposed to mitigate this
issue, a systematic evaluation of their effectiveness and generalizability
remains lacking. To bridge this gap, we introduce the first comprehensive
benchmark for evaluating 13 preference data cleaning methods in the context of
LLM alignment. PrefCleanBench offers a standardized protocol to assess cleaning
strategies in terms of alignment performance and generalizability across
diverse datasets, model architectures, and optimization algorithms. By unifying
disparate methods and rigorously comparing them, we uncover key factors that
determine the success of data cleaning in alignment tasks. This benchmark lays
the groundwork for principled and reproducible approaches to improving LLM
alignment through better data quality-highlighting the crucial but
underexplored role of data preprocessing in responsible AI development. We
release modular implementations of all methods to catalyze further research:
https://github.com/deeplearning-wisc/PrefCleanBench.
PrefCleanBench evaluates 13 preference data cleaning methods for aligning large language models with human preferences, providing a standardized protocol to assess their effectiveness and generalizability.
large language models LLMs human feedback reward models alignment automated data cleaning benchmark alignment performance generalizability datasets model architectures optimization algorithms data preprocessing responsible AI development