Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with
  Free-Form Preferences
Reward models (RMs) play a critical role in aligning AI behaviors with human
preferences, yet they face two fundamental challenges: (1) Modality Imbalance,
where most RMs are mainly focused on text and image modalities, offering
limited support for video, audio, and other modalities; and (2) Preference
Rigidity, where training on fixed binary preference pairs fails to capture the
complexity and diversity of personalized preferences. To address the above
challenges, we propose Omni-Reward, a step toward generalist omni-modal reward
modeling with support for free-form preferences, consisting of: (1) Evaluation:
We introduce Omni-RewardBench, the first omni-modal RM benchmark with free-form
preferences, covering nine tasks across five modalities including text, image,
video, audio, and 3D; (2) Data: We construct Omni-RewardData, a multimodal
preference dataset comprising 248K general preference pairs and 69K
instruction-tuning pairs for training generalist omni-modal RMs; (3) Model: We
propose Omni-RewardModel, which includes both discriminative and generative
RMs, and achieves strong performance on Omni-RewardBench as well as other
widely used reward modeling benchmarks.
69002f8622d452aac6dd42f3
Zhuoran Jin
69002f8622d452aac6dd42f4
Hongbang Yuan
69002f8622d452aac6dd42f5
Kejian Zhu
69002f8622d452aac6dd42f6
Jiachun Li
69002f8622d452aac6dd42f7
Pengfei Cao
69002f8622d452aac6dd42f8
Yubo Chen
69002f8622d452aac6dd42f9
Kang Liu
69002f8622d452aac6dd42fa
Jun Zhao
Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with
  Free-Form Preferences
Reward models (RMs) play a critical role in aligning AI behaviors with human
preferences, yet they face two fundamental challenges: (1) Modality Imbalance,
where most RMs are mainly focused on text and image modalities, offering
limited support for video, audio, and other modalities; and (2) Preference
Rigidity, where training on fixed binary preference pairs fails to capture the
complexity and diversity of personalized preferences. To address the above
challenges, we propose Omni-Reward, a step toward generalist omni-modal reward
modeling with support for free-form preferences, consisting of: (1) Evaluation:
We introduce Omni-RewardBench, the first omni-modal RM benchmark with free-form
preferences, covering nine tasks across five modalities including text, image,
video, audio, and 3D; (2) Data: We construct Omni-RewardData, a multimodal
preference dataset comprising 248K general preference pairs and 69K
instruction-tuning pairs for training generalist omni-modal RMs; (3) Model: We
propose Omni-RewardModel, which includes both discriminative and generative
RMs, and achieves strong performance on Omni-RewardBench as well as other
widely used reward modeling benchmarks.
Omni-Reward addresses modality imbalance and preference rigidity in reward models by introducing a benchmark, dataset, and model that support multiple modalities and free-form preferences.
reward models modality imbalance preference rigidity Omni-Reward Omni-RewardBench Omni-RewardData Omni-RewardModel discriminative RMs generative RMs