RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from
  Fine-grained Correctional Human Feedback
Multimodal Large Language Models (MLLMs) have recently demonstrated
impressive capabilities in multimodal understanding, reasoning, and
interaction. However, existing MLLMs prevalently suffer from serious
hallucination problems, generating text that is not factually grounded in
associated images. The problem makes existing MLLMs untrustworthy and thus
impractical in real-world (especially high-stakes) applications. To address the
challenge, we present RLHF-V, which enhances MLLM trustworthiness via behavior
alignment from fine-grained correctional human feedback. Specifically, RLHF-V
collects human preference in the form of segment-level corrections on
hallucinations, and performs dense direct preference optimization over the
human feedback. Comprehensive experiments on five benchmarks in both automatic
and human evaluation show that, RLHF-V can enable substantially more
trustworthy MLLM behaviors with promising data and computation efficiency.
Remarkably, using 1.4k annotated data samples, RLHF-V significantly reduces the
hallucination rate of the base MLLM by 34.8%, outperforming the concurrent
LLaVA-RLHF trained on 10k annotated data. The final model achieves
state-of-the-art performance in trustworthiness among open-source MLLMs, and
shows better robustness than GPT-4V in preventing hallucinations aroused from
over-generalization. We open-source our code, model, and data at
https://github.com/RLHF-V/RLHF-V.
656e8a8c2040818de9d4fb4f
Tianyu Yu
extracted_confirmed
2023-12-05T03:00:46.507Z
64abc4aa6cadc7aca585dddf
/avatars/736afea979cd0021c7a37f68731524ea.svg
Tianyu Yu
user
Yirany
656e8a8c2040818de9d4fb50
Yuan Yao
656e8a8c2040818de9d4fb51
Haoye Zhang
admin_assigned
2023-12-05T09:10:38.404Z
6566e0c493e30c8a60048eb3
/avatars/78a021f1b205dabef3d060d1edc33bb7.svg
Haoye Zhang
user
HaoyeZhang
656e8a8c2040818de9d4fb52
Taiwen He
656e8a8c2040818de9d4fb53
Yifeng Han
656e8a8c2040818de9d4fb54
Ganqu Cui
admin_assigned
2023-12-05T09:11:10.353Z
650eba9555dc1e841746f132
/avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
Ganqu Cui
user
ganqu
656e8a8c2040818de9d4fb55
Jinyi Hu
admin_assigned
2023-12-05T09:11:27.504Z
637a06580a77f602dc4ac922
https://cdn-avatars.huggingface.co/v1/production/uploads/637a06580a77f602dc4ac922/JR2VHWhG1tj3TKKNzD8GD.jpeg
Jinyi Hu
user
JamesHujy
656e8a8c2040818de9d4fb56
Zhiyuan Liu
admin_assigned
2023-12-05T09:11:52.320Z
6310a3cd531cc21f9e06de6a
https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
Zhiyuan Liu
user
acharkq
656e8a8c2040818de9d4fb57
Hai-Tao Zheng
656e8a8c2040818de9d4fb58
Maosong Sun
656e8a8c2040818de9d4fb59
Tat-Seng Chua
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from
  Fine-grained Correctional Human Feedback
Multimodal Large Language Models (MLLMs) have recently demonstrated
impressive capabilities in multimodal understanding, reasoning, and
interaction. However, existing MLLMs prevalently suffer from serious
hallucination problems, generating text that is not factually grounded in
associated images. The problem makes existing MLLMs untrustworthy and thus
impractical in real-world (especially high-stakes) applications. To address the
challenge, we present RLHF-V, which enhances MLLM trustworthiness via behavior
alignment from fine-grained correctional human feedback. Specifically, RLHF-V
collects human preference in the form of segment-level corrections on
hallucinations, and performs dense direct preference optimization over the
human feedback. Comprehensive experiments on five benchmarks in both automatic
and human evaluation show that, RLHF-V can enable substantially more
trustworthy MLLM behaviors with promising data and computation efficiency.
Remarkably, using 1.4k annotated data samples, RLHF-V significantly reduces the
hallucination rate of the base MLLM by 34.8%, outperforming the concurrent
LLaVA-RLHF trained on 10k annotated data. The final model achieves
state-of-the-art performance in trustworthiness among open-source MLLMs, and
shows better robustness than GPT-4V in preventing hallucinations aroused from
over-generalization. We open-source our code, model, and data at
https://github.com/RLHF-V/RLHF-V.
RLHF-V enhances the trustworthiness of Multimodal Large Language Models by using segment-level human feedback to reduce hallucinations.
multimodal large language models RLHF-V behavior alignment fine-grained correctional human feedback dense direct preference optimization hallucination rate LLaVA-RLHF GPT-4V