Direct Preference Optimization of Video Large Multimodal Models from
  Language Model Reward
Preference modeling techniques, such as direct preference optimization (DPO),
has shown effective in enhancing the generalization abilities of large language
model (LLM). However, in tasks involving video instruction-following, providing
informative feedback, especially for detecting hallucinations in generated
responses, remains a significant challenge. Previous studies have explored
using large large multimodal models (LMMs) as reward models to guide preference
modeling, but their ability to accurately assess the factuality of generated
responses compared to corresponding videos has not been conclusively
established. This paper introduces a novel framework that utilizes detailed
video captions as a proxy of video content, enabling language models to
incorporate this information as supporting evidence for scoring video Question
Answering (QA) predictions. Our approach demonstrates robust alignment with
OpenAI GPT-4V model's reward mechanism, which directly takes video frames as
input. Furthermore, we show that applying this tailored reward through DPO
significantly improves the performance of video LMMs on video QA tasks.
660b94e2c30c49ec534b183d
Ruohong Zhang
claimed_verified
2024-10-23T07:35:43.256Z
63eeb0bbbe5d5900b85130ca
https://cdn-avatars.huggingface.co/v1/production/uploads/63eeb0bbbe5d5900b85130ca/ofmp91Zp7xAppQACCKz4L.jpeg
Ruohong Zhang
user
ruohongz
660b94e2c30c49ec534b183e
Liangke Gui
660b94e2c30c49ec534b183f
Zhiqing Sun
660b94e2c30c49ec534b1840
Yihao Feng
660b94e2c30c49ec534b1841
Keyang Xu
660b94e2c30c49ec534b1842
Yuanhan Zhang
claimed_verified
2024-07-19T11:24:12.774Z
62a993d80472c0b7f94027df
https://cdn-avatars.huggingface.co/v1/production/uploads/62a993d80472c0b7f94027df/j5vp-IwLA2YBexylUHiQU.png
Zhang Yuanhan
user
ZhangYuanhan
660b94e2c30c49ec534b1843
Di Fu
660b94e2c30c49ec534b1844
Chunyuan Li
660b94e2c30c49ec534b1845
Alexander Hauptmann
660b94e2c30c49ec534b1846
Yonatan Bisk
660b94e2c30c49ec534b1847
Yiming Yang
Direct Preference Optimization of Video Large Multimodal Models from
  Language Model Reward
Preference modeling techniques, such as direct preference optimization (DPO),
has shown effective in enhancing the generalization abilities of large language
model (LLM). However, in tasks involving video instruction-following, providing
informative feedback, especially for detecting hallucinations in generated
responses, remains a significant challenge. Previous studies have explored
using large large multimodal models (LMMs) as reward models to guide preference
modeling, but their ability to accurately assess the factuality of generated
responses compared to corresponding videos has not been conclusively
established. This paper introduces a novel framework that utilizes detailed
video captions as a proxy of video content, enabling language models to
incorporate this information as supporting evidence for scoring video Question
Answering (QA) predictions. Our approach demonstrates robust alignment with
OpenAI GPT-4V model's reward mechanism, which directly takes video frames as
input. Furthermore, we show that applying this tailored reward through DPO
significantly improves the performance of video LMMs on video QA tasks.
A novel framework using video captions as a proxy enhances preference modeling and performance of video LMMs by aligning with GPT-4V's reward mechanism through direct preference optimization.
direct preference optimization (DPO) large language model (LLM) video instruction-following reward models large multimodal models (LMMs) video captions video Question Answering (QA) GPT-4V