TLDR: Token-Level Detective Reward Model for Large Vision Language
  Models
Although reward models have been successful in improving multimodal large
language models, the reward models themselves remain brutal and contain minimal
information. Notably, existing reward models only mimic human annotations by
assigning only one binary feedback to any text, no matter how long the text is.
In the realm of multimodal language models, where models are required to
process both images and texts, a naive reward model may learn implicit biases
toward texts and become less grounded in images. In this paper, we propose a
Token-Level Detective Reward Model
(TLDR) to provide fine-grained annotations to each text token. We
first introduce a perturbation-based method to generate synthetic hard
negatives and their token-level labels to train TLDR models. Then we show the
rich usefulness of TLDR models both in assisting off-the-shelf models to
self-correct their generations, and in serving as a hallucination evaluation
tool. Finally, we show that TLDR models can significantly speed up human
annotation by 3 times to acquire a broader range of high-quality vision
language data.
6704b02e6282fcabb75d684e
Deqing Fu
extracted_confirmed
2024-10-08T04:10:22.462Z
63c8454e46421a2efe82709d
https://cdn-avatars.huggingface.co/v1/production/uploads/63c8454e46421a2efe82709d/3BcSk4KOwAgWHEPVtsAV3.png
Deqing Fu
user
deqing
6704b02e6282fcabb75d684f
Tong Xiao
6704b02e6282fcabb75d6850
Rui Wang
6704b02e6282fcabb75d6851
Wang Zhu
claimed_verified
2024-11-26T21:21:32.657Z
6389568eec1f539adc0e0337
/avatars/6ee5c9bea9989e4618ae8d334bd889a9.svg
Wang Bill Zhu
user
Bill1235813
6704b02e6282fcabb75d6852
Pengchuan Zhang
admin_assigned
2024-10-08T09:40:10.427Z
640056f5261cfa61f39b4c35
/avatars/25aa5dee42caa69327d158a669f6bf3a.svg
Pengchuan Zhang
user
pzzhang
6704b02e6282fcabb75d6853
Guan Pang
admin_assigned
2024-10-08T09:40:04.655Z
6573a458d7f487de5f9bb44a
/avatars/4aa5626af43855ad770186f0d4191a01.svg
Guan Pang
user
tbpangolin
6704b02e6282fcabb75d6854
Robin Jia
6704b02e6282fcabb75d6855
Lawrence Chen
TLDR: Token-Level Detective Reward Model for Large Vision Language
  Models
Although reward models have been successful in improving multimodal large
language models, the reward models themselves remain brutal and contain minimal
information. Notably, existing reward models only mimic human annotations by
assigning only one binary feedback to any text, no matter how long the text is.
In the realm of multimodal language models, where models are required to
process both images and texts, a naive reward model may learn implicit biases
toward texts and become less grounded in images. In this paper, we propose a
Token-Level Detective Reward Model
(TLDR) to provide fine-grained annotations to each text token. We
first introduce a perturbation-based method to generate synthetic hard
negatives and their token-level labels to train TLDR models. Then we show the
rich usefulness of TLDR models both in assisting off-the-shelf models to
self-correct their generations, and in serving as a hallucination evaluation
tool. Finally, we show that TLDR models can significantly speed up human
annotation by 3 times to acquire a broader range of high-quality vision
language data.
A token-level detective reward model (TLDR) provides detailed feedback for text tokens, assisting in self-correction and hallucination evaluation, and accelerating human annotation.
reward models multimodal language models token-level perturbation-based synthetic hard negatives hallucination evaluation