Woodpecker: Hallucination Correction for Multimodal Large Language
  Models
Hallucination is a big shadow hanging over the rapidly evolving Multimodal
Large Language Models (MLLMs), referring to the phenomenon that the generated
text is inconsistent with the image content. In order to mitigate
hallucinations, existing studies mainly resort to an instruction-tuning manner
that requires retraining the models with specific data. In this paper, we pave
a different way, introducing a training-free method named Woodpecker. Like a
woodpecker heals trees, it picks out and corrects hallucinations from the
generated text. Concretely, Woodpecker consists of five stages: key concept
extraction, question formulation, visual knowledge validation, visual claim
generation, and hallucination correction. Implemented in a post-remedy manner,
Woodpecker can easily serve different MLLMs, while being interpretable by
accessing intermediate outputs of the five stages. We evaluate Woodpecker both
quantitatively and qualitatively and show the huge potential of this new
paradigm. On the POPE benchmark, our method obtains a 30.66%/24.33% improvement
in accuracy over the baseline MiniGPT-4/mPLUG-Owl. The source code is released
at https://github.com/BradyFU/Woodpecker.
653870d0db57c222437d9363
Shukang Yin
admin_assigned
2023-10-26T11:40:27.831Z
62d12046683d9b0360ffdf1e
/avatars/3b79b73d897cbb5176887fe579d392d7.svg
Shukang Yin
user
xjtupanda
653870d0db57c222437d9364
Chaoyou Fu
653870d0db57c222437d9365
Sirui Zhao
653870d0db57c222437d9366
Tong Xu
653870d0db57c222437d9367
Hao Wang
653870d0db57c222437d9368
Dianbo Sui
admin_assigned
2023-10-26T11:41:33.494Z
643e8b572263cdc630f80fee
/avatars/93b11c28a58c5db71afd44e782493140.svg
Dianbo Sui
user
suidianbo
653870d0db57c222437d9369
Yunhang Shen
admin_assigned
2023-10-26T11:41:48.338Z
6483143902f98c3f05aff915
https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
沈云航 Yunhang Shen
user
shenyunhang
653870d0db57c222437d936a
Ke Li
claimed_verified
2025-01-09T10:13:06.036Z
63280915eeee4dd858083092
/avatars/78347af4af42527d53e88d9969c5c934.svg
Ke Li
user
tristanli
653870d0db57c222437d936b
Xing Sun
claimed_verified
2023-12-20T13:51:53.808Z
647401e50da364bd0d002f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png
XING SUN
user
tedsun
653870d0db57c222437d936c
Enhong Chen
admin_assigned
2023-10-26T11:42:39.040Z
64dde9ccd1885643ca14b2d4
/avatars/04d83c2c7477212ad55eec24aa69eaa2.svg
EnHong Chen
user
Hongni
Woodpecker: Hallucination Correction for Multimodal Large Language
  Models
Hallucination is a big shadow hanging over the rapidly evolving Multimodal
Large Language Models (MLLMs), referring to the phenomenon that the generated
text is inconsistent with the image content. In order to mitigate
hallucinations, existing studies mainly resort to an instruction-tuning manner
that requires retraining the models with specific data. In this paper, we pave
a different way, introducing a training-free method named Woodpecker. Like a
woodpecker heals trees, it picks out and corrects hallucinations from the
generated text. Concretely, Woodpecker consists of five stages: key concept
extraction, question formulation, visual knowledge validation, visual claim
generation, and hallucination correction. Implemented in a post-remedy manner,
Woodpecker can easily serve different MLLMs, while being interpretable by
accessing intermediate outputs of the five stages. We evaluate Woodpecker both
quantitatively and qualitatively and show the huge potential of this new
paradigm. On the POPE benchmark, our method obtains a 30.66%/24.33% improvement
in accuracy over the baseline MiniGPT-4/mPLUG-Owl. The source code is released
at https://github.com/BradyFU/Woodpecker.
Woodpecker, a training-free method, corrects hallucinations in multimodal large language models by extracting key concepts, formulating questions, validating visual knowledge, generating visual claims, and correcting inconsistencies.
hallucination Multimodal Large Language Models instruction-tuning key concept extraction question formulation visual knowledge validation visual claim generation hallucination correction POPE benchmark MiniGPT-4 mPLUG-Owl