FIRE: A Dataset for Feedback Integration and Refinement Evaluation of
  Multimodal Models
Vision language models (VLMs) have achieved impressive progress in diverse
applications, becoming a prevalent research direction. In this paper, we build
FIRE, a feedback-refinement dataset, consisting of 1.1M multi-turn
conversations that are derived from 27 source datasets, empowering VLMs to
spontaneously refine their responses based on user feedback across diverse
tasks. To scale up the data collection, FIRE is collected in two components:
FIRE-100K and FIRE-1M, where FIRE-100K is generated by GPT-4V, and FIRE-1M is
freely generated via models trained on FIRE-100K. Then, we build FIRE-Bench, a
benchmark to comprehensively evaluate the feedback-refining capability of VLMs,
which contains 11K feedback-refinement conversations as the test data, two
evaluation settings, and a model to provide feedback for VLMs. We develop the
FIRE-LLaVA model by fine-tuning LLaVA on FIRE-100K and FIRE-1M, which shows
remarkable feedback-refining capability on FIRE-Bench and outperforms untrained
VLMs by 50%, making more efficient user-agent interactions and underscoring the
significance of the FIRE dataset.
6697a6d09ad36faed17bee68
Pengxiang Li
claimed_verified
2024-08-09T07:48:38.848Z
65d6bd8ebb862d1b367580fe
https://cdn-avatars.huggingface.co/v1/production/uploads/65d6bd8ebb862d1b367580fe/Pf5o-Oxi2IZxJPa-ZgEtV.jpeg
Pengxiang Li
user
PengxiangLi
6697a6d09ad36faed17bee69
Zhi Gao
6697a6d09ad36faed17bee6a
Bofei Zhang
claimed_verified
2025-06-16T07:18:29.222Z
643e2ff70efe9e9de09c3881
/avatars/8e4433332bf6983e1b7956d0c96241b2.svg
Bofei Zhang
user
Bofeee5675
6697a6d09ad36faed17bee6b
Tao Yuan
6697a6d09ad36faed17bee6c
Yuwei Wu
6697a6d09ad36faed17bee6d
Mehrtash Harandi
6697a6d09ad36faed17bee6e
Yunde Jia
6697a6d09ad36faed17bee6f
Song-Chun Zhu
6697a6d09ad36faed17bee70
Qing Li
claimed_verified
2024-08-03T08:26:08.641Z
6455f8c2c8f569b995d603c9
/avatars/3cb2cad3ab123887c270234bb6a4ca43.svg
Qing Li
user
li-qing
FIRE: A Dataset for Feedback Integration and Refinement Evaluation of
  Multimodal Models
Vision language models (VLMs) have achieved impressive progress in diverse
applications, becoming a prevalent research direction. In this paper, we build
FIRE, a feedback-refinement dataset, consisting of 1.1M multi-turn
conversations that are derived from 27 source datasets, empowering VLMs to
spontaneously refine their responses based on user feedback across diverse
tasks. To scale up the data collection, FIRE is collected in two components:
FIRE-100K and FIRE-1M, where FIRE-100K is generated by GPT-4V, and FIRE-1M is
freely generated via models trained on FIRE-100K. Then, we build FIRE-Bench, a
benchmark to comprehensively evaluate the feedback-refining capability of VLMs,
which contains 11K feedback-refinement conversations as the test data, two
evaluation settings, and a model to provide feedback for VLMs. We develop the
FIRE-LLaVA model by fine-tuning LLaVA on FIRE-100K and FIRE-1M, which shows
remarkable feedback-refining capability on FIRE-Bench and outperforms untrained
VLMs by 50%, making more efficient user-agent interactions and underscoring the
significance of the FIRE dataset.
FIRE, a feedback-refinement dataset, enhances VLMs' performance through user feedback, leading to a FIRE-LLaVA model that outperforms its untrained counterparts by 50% on a comprehensive benchmark.
Vision language models feedback-refinement dataset multi-turn conversations GPT-4V benchmark evaluation settings fine-tuning LLaVA model feedback user-agent interactions