Q-Instruct: Improving Low-level Visual Abilities for Multi-modality
  Foundation Models
Multi-modality foundation models, as represented by GPT-4V, have brought a
new paradigm for low-level visual perception and understanding tasks, that can
respond to a broad range of natural human instructions in a model. While
existing foundation models have shown exciting potentials on low-level visual
tasks, their related abilities are still preliminary and need to be improved.
In order to enhance these models, we conduct a large-scale subjective
experiment collecting a vast number of real human feedbacks on low-level
vision. Each feedback follows a pathway that starts with a detailed description
on the low-level visual appearance (*e.g. clarity, color, brightness* of an
image, and ends with an overall conclusion, with an average length of 45 words.
The constructed **Q-Pathway** dataset includes 58K detailed human feedbacks on
18,973 images with diverse low-level appearance. Moreover, to enable foundation
models to robustly respond to diverse types of questions, we design a
GPT-participated conversion to process these feedbacks into diverse-format 200K
instruction-response pairs. Experimental results indicate that the
**Q-Instruct** consistently elevates low-level perception and understanding
abilities across several foundational models. We anticipate that our datasets
can pave the way for a future that general intelligence can perceive,
understand low-level visual appearance and evaluate visual quality like a
human. Our dataset, model zoo, and demo is published at:
https://q-future.github.io/Q-Instruct.
6552e2c9812ed6f6ea59459b
Haoning Wu
claimed_verified
2023-11-14T09:36:57.376Z
63047ed2412a1b9d381b09c9
https://cdn-avatars.huggingface.co/v1/production/uploads/63047ed2412a1b9d381b09c9/2Ill5G0uSMyGstrawgmIb.jpeg
Haoning Wu, Teo
user
teowu
6552e2c9812ed6f6ea59459c
Zicheng Zhang
claimed_verified
2023-11-14T10:07:11.940Z
6526cc6bab4f5d98382f5603
https://cdn-avatars.huggingface.co/v1/production/uploads/6526cc6bab4f5d98382f5603/NekYYb61I5nt4au6gXsVK.jpeg
Zicheng Zhang
user
zhangzicheng
6552e2c9812ed6f6ea59459d
Erli Zhang
admin_assigned
2023-11-14T09:50:40.732Z
639fd77cbeb95d698de6671e
https://cdn-avatars.huggingface.co/v1/production/uploads/639fd77cbeb95d698de6671e/nP7IkgzDa2NhyP5K2CTMi.jpeg
Zhang Erli
user
KarlZEL
6552e2c9812ed6f6ea59459e
Chaofeng Chen
admin_assigned
2023-11-14T09:50:52.872Z
64bb4468b567ae97c3023293
/avatars/e575b7e0198b9622d4be3cb04c677d92.svg
chenchaofeng
user
chenchaofeng
6552e2c9812ed6f6ea59459f
Liang Liao
6552e2c9812ed6f6ea5945a0
Annan Wang
claimed_verified
2023-11-14T10:34:07.136Z
65534c8cd74d73c8c7ec6433
/avatars/18c5fe8d1b5ba5a06f59e7f494dbd146.svg
Annan Wang
user
AnnanWang
6552e2c9812ed6f6ea5945a1
Kaixin Xu
claimed_verified
2023-11-14T10:34:10.203Z
65534cf5b0b46eaf2f9499b3
/avatars/4e681322d2e8c1c705589596bf0c3650.svg
Kaixin Xu
user
kartmannXu
6552e2c9812ed6f6ea5945a2
Chunyi Li
admin_assigned
2023-11-14T09:51:26.122Z
644cf72f6d84e25fad73ae23
https://cdn-avatars.huggingface.co/v1/production/uploads/644cf72f6d84e25fad73ae23/VF0V38obLNyKAbbTM65r8.jpeg
Chunyi Li
user
lcysyzxdxc
6552e2c9812ed6f6ea5945a3
Jingwen Hou
6552e2c9812ed6f6ea5945a4
Guangtao Zhai
claimed_verified
2023-11-14T12:32:05.837Z
65535b125413c1a54e6fb243
/avatars/03bcf1d58865f5406aff49a415e78bdc.svg
Guangtao Zhai
user
GTZhai
6552e2c9812ed6f6ea5945a5
Geng Xue
6552e2c9812ed6f6ea5945a6
Wenxiu Sun
6552e2c9812ed6f6ea5945a7
Qiong Yan
6552e2c9812ed6f6ea5945a8
Weisi Lin
Q-Instruct: Improving Low-level Visual Abilities for Multi-modality
  Foundation Models
Multi-modality foundation models, as represented by GPT-4V, have brought a
new paradigm for low-level visual perception and understanding tasks, that can
respond to a broad range of natural human instructions in a model. While
existing foundation models have shown exciting potentials on low-level visual
tasks, their related abilities are still preliminary and need to be improved.
In order to enhance these models, we conduct a large-scale subjective
experiment collecting a vast number of real human feedbacks on low-level
vision. Each feedback follows a pathway that starts with a detailed description
on the low-level visual appearance (*e.g. clarity, color, brightness* of an
image, and ends with an overall conclusion, with an average length of 45 words.
The constructed **Q-Pathway** dataset includes 58K detailed human feedbacks on
18,973 images with diverse low-level appearance. Moreover, to enable foundation
models to robustly respond to diverse types of questions, we design a
GPT-participated conversion to process these feedbacks into diverse-format 200K
instruction-response pairs. Experimental results indicate that the
**Q-Instruct** consistently elevates low-level perception and understanding
abilities across several foundational models. We anticipate that our datasets
can pave the way for a future that general intelligence can perceive,
understand low-level visual appearance and evaluate visual quality like a
human. Our dataset, model zoo, and demo is published at:
https://q-future.github.io/Q-Instruct.
The Q-Instruct dataset, enhanced with diverse human feedback, improves low-level visual perception and understanding in multi-modality foundation models.
multi-modality foundation models GPT-4V low-level visual perception low-level visual tasks subjective experiment Q-Pathway dataset Q-Instruct instruction-response pairs visual appearance visual quality