Towards Open-ended Visual Quality Comparison
Comparative settings (e.g. pairwise choice, listwise ranking) have been
adopted by a wide range of subjective studies for image quality assessment
(IQA), as it inherently standardizes the evaluation criteria across different
observers and offer more clear-cut responses. In this work, we extend the edge
of emerging large multi-modality models (LMMs) to further advance visual
quality comparison into open-ended settings, that 1) can respond to open-range
questions on quality comparison; 2) can provide detailed reasonings beyond
direct answers. To this end, we propose the Co-Instruct. To train this
first-of-its-kind open-source open-ended visual quality comparer, we collect
the Co-Instruct-562K dataset, from two sources: (a) LMM-merged single image
quality description, (b) GPT-4V "teacher" responses on unlabeled data.
Furthermore, to better evaluate this setting, we propose the MICBench, the
first benchmark on multi-image comparison for LMMs. We demonstrate that
Co-Instruct not only achieves 30% higher superior accuracy than
state-of-the-art open-source LMMs, but also outperforms GPT-4V (its teacher),
on both existing related benchmarks and the proposed MICBench. Our model is
published at https://huggingface.co/q-future/co-instruct.
65dd6b22538de889604760fa
Haoning Wu
claimed_verified
2024-02-27T09:04:23.164Z
63047ed2412a1b9d381b09c9
https://cdn-avatars.huggingface.co/v1/production/uploads/63047ed2412a1b9d381b09c9/2Ill5G0uSMyGstrawgmIb.jpeg
Haoning Wu, Teo
user
teowu
65dd6b22538de889604760fb
Hanwei Zhu
admin_assigned
2024-02-27T09:47:00.109Z
6437970dcd93f4c9a34e4c5a
/avatars/5ef5cee3be250c600f0ed8cc5aad8dda.svg
Hanwei Zhu
user
HanweiZ
65dd6b22538de889604760fc
Zicheng Zhang
claimed_verified
2024-02-27T09:04:24.805Z
6526cc6bab4f5d98382f5603
https://cdn-avatars.huggingface.co/v1/production/uploads/6526cc6bab4f5d98382f5603/NekYYb61I5nt4au6gXsVK.jpeg
Zicheng Zhang
user
zhangzicheng
65dd6b22538de889604760fd
Erli Zhang
admin_assigned
2024-02-27T10:26:20.949Z
639fd77cbeb95d698de6671e
https://cdn-avatars.huggingface.co/v1/production/uploads/639fd77cbeb95d698de6671e/nP7IkgzDa2NhyP5K2CTMi.jpeg
Zhang Erli
user
KarlZEL
65dd6b22538de889604760fe
Chaofeng Chen
admin_assigned
2024-02-27T10:27:35.212Z
6304798d41387c7f117558f7
https://cdn-avatars.huggingface.co/v1/production/uploads/6304798d41387c7f117558f7/PBtL5BUvBDItAPdBEUwcv.jpeg
Chaofeng Chen
user
chaofengc
65dd6b22538de889604760ff
Liang Liao
admin_assigned
2024-02-27T10:27:42.113Z
6582fe058fb810eaad787d72
/avatars/54f03ee3e0c21664b20ea9b93ead4626.svg
Liang Liao
user
mercurialliao
65dd6b22538de88960476100
Chunyi Li
admin_assigned
2024-02-27T10:27:49.342Z
644cf72f6d84e25fad73ae23
https://cdn-avatars.huggingface.co/v1/production/uploads/644cf72f6d84e25fad73ae23/VF0V38obLNyKAbbTM65r8.jpeg
Chunyi Li
user
lcysyzxdxc
65dd6b22538de88960476101
Annan Wang
admin_assigned
2024-02-27T10:27:56.973Z
65534c8cd74d73c8c7ec6433
/avatars/18c5fe8d1b5ba5a06f59e7f494dbd146.svg
Annan Wang
user
AnnanWang
65dd6b22538de88960476102
Wenxiu Sun
65dd6b22538de88960476103
Qiong Yan
65dd6b22538de88960476104
Xiaohong Liu
65dd6b22538de88960476105
Guangtao Zhai
admin_assigned
2024-02-27T10:28:32.187Z
65535b125413c1a54e6fb243
/avatars/03bcf1d58865f5406aff49a415e78bdc.svg
Guangtao Zhai
user
GTZhai
65dd6b22538de88960476106
Shiqi Wang
65dd6b22538de88960476107
Weisi Lin
Towards Open-ended Visual Quality Comparison
Comparative settings (e.g. pairwise choice, listwise ranking) have been
adopted by a wide range of subjective studies for image quality assessment
(IQA), as it inherently standardizes the evaluation criteria across different
observers and offer more clear-cut responses. In this work, we extend the edge
of emerging large multi-modality models (LMMs) to further advance visual
quality comparison into open-ended settings, that 1) can respond to open-range
questions on quality comparison; 2) can provide detailed reasonings beyond
direct answers. To this end, we propose the Co-Instruct. To train this
first-of-its-kind open-source open-ended visual quality comparer, we collect
the Co-Instruct-562K dataset, from two sources: (a) LMM-merged single image
quality description, (b) GPT-4V "teacher" responses on unlabeled data.
Furthermore, to better evaluate this setting, we propose the MICBench, the
first benchmark on multi-image comparison for LMMs. We demonstrate that
Co-Instruct not only achieves 30% higher superior accuracy than
state-of-the-art open-source LMMs, but also outperforms GPT-4V (its teacher),
on both existing related benchmarks and the proposed MICBench. Our model is
published at https://huggingface.co/q-future/co-instruct.
Co-Instruct, an open-source open-ended visual quality comparer trained on Co-Instruct-562K dataset, outperforms state-of-the-art models and its teacher GPT-4V on image quality assessment tasks.
multi-modality models LMMs open-ended settings single image quality description GPT-4V MICBench multi-image comparison