InternChat: Solving Vision-Centric Tasks by Interacting with Chatbots
  Beyond Language
We present an interactive visual framework named InternChat, or iChat for
short. The framework integrates chatbots that have planning and reasoning
capabilities, such as ChatGPT, with non-verbal instructions like pointing
movements that enable users to directly manipulate images or videos on the
screen. Pointing (including gestures, cursors, etc.) movements can provide more
flexibility and precision in performing vision-centric tasks that require
fine-grained control, editing, and generation of visual content. The name
InternChat stands for interaction, nonverbal, and chatbots. Different from
existing interactive systems that rely on pure language, by incorporating
pointing instructions, the proposed iChat significantly improves the efficiency
of communication between users and chatbots, as well as the accuracy of
chatbots in vision-centric tasks, especially in complicated visual scenarios
where the number of objects is greater than 2. Additionally, in iChat, an
auxiliary control mechanism is used to improve the control capability of LLM,
and a large vision-language model termed Husky is fine-tuned for high-quality
multi-modal dialogue (impressing ChatGPT-3.5-turbo with 93.89% GPT-4 Quality).
We hope this work can spark new ideas and directions for future interactive
visual systems. Welcome to watch the code at
https://github.com/OpenGVLab/InternChat.
645aef997d8ec84b7b5e466f
Zhaoyang Liu
claimed_verified
2024-12-16T13:44:41.363Z
6432c6152bfb2b0ec7572479
/avatars/7c2f908f511cdfa38a19e967d86f9c40.svg
Zhaoyang Liu
user
zyliu
645aef997d8ec84b7b5e4670
Yinan He
admin_assigned
2023-05-24T09:32:22.397Z
62aafa49f29ff279b51f0182
https://cdn-avatars.huggingface.co/v1/production/uploads/62aafa49f29ff279b51f0182/rQx8QFQGOY2qIhqJ8zSRj.jpeg
yinanhe
user
ynhe
645aef997d8ec84b7b5e4671
Wenhai Wang
645aef997d8ec84b7b5e4672
Weiyun Wang
admin_assigned
2023-05-24T09:32:51.243Z
619507e7b74b6c591f794340
https://cdn-avatars.huggingface.co/v1/production/uploads/619507e7b74b6c591f794340/JbPDoy6Ko1V1-6oJJwFV8.jpeg
Weiyun Wang
user
Weiyun1025
645aef997d8ec84b7b5e4673
Yi Wang
claimed_verified
2023-07-17T06:49:33.545Z
63119a987680dc699b2031df
/avatars/db63ab73806ee9cb8aa01be82d9effdd.svg
Yi Wang
user
shepnerd
645aef997d8ec84b7b5e4674
Shoufa Chen
admin_assigned
2023-05-24T09:34:02.089Z
6412a33900634c4fe9873652
https://cdn-avatars.huggingface.co/v1/production/uploads/6412a33900634c4fe9873652/Nmn_yRA1gGD2VO1YbSOYF.jpeg
Shoufa Chen
user
ShoufaChen
645aef997d8ec84b7b5e4675
Qinglong Zhang
admin_assigned
2023-05-24T09:34:46.454Z
63db4d3b9f2687298a148b08
https://cdn-avatars.huggingface.co/v1/production/uploads/1675316592071-63db4d3b9f2687298a148b08.jpeg
Qinglong Zhang
user
wofmanaf
645aef997d8ec84b7b5e4676
Yang Yang
645aef997d8ec84b7b5e4677
Qingyun Li
admin_assigned
2023-05-24T09:35:27.545Z
637f100c1dbae09191088e4c
https://cdn-avatars.huggingface.co/v1/production/uploads/1669271529782-noauth.jpeg
Qingyun Li
user
Qingyun
645aef997d8ec84b7b5e4678
Jiashuo Yu
claimed_verified
2024-07-29T07:10:40.478Z
64b89a14cf14c2fabe96664c
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tEd3fBjMcEubF4plqzcUz.jpeg
Jiashuo Yu
user
awojustin
645aef997d8ec84b7b5e4679
Kunchang Li
admin_assigned
2023-05-24T09:37:36.315Z
61fb81006374891646732f37
https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
Kunchang Li
user
Andy1621
645aef997d8ec84b7b5e467a
Zhe Chen
claimed_verified
2024-12-09T14:51:48.003Z
64119264f0f81eb569e0d569
/avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
Zhe Chen
user
czczup
645aef997d8ec84b7b5e467b
Xue Yang
645aef997d8ec84b7b5e467c
Xizhou Zhu
645aef997d8ec84b7b5e467d
Yali Wang
645aef997d8ec84b7b5e467e
Limin Wang
admin_assigned
2023-05-24T09:38:51.807Z
62c77f4352d8ae531f5511f9
/avatars/50198ccb02ccd286975a4613fbabee28.svg
Limin Wang
user
lmwang
645aef997d8ec84b7b5e467f
Ping Luo
645aef997d8ec84b7b5e4680
Jifeng Dai
admin_assigned
2023-05-24T09:39:39.552Z
64686f7172d9180d4ac8b4e4
/avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
Jifeng Dai
user
daijifeng
645aef997d8ec84b7b5e4681
Yu Qiao
InternChat: Solving Vision-Centric Tasks by Interacting with Chatbots
  Beyond Language
We present an interactive visual framework named InternChat, or iChat for
short. The framework integrates chatbots that have planning and reasoning
capabilities, such as ChatGPT, with non-verbal instructions like pointing
movements that enable users to directly manipulate images or videos on the
screen. Pointing (including gestures, cursors, etc.) movements can provide more
flexibility and precision in performing vision-centric tasks that require
fine-grained control, editing, and generation of visual content. The name
InternChat stands for interaction, nonverbal, and chatbots. Different from
existing interactive systems that rely on pure language, by incorporating
pointing instructions, the proposed iChat significantly improves the efficiency
of communication between users and chatbots, as well as the accuracy of
chatbots in vision-centric tasks, especially in complicated visual scenarios
where the number of objects is greater than 2. Additionally, in iChat, an
auxiliary control mechanism is used to improve the control capability of LLM,
and a large vision-language model termed Husky is fine-tuned for high-quality
multi-modal dialogue (impressing ChatGPT-3.5-turbo with 93.89% GPT-4 Quality).
We hope this work can spark new ideas and directions for future interactive
visual systems. Welcome to watch the code at
https://github.com/OpenGVLab/InternChat.
The InternChat framework combines chatbots with pointing gestures to enhance the precision and efficiency of vision-centric tasks, particularly improving the accuracy of large vision-language models in complex visual scenarios.
chatbots pointing movements vision-centric tasks fine-grained control vision-language model Husky multi-modal dialogue GPT-4 Quality