One Token to Seg Them All: Language Instructed Reasoning Segmentation in
  Videos
We introduce VideoLISA, a video-based multimodal large language model
designed to tackle the problem of language-instructed reasoning segmentation in
videos. Leveraging the reasoning capabilities and world knowledge of large
language models, and augmented by the Segment Anything Model, VideoLISA
generates temporally consistent segmentation masks in videos based on language
instructions. Existing image-based methods, such as LISA, struggle with video
tasks due to the additional temporal dimension, which requires temporal dynamic
understanding and consistent segmentation across frames. VideoLISA addresses
these challenges by integrating a Sparse Dense Sampling strategy into the
video-LLM, which balances temporal context and spatial detail within
computational constraints. Additionally, we propose a One-Token-Seg-All
approach using a specially designed <TRK> token, enabling the model to segment
and track objects across multiple frames. Extensive evaluations on diverse
benchmarks, including our newly introduced ReasonVOS benchmark, demonstrate
VideoLISA's superior performance in video object segmentation tasks involving
complex reasoning, temporal understanding, and object tracking. While optimized
for videos, VideoLISA also shows promising generalization to image
segmentation, revealing its potential as a unified foundation model for
language-instructed object segmentation. Code and model will be available at:
https://github.com/showlab/VideoLISA.
66fcf944c0a62736ec4530e6
Zechen Bai
admin_assigned
2024-10-02T08:41:34.135Z
64b7833aa5018e3c7c9b50d8
/avatars/782415605ed786b73f484fcc86a6384f.svg
Zechen Bai
user
ZechenBai
66fcf944c0a62736ec4530e7
Tong He
admin_assigned
2024-10-02T08:41:41.962Z
64478c64e2148488340229db
/avatars/f5c23489a068e896381cdc25836ce3dd.svg
he
user
tonghe
66fcf944c0a62736ec4530e8
Haiyang Mei
admin_assigned
2024-10-02T08:41:48.251Z
66fcfa6e05638227c44233a9
/avatars/4a88765c7f5c5ca77da6d21eb01f73e0.svg
Haiyang Mei
user
meihaiyang
66fcf944c0a62736ec4530e9
Pichao Wang
admin_assigned
2024-10-02T08:41:54.212Z
65381380c104b8d4556301c3
/avatars/3b6e1283cba0e6a9147f7cfcebc01a0b.svg
Pichao Wang
user
wangpichao
66fcf944c0a62736ec4530ea
Ziteng Gao
admin_assigned
2024-10-02T08:42:01.234Z
648ac3d53470b17ccc90deaf
/avatars/35b678baa7a91959f40f704c5de2a3e1.svg
Ziteng Gao
user
sebgao
66fcf944c0a62736ec4530eb
Joya Chen
admin_assigned
2024-10-02T08:42:07.352Z
642435a1a3adbc7142c3b0a6
https://cdn-avatars.huggingface.co/v1/production/uploads/642435a1a3adbc7142c3b0a6/wgLT_w9jNWRU3O0jU0646.jpeg
Joya Chen
user
chenjoya
66fcf944c0a62736ec4530ec
Lei Liu
66fcf944c0a62736ec4530ed
Zheng Zhang
66fcf944c0a62736ec4530ee
Mike Zheng Shou
admin_assigned
2024-10-02T08:43:03.311Z
661ab3da2b14565c7acccf5c
/avatars/fa4fc03664803e02aede4d4c3d50b393.svg
Mike Zheng Shou
user
AnalMom
One Token to Seg Them All: Language Instructed Reasoning Segmentation in
  Videos
We introduce VideoLISA, a video-based multimodal large language model
designed to tackle the problem of language-instructed reasoning segmentation in
videos. Leveraging the reasoning capabilities and world knowledge of large
language models, and augmented by the Segment Anything Model, VideoLISA
generates temporally consistent segmentation masks in videos based on language
instructions. Existing image-based methods, such as LISA, struggle with video
tasks due to the additional temporal dimension, which requires temporal dynamic
understanding and consistent segmentation across frames. VideoLISA addresses
these challenges by integrating a Sparse Dense Sampling strategy into the
video-LLM, which balances temporal context and spatial detail within
computational constraints. Additionally, we propose a One-Token-Seg-All
approach using a specially designed <TRK> token, enabling the model to segment
and track objects across multiple frames. Extensive evaluations on diverse
benchmarks, including our newly introduced ReasonVOS benchmark, demonstrate
VideoLISA's superior performance in video object segmentation tasks involving
complex reasoning, temporal understanding, and object tracking. While optimized
for videos, VideoLISA also shows promising generalization to image
segmentation, revealing its potential as a unified foundation model for
language-instructed object segmentation. Code and model will be available at:
https://github.com/showlab/VideoLISA.
VideoLISA, a video-based multimodal large language model, uses reasoning and a Sparse Dense Sampling strategy to perform temporally consistent segmentation and object tracking in videos based on language instructions.
VideoLISA multimodal large language model language-instructed reasoning segmentation Segment Anything Model temporal dynamic understanding temporal context spatial detail Sparse Dense Sampling strategy One-Token-Seg-All <TRK> token ReasonVOS benchmark video object segmentation language-instructed object segmentation unified foundation model