Autoregressive Universal Video Segmentation Model
Recent video foundation models such as SAM2 excel at prompted video
segmentation by treating masks as a general-purpose primitive. However, many
real-world settings require unprompted segmentation that aims to detect and
track all objects in a video without external cues, leaving today's landscape
fragmented across task-specific models and pipelines. We recast streaming video
segmentation as sequential mask prediction, analogous to language modeling, and
introduce the Autoregressive Universal Segmentation Model (AUSM), a single
architecture that unifies both prompted and unprompted video segmentation.
Built on recent state-space models, AUSM maintains a fixed-size spatial state
and scales to video streams of arbitrary length. Furthermore, all components of
AUSM are designed for parallel training across frames, yielding substantial
speedups over iterative training. On standard benchmarks (DAVIS17, YouTube-VOS
2018 & 2019, MOSE, YouTube-VIS 2019 & 2021, and OVIS) AUSM outperforms prior
universal streaming video segmentation methods and achieves up to 2.5x faster
training on 16-frame sequences.
68ae674d364411bea07df6f2
Miran Heo
68ae674d364411bea07df6f3
Sukjun Hwang
68ae674d364411bea07df6f4
Min-Hung Chen
claimed_verified
2025-08-27T07:11:04.091Z
64ae22dd1aee69ece065cdcd
https://cdn-avatars.huggingface.co/v1/production/uploads/64ae22dd1aee69ece065cdcd/JG7QaHIrr4i2k4uwR4pZK.png
Min-Hung Chen
user
cmhungsteve
68ae674d364411bea07df6f5
Yu-Chiang Frank Wang
68ae674d364411bea07df6f6
Albert Gu
68ae674d364411bea07df6f7
Seon Joo Kim
68ae674d364411bea07df6f8
Ryo Hachiuma
claimed_verified
2025-09-09T13:50:17.705Z
65b33e5f7cd0069ad648c4e8
/avatars/1a746ea535cffa92ea08006e05ea414a.svg
Ryo Hachiuma
user
rhachiuma
Autoregressive Universal Video Segmentation Model
Recent video foundation models such as SAM2 excel at prompted video
segmentation by treating masks as a general-purpose primitive. However, many
real-world settings require unprompted segmentation that aims to detect and
track all objects in a video without external cues, leaving today's landscape
fragmented across task-specific models and pipelines. We recast streaming video
segmentation as sequential mask prediction, analogous to language modeling, and
introduce the Autoregressive Universal Segmentation Model (AUSM), a single
architecture that unifies both prompted and unprompted video segmentation.
Built on recent state-space models, AUSM maintains a fixed-size spatial state
and scales to video streams of arbitrary length. Furthermore, all components of
AUSM are designed for parallel training across frames, yielding substantial
speedups over iterative training. On standard benchmarks (DAVIS17, YouTube-VOS
2018 & 2019, MOSE, YouTube-VIS 2019 & 2021, and OVIS) AUSM outperforms prior
universal streaming video segmentation methods and achieves up to 2.5x faster
training on 16-frame sequences.
AUSM, an autoregressive universal segmentation model, unifies prompted and unprompted video segmentation by treating it as sequential mask prediction, achieving superior performance and faster training on standard benchmarks.
sequential mask prediction language modeling state-space models parallel training DAVIS17 YouTube-VOS 2018 & 2019 MOSE YouTube-VIS 2019 & 2021 OVIS