MOSEv2: A More Challenging Dataset for Video Object Segmentation in
  Complex Scenes
Video object segmentation (VOS) aims to segment specified target objects
throughout a video. Although state-of-the-art methods have achieved impressive
performance (e.g., 90+% J&F) on existing benchmarks such as DAVIS and
YouTube-VOS, these datasets primarily contain salient, dominant, and isolated
objects, limiting their generalization to real-world scenarios. To advance VOS
toward more realistic environments, coMplex video Object SEgmentation (MOSEv1)
was introduced to facilitate VOS research in complex scenes. Building on the
strengths and limitations of MOSEv1, we present MOSEv2, a significantly more
challenging dataset designed to further advance VOS methods under real-world
conditions. MOSEv2 consists of 5,024 videos and over 701,976 high-quality masks
for 10,074 objects across 200 categories. Compared to its predecessor, MOSEv2
introduces significantly greater scene complexity, including more frequent
object disappearance and reappearance, severe occlusions and crowding, smaller
objects, as well as a range of new challenges such as adverse weather (e.g.,
rain, snow, fog), low-light scenes (e.g., nighttime, underwater), multi-shot
sequences, camouflaged objects, non-physical targets (e.g., shadows,
reflections), scenarios requiring external knowledge, etc. We benchmark 20
representative VOS methods under 5 different settings and observe consistent
performance drops. For example, SAM2 drops from 76.4% on MOSEv1 to only 50.9%
on MOSEv2. We further evaluate 9 video object tracking methods and find similar
declines, demonstrating that MOSEv2 presents challenges across tasks. These
results highlight that despite high accuracy on existing datasets, current VOS
methods still struggle under real-world complexities. MOSEv2 is publicly
available at https://MOSE.video.
689554af48b0ae5ca2710cdd
Henghui Ding
689554af48b0ae5ca2710cde
Kaining Ying
689554af48b0ae5ca2710cdf
Chang Liu
689554af48b0ae5ca2710ce0
Shuting He
689554af48b0ae5ca2710ce1
Xudong Jiang
689554af48b0ae5ca2710ce2
Yu-Gang Jiang
689554af48b0ae5ca2710ce3
Philip H. S. Torr
689554af48b0ae5ca2710ce4
Song Bai
MOSEv2: A More Challenging Dataset for Video Object Segmentation in
  Complex Scenes
Video object segmentation (VOS) aims to segment specified target objects
throughout a video. Although state-of-the-art methods have achieved impressive
performance (e.g., 90+% J&F) on existing benchmarks such as DAVIS and
YouTube-VOS, these datasets primarily contain salient, dominant, and isolated
objects, limiting their generalization to real-world scenarios. To advance VOS
toward more realistic environments, coMplex video Object SEgmentation (MOSEv1)
was introduced to facilitate VOS research in complex scenes. Building on the
strengths and limitations of MOSEv1, we present MOSEv2, a significantly more
challenging dataset designed to further advance VOS methods under real-world
conditions. MOSEv2 consists of 5,024 videos and over 701,976 high-quality masks
for 10,074 objects across 200 categories. Compared to its predecessor, MOSEv2
introduces significantly greater scene complexity, including more frequent
object disappearance and reappearance, severe occlusions and crowding, smaller
objects, as well as a range of new challenges such as adverse weather (e.g.,
rain, snow, fog), low-light scenes (e.g., nighttime, underwater), multi-shot
sequences, camouflaged objects, non-physical targets (e.g., shadows,
reflections), scenarios requiring external knowledge, etc. We benchmark 20
representative VOS methods under 5 different settings and observe consistent
performance drops. For example, SAM2 drops from 76.4% on MOSEv1 to only 50.9%
on MOSEv2. We further evaluate 9 video object tracking methods and find similar
declines, demonstrating that MOSEv2 presents challenges across tasks. These
results highlight that despite high accuracy on existing datasets, current VOS
methods still struggle under real-world complexities. MOSEv2 is publicly
available at https://MOSE.video.
MOSEv2, a more challenging dataset, highlights the limitations of current VOS methods in real-world scenarios with increased complexity and diverse challenges.
Video object segmentation VOS DAVIS YouTube-VOS J&F MOSEv1 MOSEv2 high-quality masks scene complexity object disappearance occlusions crowding adverse weather low-light scenes multi-shot sequences camouflaged objects non-physical targets external knowledge SAM2 video object tracking