ROSE: Remove Objects with Side Effects in Videos
Video object removal has achieved advanced performance due to the recent
success of video generative models. However, when addressing the side effects
of objects, e.g., their shadows and reflections, existing works struggle to
eliminate these effects for the scarcity of paired video data as supervision.
This paper presents ROSE, termed Remove Objects with Side Effects, a framework
that systematically studies the object's effects on environment, which can be
categorized into five common cases: shadows, reflections, light, translucency
and mirror. Given the challenges of curating paired videos exhibiting the
aforementioned effects, we leverage a 3D rendering engine for synthetic data
generation. We carefully construct a fully-automatic pipeline for data
preparation, which simulates a large-scale paired dataset with diverse scenes,
objects, shooting angles, and camera trajectories. ROSE is implemented as an
video inpainting model built on diffusion transformer. To localize all
object-correlated areas, the entire video is fed into the model for
reference-based erasing. Moreover, additional supervision is introduced to
explicitly predict the areas affected by side effects, which can be revealed
through the differential mask between the paired videos. To fully investigate
the model performance on various side effect removal, we presents a new
benchmark, dubbed ROSE-Bench, incorporating both common scenarios and the five
special side effects for comprehensive evaluation. Experimental results
demonstrate that ROSE achieves superior performance compared to existing video
object erasing models and generalizes well to real-world video scenarios. The
project page is https://rose2025-inpaint.github.io/.
68ae7a62364411bea07df79e
Chenxuan Miao
68ae7a62364411bea07df79f
Yutong Feng
68ae7a62364411bea07df7a0
Jianshu Zeng
68ae7a62364411bea07df7a1
Zixiang Gao
68ae7a62364411bea07df7a2
Hantang Liu
68ae7a62364411bea07df7a3
Yunfeng Yan
68ae7a62364411bea07df7a4
Donglian Qi
68ae7a62364411bea07df7a5
Xi Chen
68ae7a62364411bea07df7a6
Bin Wang
68ae7a62364411bea07df7a7
Hengshuang Zhao
ROSE: Remove Objects with Side Effects in Videos
Video object removal has achieved advanced performance due to the recent
success of video generative models. However, when addressing the side effects
of objects, e.g., their shadows and reflections, existing works struggle to
eliminate these effects for the scarcity of paired video data as supervision.
This paper presents ROSE, termed Remove Objects with Side Effects, a framework
that systematically studies the object's effects on environment, which can be
categorized into five common cases: shadows, reflections, light, translucency
and mirror. Given the challenges of curating paired videos exhibiting the
aforementioned effects, we leverage a 3D rendering engine for synthetic data
generation. We carefully construct a fully-automatic pipeline for data
preparation, which simulates a large-scale paired dataset with diverse scenes,
objects, shooting angles, and camera trajectories. ROSE is implemented as an
video inpainting model built on diffusion transformer. To localize all
object-correlated areas, the entire video is fed into the model for
reference-based erasing. Moreover, additional supervision is introduced to
explicitly predict the areas affected by side effects, which can be revealed
through the differential mask between the paired videos. To fully investigate
the model performance on various side effect removal, we presents a new
benchmark, dubbed ROSE-Bench, incorporating both common scenarios and the five
special side effects for comprehensive evaluation. Experimental results
demonstrate that ROSE achieves superior performance compared to existing video
object erasing models and generalizes well to real-world video scenarios. The
project page is https://rose2025-inpaint.github.io/.
ROSE, a video inpainting framework using diffusion transformers, effectively removes objects and their side effects like shadows and reflections by leveraging synthetic data and differential masks.
video generative models shadows reflections light translucency mirror 3D rendering engine diffusion transformer video inpainting differential mask ROSE-Bench