MonST3R: A Simple Approach for Estimating Geometry in the Presence of
  Motion
Estimating geometry from dynamic scenes, where objects move and deform over
time, remains a core challenge in computer vision. Current approaches often
rely on multi-stage pipelines or global optimizations that decompose the
problem into subtasks, like depth and flow, leading to complex systems prone to
errors. In this paper, we present Motion DUSt3R (MonST3R), a novel
geometry-first approach that directly estimates per-timestep geometry from
dynamic scenes. Our key insight is that by simply estimating a pointmap for
each timestep, we can effectively adapt DUST3R's representation, previously
only used for static scenes, to dynamic scenes. However, this approach presents
a significant challenge: the scarcity of suitable training data, namely
dynamic, posed videos with depth labels. Despite this, we show that by posing
the problem as a fine-tuning task, identifying several suitable datasets, and
strategically training the model on this limited data, we can surprisingly
enable the model to handle dynamics, even without an explicit motion
representation. Based on this, we introduce new optimizations for several
downstream video-specific tasks and demonstrate strong performance on video
depth and camera pose estimation, outperforming prior work in terms of
robustness and efficiency. Moreover, MonST3R shows promising results for
primarily feed-forward 4D reconstruction.
6704c2324d2db34694c9cd2b
Junyi Zhang
claimed_verified
2024-10-08T07:19:11.801Z
62f0ecd2700bdc19558360de
/avatars/5325b4b763f30c41f30e3aec0d2b59fa.svg
Junyi Zhang
user
Junyi42
6704c2324d2db34694c9cd2c
Charles Herrmann
admin_assigned
2024-10-08T09:51:09.915Z
64c7f088c864d962eddeaf8c
/avatars/e6364c56e6cb6907b6dd2d34eed83720.svg
Charles Herrmann
user
irwinherrmann
6704c2324d2db34694c9cd2d
Junhwa Hur
admin_assigned
2024-10-08T09:51:41.442Z
647f3046b0e96764589e107a
/avatars/1074ee5ec1b2d5d297406f9d455213c4.svg
Junhwa Hur
user
hurjunhwa
6704c2324d2db34694c9cd2e
Varun Jampani
6704c2324d2db34694c9cd2f
Trevor Darrell
admin_assigned
2024-10-08T09:51:33.519Z
64cbdf02f103036e23d1c7f3
/avatars/496069463900dea20929b57381182d39.svg
Trevor Darrell
user
trevordarrell
6704c2324d2db34694c9cd30
Forrester Cole
admin_assigned
2024-10-08T09:51:26.698Z
635003011d81beb8e244e68d
/avatars/32f46e6778d1c52ebdd6f2511ba48257.svg
Forrester Cole
user
fcole
6704c2324d2db34694c9cd31
Deqing Sun
6704c2324d2db34694c9cd32
Ming-Hsuan Yang
MonST3R: A Simple Approach for Estimating Geometry in the Presence of
  Motion
Estimating geometry from dynamic scenes, where objects move and deform over
time, remains a core challenge in computer vision. Current approaches often
rely on multi-stage pipelines or global optimizations that decompose the
problem into subtasks, like depth and flow, leading to complex systems prone to
errors. In this paper, we present Motion DUSt3R (MonST3R), a novel
geometry-first approach that directly estimates per-timestep geometry from
dynamic scenes. Our key insight is that by simply estimating a pointmap for
each timestep, we can effectively adapt DUST3R's representation, previously
only used for static scenes, to dynamic scenes. However, this approach presents
a significant challenge: the scarcity of suitable training data, namely
dynamic, posed videos with depth labels. Despite this, we show that by posing
the problem as a fine-tuning task, identifying several suitable datasets, and
strategically training the model on this limited data, we can surprisingly
enable the model to handle dynamics, even without an explicit motion
representation. Based on this, we introduce new optimizations for several
downstream video-specific tasks and demonstrate strong performance on video
depth and camera pose estimation, outperforming prior work in terms of
robustness and efficiency. Moreover, MonST3R shows promising results for
primarily feed-forward 4D reconstruction.
A novel geometry-first approach, Motion DUSt3R, directly estimates per-timestep geometry in dynamic scenes, demonstrating strong performance on tasks like video depth and camera pose estimation through fine-tuning on limited dynamic datasets.
dynamic scenes depth estimation optical flow fine-tuning 4D reconstruction