Trace Anything: Representing Any Video in 4D via Trajectory Fields
Effective spatio-temporal representation is fundamental to modeling,
understanding, and predicting dynamics in videos. The atomic unit of a video,
the pixel, traces a continuous 3D trajectory over time, serving as the
primitive element of dynamics. Based on this principle, we propose representing
any video as a Trajectory Field: a dense mapping that assigns a continuous 3D
trajectory function of time to each pixel in every frame. With this
representation, we introduce Trace Anything, a neural network that predicts the
entire trajectory field in a single feed-forward pass. Specifically, for each
pixel in each frame, our model predicts a set of control points that
parameterizes a trajectory (i.e., a B-spline), yielding its 3D position at
arbitrary query time instants. We trained the Trace Anything model on
large-scale 4D data, including data from our new platform, and our experiments
demonstrate that: (i) Trace Anything achieves state-of-the-art performance on
our new benchmark for trajectory field estimation and performs competitively on
established point-tracking benchmarks; (ii) it offers significant efficiency
gains thanks to its one-pass paradigm, without requiring iterative optimization
or auxiliary estimators; and (iii) it exhibits emergent abilities, including
goal-conditioned manipulation, motion forecasting, and spatio-temporal fusion.
Project page: https://trace-anything.github.io/.
68f04f2d36f8b025381e18e3
Xinhang Liu
claimed_verified
2025-10-16T06:39:12.666Z
64b6e2abd3711dd619e476fc
/avatars/430f53c44d12c578a5128e42ada62527.svg
Xinhang Liu
user
xinhang0111
68f04f2d36f8b025381e18e4
Yuxi Xiao
claimed_verified
2025-10-16T07:25:42.979Z
6688a8f30bf195d6e53ac28d
https://cdn-avatars.huggingface.co/v1/production/uploads/6688a8f30bf195d6e53ac28d/5izbVmdCWWwA1wBjcxZPB.jpeg
Yuxi Xiao
user
Yuxihenry
68f04f2d36f8b025381e18e5
Donny Y. Chen
claimed_verified
2025-10-16T06:39:10.430Z
653862bdbe39573b3b247b44
https://cdn-avatars.huggingface.co/v1/production/uploads/653862bdbe39573b3b247b44/oMqOC4USbQlPX5HFLDXt6.jpeg
Donny Chen
user
donydchen
68f04f2d36f8b025381e18e6
Jiashi Feng
68f04f2d36f8b025381e18e7
Yu-Wing Tai
68f04f2d36f8b025381e18e8
Chi-Keung Tang
68f04f2d36f8b025381e18e9
Bingyi Kang
Trace Anything: Representing Any Video in 4D via Trajectory Fields
Effective spatio-temporal representation is fundamental to modeling,
understanding, and predicting dynamics in videos. The atomic unit of a video,
the pixel, traces a continuous 3D trajectory over time, serving as the
primitive element of dynamics. Based on this principle, we propose representing
any video as a Trajectory Field: a dense mapping that assigns a continuous 3D
trajectory function of time to each pixel in every frame. With this
representation, we introduce Trace Anything, a neural network that predicts the
entire trajectory field in a single feed-forward pass. Specifically, for each
pixel in each frame, our model predicts a set of control points that
parameterizes a trajectory (i.e., a B-spline), yielding its 3D position at
arbitrary query time instants. We trained the Trace Anything model on
large-scale 4D data, including data from our new platform, and our experiments
demonstrate that: (i) Trace Anything achieves state-of-the-art performance on
our new benchmark for trajectory field estimation and performs competitively on
established point-tracking benchmarks; (ii) it offers significant efficiency
gains thanks to its one-pass paradigm, without requiring iterative optimization
or auxiliary estimators; and (iii) it exhibits emergent abilities, including
goal-conditioned manipulation, motion forecasting, and spatio-temporal fusion.
Project page: https://trace-anything.github.io/.
Trace Anything, a neural network, predicts video trajectories in a single pass, achieving state-of-the-art performance and demonstrating efficiency and emergent abilities like motion forecasting.
Trajectory Field Trace Anything B-spline trajectory field estimation point-tracking benchmarks goal-conditioned manipulation motion forecasting spatio-temporal fusion