Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers
Video Joint Embedding Predictive Architectures (V-JEPA) learn generalizable
off-the-shelf video representation by predicting masked regions in latent space
with an exponential moving average (EMA)-updated teacher. While EMA prevents
representation collapse, it complicates scalable model selection and couples
teacher and student architectures. We revisit masked-latent prediction and show
that a frozen teacher suffices. Concretely, we (i) train a target encoder with
a simple pixel-reconstruction objective under V-JEPA masking, then (ii) freeze
it and train a student to predict the teacher's latents on masked regions. This
leads to a two-stage, unregularized scheme that we refer to as SALT
(Static-teacher Asymmetric Latent Training). SALT decouples optimization into
pixel reconstruction (teacher) and masked latent prediction (student),
increasing transparency, efficiency, and scalability while preserving the
ability of representation to generalize under frozen evaluation. Empirically,
our student models outperform recently proposed V-JEPA 2 encoders under frozen
backbone evaluation across diverse benchmarks. They are also more
compute-optimal: at matched pretraining FLOPs, our method achieves higher
probing accuracy, and its scaling curves dominate V-JEPA's accuracy-FLOPs
Pareto frontier. Finally, we find that student quality is remarkably robust to
teacher quality: high-performing students emerge even with small, sub-optimal
teachers. This points to a compute budget allocation that should overwhelmingly
favor the student. These results position SALT as a simple, scalable, and
compute-efficient alternative to EMA-based self-distillation for video
representation learning.
68dc47b84159d1f2418f9916
Xianhang Li
claimed_verified
2025-10-01T10:22:46.847Z
645e72d4b5c9a8666d0ddc1c
/avatars/f0f1684731eb5ae148eadf841cceb18e.svg
xhl
user
Xianhang
68dc47b84159d1f2418f9917
Chen Huang
68dc47b84159d1f2418f9918
Chun-Liang Li
68dc47b84159d1f2418f9919
Eran Malach
68dc47b84159d1f2418f991a
Josh Susskind
68dc47b84159d1f2418f991b
Vimal Thilak
claimed_verified
2025-10-02T03:09:04.564Z
6737e918edaf7e05e4b35791
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/uk--P2mE2Jeg1QoxmXSTS.png
Vimal Thilak
user
vimalthilak
68dc47b84159d1f2418f991c
Etai Littwin
Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers
Video Joint Embedding Predictive Architectures (V-JEPA) learn generalizable
off-the-shelf video representation by predicting masked regions in latent space
with an exponential moving average (EMA)-updated teacher. While EMA prevents
representation collapse, it complicates scalable model selection and couples
teacher and student architectures. We revisit masked-latent prediction and show
that a frozen teacher suffices. Concretely, we (i) train a target encoder with
a simple pixel-reconstruction objective under V-JEPA masking, then (ii) freeze
it and train a student to predict the teacher's latents on masked regions. This
leads to a two-stage, unregularized scheme that we refer to as SALT
(Static-teacher Asymmetric Latent Training). SALT decouples optimization into
pixel reconstruction (teacher) and masked latent prediction (student),
increasing transparency, efficiency, and scalability while preserving the
ability of representation to generalize under frozen evaluation. Empirically,
our student models outperform recently proposed V-JEPA 2 encoders under frozen
backbone evaluation across diverse benchmarks. They are also more
compute-optimal: at matched pretraining FLOPs, our method achieves higher
probing accuracy, and its scaling curves dominate V-JEPA's accuracy-FLOPs
Pareto frontier. Finally, we find that student quality is remarkably robust to
teacher quality: high-performing students emerge even with small, sub-optimal
teachers. This points to a compute budget allocation that should overwhelmingly
favor the student. These results position SALT as a simple, scalable, and
compute-efficient alternative to EMA-based self-distillation for video
representation learning.
SALT, a two-stage training method using a frozen teacher, achieves better video representation learning with higher efficiency and scalability compared to EMA-based approaches.
Video Joint Embedding Predictive Architectures V-JEPA masked regions latent space exponential moving average EMA representation collapse pixel-reconstruction objective SALT Static-teacher Asymmetric Latent Training probing accuracy Pareto frontier