Scaling Instruction-Based Video Editing with a High-Quality Synthetic
  Dataset
Instruction-based video editing promises to democratize content creation, yet
its progress is severely hampered by the scarcity of large-scale, high-quality
training data. We introduce Ditto, a holistic framework designed to tackle this
fundamental challenge. At its heart, Ditto features a novel data generation
pipeline that fuses the creative diversity of a leading image editor with an
in-context video generator, overcoming the limited scope of existing models. To
make this process viable, our framework resolves the prohibitive cost-quality
trade-off by employing an efficient, distilled model architecture augmented by
a temporal enhancer, which simultaneously reduces computational overhead and
improves temporal coherence. Finally, to achieve full scalability, this entire
pipeline is driven by an intelligent agent that crafts diverse instructions and
rigorously filters the output, ensuring quality control at scale. Using this
framework, we invested over 12,000 GPU-days to build Ditto-1M, a new dataset of
one million high-fidelity video editing examples. We trained our model, Editto,
on Ditto-1M with a curriculum learning strategy. The results demonstrate
superior instruction-following ability and establish a new state-of-the-art in
instruction-based video editing.
68f5a20f8589920bf4d3212e
Qingyan Bai
claimed_verified
2025-10-20T12:10:45.283Z
63f0baf66309c84d5f4a2226
https://cdn-avatars.huggingface.co/v1/production/uploads/63f0baf66309c84d5f4a2226/ihOgtwseRkfP1t-60IgyT.jpeg
Qingyan
user
QingyanBai
68f5a20f8589920bf4d3212f
Qiuyu Wang
claimed_verified
2025-10-20T12:10:49.093Z
64981bea09cea550852652af
/avatars/df528e9008972c8e5ae4d278e617476c.svg
Qiuyu Wang
user
qiuyuu
68f5a20f8589920bf4d32130
Hao Ouyang
68f5a20f8589920bf4d32131
Yue Yu
claimed_verified
2025-10-20T12:10:54.813Z
662128ec9ca2cd4e6db2fb44
https://cdn-avatars.huggingface.co/v1/production/uploads/662128ec9ca2cd4e6db2fb44/uUg1V-pVfxT3mLuFgJuAN.jpeg
Bruce Yu
user
bruceyyu
68f5a20f8589920bf4d32132
Hanlin Wang
68f5a20f8589920bf4d32133
Wen Wang
claimed_verified
2025-10-20T12:10:52.179Z
63f089456309c84d5f47f951
/avatars/04b926a7f2ad091ee00fef0c59903492.svg
Wen Wang
user
wwen1997
68f5a20f8589920bf4d32134
Ka Leong Cheng
68f5a20f8589920bf4d32135
Shuailei Ma
68f5a20f8589920bf4d32136
Yanhong Zeng
68f5a20f8589920bf4d32137
Zichen Liu
68f5a20f8589920bf4d32138
Yinghao Xu
68f5a20f8589920bf4d32139
Yujun Shen
68f5a20f8589920bf4d3213a
Qifeng Chen
Scaling Instruction-Based Video Editing with a High-Quality Synthetic
  Dataset
Instruction-based video editing promises to democratize content creation, yet
its progress is severely hampered by the scarcity of large-scale, high-quality
training data. We introduce Ditto, a holistic framework designed to tackle this
fundamental challenge. At its heart, Ditto features a novel data generation
pipeline that fuses the creative diversity of a leading image editor with an
in-context video generator, overcoming the limited scope of existing models. To
make this process viable, our framework resolves the prohibitive cost-quality
trade-off by employing an efficient, distilled model architecture augmented by
a temporal enhancer, which simultaneously reduces computational overhead and
improves temporal coherence. Finally, to achieve full scalability, this entire
pipeline is driven by an intelligent agent that crafts diverse instructions and
rigorously filters the output, ensuring quality control at scale. Using this
framework, we invested over 12,000 GPU-days to build Ditto-1M, a new dataset of
one million high-fidelity video editing examples. We trained our model, Editto,
on Ditto-1M with a curriculum learning strategy. The results demonstrate
superior instruction-following ability and establish a new state-of-the-art in
instruction-based video editing.
Ditto framework addresses data scarcity in instruction-based video editing by generating a large dataset and using a curriculum learning strategy to train Editto, achieving superior instruction-following ability.
data generation pipeline image editor in-context video generator distilled model architecture temporal enhancer temporal coherence intelligent agent curriculum learning strategy instruction-following ability