Animated Stickers: Bringing Stickers to Life with Video Diffusion
We introduce animated stickers, a video diffusion model which generates an
animation conditioned on a text prompt and static sticker image. Our model is
built on top of the state-of-the-art Emu text-to-image model, with the addition
of temporal layers to model motion. Due to the domain gap, i.e. differences in
visual and motion style, a model which performed well on generating natural
videos can no longer generate vivid videos when applied to stickers. To bridge
this gap, we employ a two-stage finetuning pipeline: first with weakly
in-domain data, followed by human-in-the-loop (HITL) strategy which we term
ensemble-of-teachers. It distills the best qualities of multiple teachers into
a smaller student model. We show that this strategy allows us to specifically
target improvements to motion quality while maintaining the style from the
static image. With inference optimizations, our model is able to generate an
eight-frame video with high-quality, interesting, and relevant motion in under
one second.
65c991bafe3148b9187d576c
David Yan
65c991bafe3148b9187d576d
Winnie Zhang
admin_assigned
2024-02-12T17:07:18.270Z
64564b304a7ffb7d5a47f515
/avatars/b848f02609d9ee8ab7df350ccb274c23.svg
Winnie Zhang
user
WinnieJackJack
65c991bafe3148b9187d576e
Luxin Zhang
claimed_verified
2025-04-02T08:58:33.971Z
67ecad336560da48c5c2bf08
/avatars/5ef8381504bfc226b0d82c5a36f04ca3.svg
Luxin Zhang
user
luxinzhang
65c991bafe3148b9187d576f
Anmol Kalia
admin_assigned
2024-02-12T17:07:32.582Z
6573c2a8769f3ee9bdf98a74
/avatars/df40b6153ec7368561049746a9c8adc2.svg
Anmol Kalia
user
anmolkalia
65c991bafe3148b9187d5770
Dingkang Wang
65c991bafe3148b9187d5771
Ankit Ramchandani
65c991bafe3148b9187d5772
Miao Liu
65c991bafe3148b9187d5773
Albert Pumarola
65c991bafe3148b9187d5774
Edgar Schoenfeld
admin_assigned
2024-02-12T17:08:30.615Z
6571c662d5c6a6d3b0bdae88
/avatars/8080bcdf8b331f62383b724050189660.svg
Edgar Schoenfeld
user
edgarschoenfeld
65c991bafe3148b9187d5775
Elliot Blanchard
admin_assigned
2024-02-12T17:08:36.850Z
6563cf4a718b9db5215d07b4
/avatars/20e5be0bd3fcd888959a0af0f0b462e4.svg
Elliot Blanchard
user
elliotblanchard
65c991bafe3148b9187d5776
Krishna Narni
admin_assigned
2024-02-12T17:08:42.582Z
63441505a56e97f0495056a6
/avatars/1497958399aeff861ed0b2fbb2a9b7b7.svg
Krishna Narni
user
krishnanarni
65c991bafe3148b9187d5777
Yaqiao Luo
admin_assigned
2024-02-12T17:08:52.709Z
63d2a5a6c20657d24347000c
https://cdn-avatars.huggingface.co/v1/production/uploads/1674749322268-noauth.jpeg
Luo
user
Yaqiao
65c991bafe3148b9187d5778
Lawrence Chen
65c991bafe3148b9187d5779
Guan Pang
65c991bafe3148b9187d577a
Ali Thabet
65c991bafe3148b9187d577b
Peter Vajda
admin_assigned
2024-02-12T17:11:37.356Z
636f4b35af79dd193603342b
/avatars/577328195738d92ac41a3d0ff3108d7c.svg
Peter Vajda
user
vaphab
65c991bafe3148b9187d577c
Amy Bearman
65c991bafe3148b9187d577d
Licheng Yu
admin_assigned
2024-02-12T17:12:02.180Z
6372b1f9bd81fae2b3a712e8
/avatars/bcdf7b540cc94882e0ae6035bb07a8f2.svg
Licheng Yu
user
lichengyu
Animated Stickers: Bringing Stickers to Life with Video Diffusion
We introduce animated stickers, a video diffusion model which generates an
animation conditioned on a text prompt and static sticker image. Our model is
built on top of the state-of-the-art Emu text-to-image model, with the addition
of temporal layers to model motion. Due to the domain gap, i.e. differences in
visual and motion style, a model which performed well on generating natural
videos can no longer generate vivid videos when applied to stickers. To bridge
this gap, we employ a two-stage finetuning pipeline: first with weakly
in-domain data, followed by human-in-the-loop (HITL) strategy which we term
ensemble-of-teachers. It distills the best qualities of multiple teachers into
a smaller student model. We show that this strategy allows us to specifically
target improvements to motion quality while maintaining the style from the
static image. With inference optimizations, our model is able to generate an
eight-frame video with high-quality, interesting, and relevant motion in under
one second.
A video diffusion model generates animations from text prompts and static images using a two-stage finetuning pipeline, achieving high-quality motion in under one second.
video diffusion model text-to-image model temporal layers domain gap weakly in-domain data human-in-the-loop ensemble-of-teachers motion quality inference optimizations