EgoTwin: Dreaming Body and View in First Person
While exocentric video synthesis has achieved great progress, egocentric
video generation remains largely underexplored, which requires modeling
first-person view content along with camera motion patterns induced by the
wearer's body movements. To bridge this gap, we introduce a novel task of joint
egocentric video and human motion generation, characterized by two key
challenges: 1) Viewpoint Alignment: the camera trajectory in the generated
video must accurately align with the head trajectory derived from human motion;
2) Causal Interplay: the synthesized human motion must causally align with the
observed visual dynamics across adjacent video frames. To address these
challenges, we propose EgoTwin, a joint video-motion generation framework built
on the diffusion transformer architecture. Specifically, EgoTwin introduces a
head-centric motion representation that anchors the human motion to the head
joint and incorporates a cybernetics-inspired interaction mechanism that
explicitly captures the causal interplay between video and motion within
attention operations. For comprehensive evaluation, we curate a large-scale
real-world dataset of synchronized text-video-motion triplets and design novel
metrics to assess video-motion consistency. Extensive experiments demonstrate
the effectiveness of the EgoTwin framework.
68ac07b886b21a0e2e358b54
Jingqiao Xiu
admin_assigned
2025-08-25T20:35:06.193Z
6474d585bb0e9dd77647fd1a
/avatars/95cd50f164cb8b3e8885235fd78bc7e7.svg
Qogir
user
JingqiaoXiu
68ac07b886b21a0e2e358b55
Fangzhou Hong
68ac07b886b21a0e2e358b56
Yicong Li
68ac07b886b21a0e2e358b57
Mengze Li
68ac07b886b21a0e2e358b58
Wentao Wang
68ac07b886b21a0e2e358b59
Sirui Han
68ac07b886b21a0e2e358b5a
Liang Pan
68ac07b886b21a0e2e358b5b
Ziwei Liu
admin_assigned
2025-08-25T20:35:00.219Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
EgoTwin: Dreaming Body and View in First Person
While exocentric video synthesis has achieved great progress, egocentric
video generation remains largely underexplored, which requires modeling
first-person view content along with camera motion patterns induced by the
wearer's body movements. To bridge this gap, we introduce a novel task of joint
egocentric video and human motion generation, characterized by two key
challenges: 1) Viewpoint Alignment: the camera trajectory in the generated
video must accurately align with the head trajectory derived from human motion;
2) Causal Interplay: the synthesized human motion must causally align with the
observed visual dynamics across adjacent video frames. To address these
challenges, we propose EgoTwin, a joint video-motion generation framework built
on the diffusion transformer architecture. Specifically, EgoTwin introduces a
head-centric motion representation that anchors the human motion to the head
joint and incorporates a cybernetics-inspired interaction mechanism that
explicitly captures the causal interplay between video and motion within
attention operations. For comprehensive evaluation, we curate a large-scale
real-world dataset of synchronized text-video-motion triplets and design novel
metrics to assess video-motion consistency. Extensive experiments demonstrate
the effectiveness of the EgoTwin framework.
EgoTwin, a diffusion transformer framework, addresses viewpoint alignment and causal interplay in joint egocentric video and human motion generation using a head-centric motion representation and cybernetics-inspired interaction mechanism.
exocentric video synthesis egocentric video generation camera trajectory head trajectory human motion viewpoint alignment causal interplay diffusion transformer head-centric motion representation cybernetics-inspired interaction mechanism