Precise Action-to-Video Generation Through Visual Action Prompts
We present visual action prompts, a unified action representation for
action-to-video generation of complex high-DoF interactions while maintaining
transferable visual dynamics across domains. Action-driven video generation
faces a precision-generality trade-off: existing methods using text, primitive
actions, or coarse masks offer generality but lack precision, while
agent-centric action signals provide precision at the cost of cross-domain
transferability. To balance action precision and dynamic transferability, we
propose to "render" actions into precise visual prompts as domain-agnostic
representations that preserve both geometric precision and cross-domain
adaptability for complex actions; specifically, we choose visual skeletons for
their generality and accessibility. We propose robust pipelines to construct
skeletons from two interaction-rich data sources - human-object interactions
(HOI) and dexterous robotic manipulation - enabling cross-domain training of
action-driven generative models. By integrating visual skeletons into
pretrained video generation models via lightweight fine-tuning, we enable
precise action control of complex interaction while preserving the learning of
cross-domain dynamics. Experiments on EgoVid, RT-1 and DROID demonstrate the
effectiveness of our proposed approach. Project page:
https://zju3dv.github.io/VAP/.
68a40c90b65388761d074570
Yuang Wang
claimed_verified
2025-08-19T08:28:42.262Z
6422dd402f38c0a50cfd5405
/avatars/d3943b7b34c8026fa26eee3837590260.svg
Yuang Wang
user
angshineee
68a40c90b65388761d074571
Chao Wen
claimed_verified
2025-08-19T08:04:06.723Z
650d0c442a602ba349183fca
https://cdn-avatars.huggingface.co/v1/production/uploads/650d0c442a602ba349183fca/-UId-lPK64nRXjGxPJE9k.png
Chao Wen
user
walsvid
68a40c90b65388761d074572
Haoyu Guo
claimed_verified
2025-08-20T08:52:56.538Z
652ce0d4c543a08aa92e010f
/avatars/7978304e3fe99b0d4d0712441c6a24f3.svg
Haoyu Guo
user
ghy0324
68a40c90b65388761d074573
Sida Peng
68a40c90b65388761d074574
Minghan Qin
claimed_verified
2025-08-20T08:52:58.723Z
642002b51ccd411979d72b18
https://cdn-avatars.huggingface.co/v1/production/uploads/642002b51ccd411979d72b18/JO9e0o8fAFNC-eYFbx_JW.png
Minghan Qin
user
Qmh
68a40c90b65388761d074575
Hujun Bao
68a40c90b65388761d074576
Xiaowei Zhou
68a40c90b65388761d074577
Ruizhen Hu
Precise Action-to-Video Generation Through Visual Action Prompts
We present visual action prompts, a unified action representation for
action-to-video generation of complex high-DoF interactions while maintaining
transferable visual dynamics across domains. Action-driven video generation
faces a precision-generality trade-off: existing methods using text, primitive
actions, or coarse masks offer generality but lack precision, while
agent-centric action signals provide precision at the cost of cross-domain
transferability. To balance action precision and dynamic transferability, we
propose to "render" actions into precise visual prompts as domain-agnostic
representations that preserve both geometric precision and cross-domain
adaptability for complex actions; specifically, we choose visual skeletons for
their generality and accessibility. We propose robust pipelines to construct
skeletons from two interaction-rich data sources - human-object interactions
(HOI) and dexterous robotic manipulation - enabling cross-domain training of
action-driven generative models. By integrating visual skeletons into
pretrained video generation models via lightweight fine-tuning, we enable
precise action control of complex interaction while preserving the learning of
cross-domain dynamics. Experiments on EgoVid, RT-1 and DROID demonstrate the
effectiveness of our proposed approach. Project page:
https://zju3dv.github.io/VAP/.
Visual action prompts, using visual skeletons, enable precise action control in video generation while maintaining cross-domain transferability.
visual action prompts action-to-video generation high-DoF interactions visual dynamics action-driven video generation precision-generality trade-off agent-centric action signals visual skeletons human-object interactions dexterous robotic manipulation pretrained video generation models lightweight fine-tuning EgoVid RT-1 DROID