Story-to-Motion: Synthesizing Infinite and Controllable Character
  Animation from Long Text
Generating natural human motion from a story has the potential to transform
the landscape of animation, gaming, and film industries. A new and challenging
task, Story-to-Motion, arises when characters are required to move to various
locations and perform specific motions based on a long text description. This
task demands a fusion of low-level control (trajectories) and high-level
control (motion semantics). Previous works in character control and
text-to-motion have addressed related aspects, yet a comprehensive solution
remains elusive: character control methods do not handle text description,
whereas text-to-motion methods lack position constraints and often produce
unstable motions. In light of these limitations, we propose a novel system that
generates controllable, infinitely long motions and trajectories aligned with
the input text. (1) We leverage contemporary Large Language Models to act as a
text-driven motion scheduler to extract a series of (text, position, duration)
pairs from long text. (2) We develop a text-driven motion retrieval scheme that
incorporates motion matching with motion semantic and trajectory constraints.
(3) We design a progressive mask transformer that addresses common artifacts in
the transition motion such as unnatural pose and foot sliding. Beyond its
pioneering role as the first comprehensive solution for Story-to-Motion, our
system undergoes evaluation across three distinct sub-tasks: trajectory
following, temporal action composition, and motion blending, where it
outperforms previous state-of-the-art motion synthesis methods across the
board. Homepage: https://story2motion.github.io/.
6552ef3b73a5a6f9385f8fae
Zhongfei Qing
6552ef3b73a5a6f9385f8faf
Zhongang Cai
admin_assigned
2023-11-14T10:36:25.229Z
652d06833b5997ed71ce5c46
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/xZTXEcnEogEmBm_ledJQr.jpeg
Zhongang Cai
user
caizhongang
6552ef3b73a5a6f9385f8fb0
Zhitao Yang
6552ef3b73a5a6f9385f8fb1
Lei Yang
claimed_verified
2025-03-10T08:06:57.569Z
6626a471430a124253f197c8
/avatars/f5747fdbe495d1296fed9d16d8c95857.svg
yl-1993
user
yl-1993
Story-to-Motion: Synthesizing Infinite and Controllable Character
  Animation from Long Text
Generating natural human motion from a story has the potential to transform
the landscape of animation, gaming, and film industries. A new and challenging
task, Story-to-Motion, arises when characters are required to move to various
locations and perform specific motions based on a long text description. This
task demands a fusion of low-level control (trajectories) and high-level
control (motion semantics). Previous works in character control and
text-to-motion have addressed related aspects, yet a comprehensive solution
remains elusive: character control methods do not handle text description,
whereas text-to-motion methods lack position constraints and often produce
unstable motions. In light of these limitations, we propose a novel system that
generates controllable, infinitely long motions and trajectories aligned with
the input text. (1) We leverage contemporary Large Language Models to act as a
text-driven motion scheduler to extract a series of (text, position, duration)
pairs from long text. (2) We develop a text-driven motion retrieval scheme that
incorporates motion matching with motion semantic and trajectory constraints.
(3) We design a progressive mask transformer that addresses common artifacts in
the transition motion such as unnatural pose and foot sliding. Beyond its
pioneering role as the first comprehensive solution for Story-to-Motion, our
system undergoes evaluation across three distinct sub-tasks: trajectory
following, temporal action composition, and motion blending, where it
outperforms previous state-of-the-art motion synthesis methods across the
board. Homepage: https://story2motion.github.io/.
Large Language Models text-driven motion scheduler motion retrieval motion matching motion semantic trajectory constraints progressive mask transformer unnatural pose foot sliding trajectory following temporal action composition motion blending motion synthesis