MotionLLM: Understanding Human Behaviors from Human Motions and Videos
This study delves into the realm of multi-modality (i.e., video and motion
modalities) human behavior understanding by leveraging the powerful
capabilities of Large Language Models (LLMs). Diverging from recent LLMs
designed for video-only or motion-only understanding, we argue that
understanding human behavior necessitates joint modeling from both videos and
motion sequences (e.g., SMPL sequences) to capture nuanced body part dynamics
and semantics effectively. In light of this, we present MotionLLM, a
straightforward yet effective framework for human motion understanding,
captioning, and reasoning. Specifically, MotionLLM adopts a unified
video-motion training strategy that leverages the complementary advantages of
existing coarse video-text data and fine-grained motion-text data to glean rich
spatial-temporal insights. Furthermore, we collect a substantial dataset,
MoVid, comprising diverse videos, motions, captions, and instructions.
Additionally, we propose the MoVid-Bench, with carefully manual annotations,
for better evaluation of human behavior understanding on video and motion.
Extensive experiments show the superiority of MotionLLM in the caption,
spatial-temporal comprehension, and reasoning ability.
6659ef49b29989570b61e15d
Ling-Hao Chen
admin_assigned
2024-05-31T19:45:39.187Z
63109a4d61cab0446e48c83b
https://cdn-avatars.huggingface.co/v1/production/uploads/63109a4d61cab0446e48c83b/JQlVkQp0ok586ND1GmB0w.png
Ling-Hao Chen
user
EvanTHU
6659ef49b29989570b61e15e
Shunlin Lu
admin_assigned
2024-05-31T19:45:45.873Z
65378eba7f862897572020fc
/avatars/5cbc4165ddf7c2c2ae886cd7fa7d1186.svg
Shunlin Lu
user
shunlinl
6659ef49b29989570b61e15f
Ailing Zeng
6659ef49b29989570b61e160
Hao Zhang
6659ef49b29989570b61e161
Benyou Wang
admin_assigned
2024-05-31T19:46:33.967Z
637c6703ca8542a0ba900ccb
/avatars/288ed63a1efa566c3f01e850c6ba5dd5.svg
Wang
user
Benyou
6659ef49b29989570b61e162
Ruimao Zhang
admin_assigned
2024-05-31T19:46:45.207Z
65324848220f490dbaa3b8c9
/avatars/cec7dcadf4657bb4f61e2c9a8075521b.svg
Zhang
user
Ruimao
6659ef49b29989570b61e163
Lei Zhang
MotionLLM: Understanding Human Behaviors from Human Motions and Videos
This study delves into the realm of multi-modality (i.e., video and motion
modalities) human behavior understanding by leveraging the powerful
capabilities of Large Language Models (LLMs). Diverging from recent LLMs
designed for video-only or motion-only understanding, we argue that
understanding human behavior necessitates joint modeling from both videos and
motion sequences (e.g., SMPL sequences) to capture nuanced body part dynamics
and semantics effectively. In light of this, we present MotionLLM, a
straightforward yet effective framework for human motion understanding,
captioning, and reasoning. Specifically, MotionLLM adopts a unified
video-motion training strategy that leverages the complementary advantages of
existing coarse video-text data and fine-grained motion-text data to glean rich
spatial-temporal insights. Furthermore, we collect a substantial dataset,
MoVid, comprising diverse videos, motions, captions, and instructions.
Additionally, we propose the MoVid-Bench, with carefully manual annotations,
for better evaluation of human behavior understanding on video and motion.
Extensive experiments show the superiority of MotionLLM in the caption,
spatial-temporal comprehension, and reasoning ability.
MotionLLM, a unified video-motion framework, leverages Large Language Models for effective human behavior understanding using diverse datasets and evaluation benchmarks.
Large Language Models MotionLLM multi-modality video-motion training strategy MoVid MoVid-Bench human motion understanding captioning spatial-temporal comprehension reasoning ability