MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior
  Understanding
Humans are integral components of the transportation ecosystem, and
understanding their behaviors is crucial to facilitating the development of
safe driving systems. Although recent progress has explored various aspects of
human behaviorx2014such as motion, trajectories, and
intentionx2014a comprehensive benchmark for evaluating human
behavior understanding in autonomous driving remains unavailable. In this work,
we propose MMHU, a large-scale benchmark for human behavior analysis
featuring rich annotations, such as human motion and trajectories, text
description for human motions, human intention, and critical behavior labels
relevant to driving safety. Our dataset encompasses 57k human motion clips and
1.73M frames gathered from diverse sources, including established driving
datasets such as Waymo, in-the-wild videos from YouTube, and self-collected
data. A human-in-the-loop annotation pipeline is developed to generate rich
behavior captions. We provide a thorough dataset analysis and benchmark
multiple tasksx2014ranging from motion prediction to motion
generation and human behavior question answeringx2014thereby
offering a broad evaluation suite. Project page :
https://MMHU-Benchmark.github.io.
68788789001546c83aa4f9e4
Renjie Li
claimed_verified
2025-07-17T14:59:16.875Z
686fe2365998cd8af12fc8ba
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/GQbxskSw-98mdjbM3szaC.png
Renjie Li
user
renjie-li
68788789001546c83aa4f9e5
Ruijie Ye
claimed_verified
2025-07-17T08:17:53.937Z
6825fc0e58cf56d164cb339d
/avatars/4ade4a5bbb0a805a92a83bfb233f805f.svg
Ruijie Ye
user
jerryye0110
68788789001546c83aa4f9e6
Mingyang Wu
claimed_verified
2025-07-17T18:43:17.791Z
6736d289c8a9bf8f86936201
/avatars/ca56298f9db458ba65c469b1baabda2c.svg
MingyangWu
user
mingyang-wu
68788789001546c83aa4f9e7
Hao Frank Yang
68788789001546c83aa4f9e8
Zhiwen Fan
claimed_verified
2025-07-17T08:23:18.102Z
63b354bb7091e602f1a0e2e8
/avatars/a388d93c0af2f57eadb6fa60d6789041.svg
wayne
user
waynefan
68788789001546c83aa4f9e9
Hezhen Hu
claimed_verified
2025-07-21T08:30:13.482Z
673bf280c75f9187d3637eab
/avatars/cd1a7d0ffb2635694193c2ef4a71ff18.svg
AlexHu
user
alexhu123
68788789001546c83aa4f9ea
Zhengzhong Tu
claimed_verified
2025-07-17T08:23:16.047Z
62548d5fef3debb2ddf91217
/avatars/14975b45568f9c399c92c3986b6ce83e.svg
Zhengzhong Tu
user
vztu
MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior
  Understanding
Humans are integral components of the transportation ecosystem, and
understanding their behaviors is crucial to facilitating the development of
safe driving systems. Although recent progress has explored various aspects of
human behaviorx2014such as motion, trajectories, and
intentionx2014a comprehensive benchmark for evaluating human
behavior understanding in autonomous driving remains unavailable. In this work,
we propose MMHU, a large-scale benchmark for human behavior analysis
featuring rich annotations, such as human motion and trajectories, text
description for human motions, human intention, and critical behavior labels
relevant to driving safety. Our dataset encompasses 57k human motion clips and
1.73M frames gathered from diverse sources, including established driving
datasets such as Waymo, in-the-wild videos from YouTube, and self-collected
data. A human-in-the-loop annotation pipeline is developed to generate rich
behavior captions. We provide a thorough dataset analysis and benchmark
multiple tasksx2014ranging from motion prediction to motion
generation and human behavior question answeringx2014thereby
offering a broad evaluation suite. Project page :
https://MMHU-Benchmark.github.io.
A large-scale benchmark, MMHU, is proposed for human behavior analysis in autonomous driving, featuring rich annotations and diverse data sources, and benchmarking various tasks including motion prediction and behavior question answering.
human behavior analysis motion prediction motion generation human behavior question answering human-in-the-loop annotation Waymo YouTube self-collected data