NuRisk: A Visual Question Answering Dataset for Agent-Level Risk
  Assessment in Autonomous Driving
Understanding risk in autonomous driving requires not only perception and
prediction, but also high-level reasoning about agent behavior and context.
Current Vision Language Models (VLMs)-based methods primarily ground agents in
static images and provide qualitative judgments, lacking the spatio-temporal
reasoning needed to capture how risks evolve over time. To address this gap, we
propose NuRisk, a comprehensive Visual Question Answering (VQA) dataset
comprising 2,900 scenarios and 1.1 million agent-level samples, built on
real-world data from nuScenes and Waymo, supplemented with safety-critical
scenarios from the CommonRoad simulator. The dataset provides Bird-Eye-View
(BEV) based sequential images with quantitative, agent-level risk annotations,
enabling spatio-temporal reasoning. We benchmark well-known VLMs across
different prompting techniques and find that they fail to perform explicit
spatio-temporal reasoning, resulting in a peak accuracy of 33% at high latency.
To address these shortcomings, our fine-tuned 7B VLM agent improves accuracy to
41% and reduces latency by 75%, demonstrating explicit spatio-temporal
reasoning capabilities that proprietary models lacked. While this represents a
significant step forward, the modest accuracy underscores the profound
challenge of the task, establishing NuRisk as a critical benchmark for
advancing spatio-temporal reasoning in autonomous driving.
68dd253fe795697646c0a147
Yuan Gao
claimed_verified
2025-10-05T12:47:33.761Z
672288dd7055eec76d377268
https://cdn-avatars.huggingface.co/v1/production/uploads/672288dd7055eec76d377268/bBQ51VwKZJIkHkNffuEuJ.jpeg
Yuan Gao
user
Yuan-avs
68dd253fe795697646c0a148
Mattia Piccinini
68dd253fe795697646c0a149
Roberto Brusnicki
68dd253fe795697646c0a14a
Yuchen Zhang
68dd253fe795697646c0a14b
Johannes Betz
NuRisk: A Visual Question Answering Dataset for Agent-Level Risk
  Assessment in Autonomous Driving
Understanding risk in autonomous driving requires not only perception and
prediction, but also high-level reasoning about agent behavior and context.
Current Vision Language Models (VLMs)-based methods primarily ground agents in
static images and provide qualitative judgments, lacking the spatio-temporal
reasoning needed to capture how risks evolve over time. To address this gap, we
propose NuRisk, a comprehensive Visual Question Answering (VQA) dataset
comprising 2,900 scenarios and 1.1 million agent-level samples, built on
real-world data from nuScenes and Waymo, supplemented with safety-critical
scenarios from the CommonRoad simulator. The dataset provides Bird-Eye-View
(BEV) based sequential images with quantitative, agent-level risk annotations,
enabling spatio-temporal reasoning. We benchmark well-known VLMs across
different prompting techniques and find that they fail to perform explicit
spatio-temporal reasoning, resulting in a peak accuracy of 33% at high latency.
To address these shortcomings, our fine-tuned 7B VLM agent improves accuracy to
41% and reduces latency by 75%, demonstrating explicit spatio-temporal
reasoning capabilities that proprietary models lacked. While this represents a
significant step forward, the modest accuracy underscores the profound
challenge of the task, establishing NuRisk as a critical benchmark for
advancing spatio-temporal reasoning in autonomous driving.
NuRisk, a comprehensive VQA dataset, addresses the lack of spatio-temporal reasoning in current VLMs for autonomous driving by providing agent-level risk annotations in sequential images, improving accuracy and reducing latency.
Visual Question Answering (VQA) Vision Language Models (VLMs) spatio-temporal reasoning Bird-Eye-View (BEV) agent-level risk annotations nuScenes Waymo CommonRoad simulator fine-tuned VLM