NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and
  Benchmarking
Benchmarking vision-based driving policies is challenging. On one hand,
open-loop evaluation with real data is easy, but these results do not reflect
closed-loop performance. On the other, closed-loop evaluation is possible in
simulation, but is hard to scale due to its significant computational demands.
Further, the simulators available today exhibit a large domain gap to real
data. This has resulted in an inability to draw clear conclusions from the
rapidly growing body of research on end-to-end autonomous driving. In this
paper, we present NAVSIM, a middle ground between these evaluation paradigms,
where we use large datasets in combination with a non-reactive simulator to
enable large-scale real-world benchmarking. Specifically, we gather
simulation-based metrics, such as progress and time to collision, by unrolling
bird's eye view abstractions of the test scenes for a short simulation horizon.
Our simulation is non-reactive, i.e., the evaluated policy and environment do
not influence each other. As we demonstrate empirically, this decoupling allows
open-loop metric computation while being better aligned with closed-loop
evaluations than traditional displacement errors. NAVSIM enabled a new
competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting
in several new insights. On a large set of challenging scenarios, we observe
that simple methods with moderate compute requirements such as TransFuser can
match recent large-scale end-to-end driving architectures such as UniAD. Our
modular framework can potentially be extended with new datasets, data curation
strategies, and metrics, and will be continually maintained to host future
challenges. Our code is available at
https://github.com/autonomousvision/navsim.
66790d52ec54ee1558b6ecf2
Daniel Dauner
admin_assigned
2024-06-24T14:03:11.000Z
633f377dc11d723b1809092c
https://cdn-avatars.huggingface.co/v1/production/uploads/633f377dc11d723b1809092c/Vvq6Zslw5DDYQmNTwJWfV.png
Daniel Dauner
user
DanielDauner
66790d52ec54ee1558b6ecf3
Marcel Hallgarten
claimed_verified
2024-10-30T09:56:19.360Z
65290083147d9119a64171ab
https://cdn-avatars.huggingface.co/v1/production/uploads/65290083147d9119a64171ab/k8q6tKMOZr-sIDyUjdtyp.png
Marcel
user
mh0797
66790d52ec54ee1558b6ecf4
Tianyu Li
66790d52ec54ee1558b6ecf5
Xinshuo Weng
66790d52ec54ee1558b6ecf6
Zhiyu Huang
66790d52ec54ee1558b6ecf7
Zetong Yang
admin_assigned
2024-06-24T14:04:09.245Z
64312d291e22a07ceb87f29e
/avatars/a0caa037d30ed52a177d94a622544727.svg
Zetong Yang
user
tomztyang
66790d52ec54ee1558b6ecf8
Hongyang Li
66790d52ec54ee1558b6ecf9
Igor Gilitschenski
admin_assigned
2024-06-24T14:05:11.039Z
64893f0b9ab28735f1a1ec0b
https://cdn-avatars.huggingface.co/v1/production/uploads/64893f0b9ab28735f1a1ec0b/2lPBjV16l2qLEWOKU2zwR.jpeg
Igor Gilitschenski
user
igilitschenski
66790d52ec54ee1558b6ecfa
Boris Ivanovic
66790d52ec54ee1558b6ecfb
Marco Pavone
66790d52ec54ee1558b6ecfc
Andreas Geiger
admin_assigned
2024-06-24T14:05:33.132Z
620cae049086f3c07f01e3d5
https://cdn-avatars.huggingface.co/v1/production/uploads/1644998139538-noauth.jpeg
Andreas Geiger
user
andreas-geiger
66790d52ec54ee1558b6ecfd
Kashyap Chitta
claimed_verified
2024-06-24T07:27:17.429Z
6489d7aada6bb0138f52027b
https://cdn-avatars.huggingface.co/v1/production/uploads/6489d7aada6bb0138f52027b/py4YEgVs8Y-QWzCmpa6Xx.jpeg
Kashyap Chitta
user
kashyap7x
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and
  Benchmarking
Benchmarking vision-based driving policies is challenging. On one hand,
open-loop evaluation with real data is easy, but these results do not reflect
closed-loop performance. On the other, closed-loop evaluation is possible in
simulation, but is hard to scale due to its significant computational demands.
Further, the simulators available today exhibit a large domain gap to real
data. This has resulted in an inability to draw clear conclusions from the
rapidly growing body of research on end-to-end autonomous driving. In this
paper, we present NAVSIM, a middle ground between these evaluation paradigms,
where we use large datasets in combination with a non-reactive simulator to
enable large-scale real-world benchmarking. Specifically, we gather
simulation-based metrics, such as progress and time to collision, by unrolling
bird's eye view abstractions of the test scenes for a short simulation horizon.
Our simulation is non-reactive, i.e., the evaluated policy and environment do
not influence each other. As we demonstrate empirically, this decoupling allows
open-loop metric computation while being better aligned with closed-loop
evaluations than traditional displacement errors. NAVSIM enabled a new
competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting
in several new insights. On a large set of challenging scenarios, we observe
that simple methods with moderate compute requirements such as TransFuser can
match recent large-scale end-to-end driving architectures such as UniAD. Our
modular framework can potentially be extended with new datasets, data curation
strategies, and metrics, and will be continually maintained to host future
challenges. Our code is available at
https://github.com/autonomousvision/navsim.
NAVSIM combines large datasets with a non-reactive simulator to benchmark vision-based driving policies, enabling scalable and insightful evaluation that bridges open-loop and closed-loop paradigms.
open-loop evaluation closed-loop evaluation non-reactive simulator 鸟's eye view simulation horizon TransFuser UniAD benchmarking evaluation framework dataset curation metrics