Drive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models
Recent advances in generative models have sparked exciting new possibilities
in the field of autonomous vehicles. Specifically, video generation models are
now being explored as controllable virtual testing environments.
Simultaneously, end-to-end (E2E) driving models have emerged as a streamlined
alternative to conventional modular autonomous driving systems, gaining
popularity for their simplicity and scalability. However, the application of
these techniques to simulation and planning raises important questions. First,
while video generation models can generate increasingly realistic videos, can
these videos faithfully adhere to the specified conditions and be realistic
enough for E2E autonomous planner evaluation? Second, given that data is
crucial for understanding and controlling E2E planners, how can we gain deeper
insights into their biases and improve their ability to generalize to
out-of-distribution scenarios? In this work, we bridge the gap between the
driving models and generative world models (Drive&Gen) to address these
questions. We propose novel statistical measures leveraging E2E drivers to
evaluate the realism of generated videos. By exploiting the controllability of
the video generation model, we conduct targeted experiments to investigate
distribution gaps affecting E2E planner performance. Finally, we show that
synthetic data produced by the video generation model offers a cost-effective
alternative to real-world data collection. This synthetic data effectively
improves E2E model generalization beyond existing Operational Design Domains,
facilitating the expansion of autonomous vehicle services into new operational
contexts.
68e91c0095e8e6771df38cb6
Jiahao Wang
68e91c0095e8e6771df38cb7
Zhenpei Yang
68e91c0095e8e6771df38cb8
Yijing Bai
68e91c0095e8e6771df38cb9
Yingwei Li
68e91c0095e8e6771df38cba
Yuliang Zou
68e91c0095e8e6771df38cbb
Bo Sun
68e91c0095e8e6771df38cbc
Abhijit Kundu
68e91c0095e8e6771df38cbd
Jose Lezama
68e91c0095e8e6771df38cbe
Luna Yue Huang
68e91c0095e8e6771df38cbf
Zehao Zhu
68e91c0095e8e6771df38cc0
Jyh-Jing Hwang
68e91c0095e8e6771df38cc1
Dragomir Anguelov
68e91c0095e8e6771df38cc2
Mingxing Tan
68e91c0095e8e6771df38cc3
Chiyu Max Jiang
Drive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models
Recent advances in generative models have sparked exciting new possibilities
in the field of autonomous vehicles. Specifically, video generation models are
now being explored as controllable virtual testing environments.
Simultaneously, end-to-end (E2E) driving models have emerged as a streamlined
alternative to conventional modular autonomous driving systems, gaining
popularity for their simplicity and scalability. However, the application of
these techniques to simulation and planning raises important questions. First,
while video generation models can generate increasingly realistic videos, can
these videos faithfully adhere to the specified conditions and be realistic
enough for E2E autonomous planner evaluation? Second, given that data is
crucial for understanding and controlling E2E planners, how can we gain deeper
insights into their biases and improve their ability to generalize to
out-of-distribution scenarios? In this work, we bridge the gap between the
driving models and generative world models (Drive&Gen) to address these
questions. We propose novel statistical measures leveraging E2E drivers to
evaluate the realism of generated videos. By exploiting the controllability of
the video generation model, we conduct targeted experiments to investigate
distribution gaps affecting E2E planner performance. Finally, we show that
synthetic data produced by the video generation model offers a cost-effective
alternative to real-world data collection. This synthetic data effectively
improves E2E model generalization beyond existing Operational Design Domains,
facilitating the expansion of autonomous vehicle services into new operational
contexts.
A novel approach combining driving models and generative world models evaluates and enhances the realism and generalization of synthetic video data for autonomous vehicle testing and planning.
generative models autonomous vehicles video generation models end-to-end driving models statistical measures E2E planners distribution gaps synthetic data Operational Design Domains