SAVANT: Semantic Analysis with Vision-Augmented Anomaly deTection
Autonomous driving systems remain critically vulnerable to the long-tail of
rare, out-of-distribution scenarios with semantic anomalies. While Vision
Language Models (VLMs) offer promising reasoning capabilities, naive prompting
approaches yield unreliable performance and depend on expensive proprietary
models, limiting practical deployment. We introduce SAVANT (Semantic Analysis
with Vision-Augmented Anomaly deTection), a structured reasoning framework that
achieves high accuracy and recall in detecting anomalous driving scenarios from
input images through layered scene analysis and a two-phase pipeline:
structured scene description extraction followed by multi-modal evaluation. Our
approach transforms VLM reasoning from ad-hoc prompting to systematic analysis
across four semantic layers: Street, Infrastructure, Movable Objects, and
Environment. SAVANT achieves 89.6% recall and 88.0% accuracy on real-world
driving scenarios, significantly outperforming unstructured baselines. More
importantly, we demonstrate that our structured framework enables a fine-tuned
7B parameter open-source model (Qwen2.5VL) to achieve 90.8% recall and 93.8%
accuracy - surpassing all models evaluated while enabling local deployment at
near-zero cost. By automatically labeling over 9,640 real-world images with
high accuracy, SAVANT addresses the critical data scarcity problem in anomaly
detection and provides a practical path toward reliable, accessible semantic
monitoring for autonomous systems.
68fa831af158a71c5a2f570a
Roberto Brusnicki
68fa831af158a71c5a2f570b
David Pop
68fa831af158a71c5a2f570c
Yuan Gao
claimed_verified
2025-10-24T01:06:21.063Z
672288dd7055eec76d377268
https://cdn-avatars.huggingface.co/v1/production/uploads/672288dd7055eec76d377268/bBQ51VwKZJIkHkNffuEuJ.jpeg
Yuan Gao
user
Yuan-avs
68fa831af158a71c5a2f570d
Mattia Piccinini
68fa831af158a71c5a2f570e
Johannes Betz
SAVANT: Semantic Analysis with Vision-Augmented Anomaly deTection
Autonomous driving systems remain critically vulnerable to the long-tail of
rare, out-of-distribution scenarios with semantic anomalies. While Vision
Language Models (VLMs) offer promising reasoning capabilities, naive prompting
approaches yield unreliable performance and depend on expensive proprietary
models, limiting practical deployment. We introduce SAVANT (Semantic Analysis
with Vision-Augmented Anomaly deTection), a structured reasoning framework that
achieves high accuracy and recall in detecting anomalous driving scenarios from
input images through layered scene analysis and a two-phase pipeline:
structured scene description extraction followed by multi-modal evaluation. Our
approach transforms VLM reasoning from ad-hoc prompting to systematic analysis
across four semantic layers: Street, Infrastructure, Movable Objects, and
Environment. SAVANT achieves 89.6% recall and 88.0% accuracy on real-world
driving scenarios, significantly outperforming unstructured baselines. More
importantly, we demonstrate that our structured framework enables a fine-tuned
7B parameter open-source model (Qwen2.5VL) to achieve 90.8% recall and 93.8%
accuracy - surpassing all models evaluated while enabling local deployment at
near-zero cost. By automatically labeling over 9,640 real-world images with
high accuracy, SAVANT addresses the critical data scarcity problem in anomaly
detection and provides a practical path toward reliable, accessible semantic
monitoring for autonomous systems.
SAVANT, a structured reasoning framework using VLMs, achieves high accuracy and recall in detecting anomalous driving scenarios through layered scene analysis and multi-modal evaluation.
Vision Language Models structured reasoning framework layered scene analysis multi-modal evaluation semantic layers Street Infrastructure Movable Objects Environment anomaly detection data scarcity semantic monitoring