MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks
Sanjoy Chowdhury, Mohamed Elmoghany, Yohan Abeysinghe, Junjie Fei, Sayan Nag, Salman Khan, Mohamed Elhoseiny, Dinesh Manocha
Large multimodal models (LMMs) have shown remarkable progress in audio-visual understanding, yet they struggle with real-world scenarios that requirecomplex reasoning across extensive video collections. Existing benchmarks forvideo question answering remain limited in scope, typically involving one clipper query, which falls short of representing the challenges of large-scale, audio-visual retrieval and reasoning encountered in practical applications. To bridgethis gap, we introduce a novel task named AV-HaystacksQA, where the goalis to identify salient segments across different videos in response to a query andlink them together to generate the most informative answer. To this end, wepresent AVHaystacks, an audio-visual benchmark comprising 3100 annotated QApairs designed to assess the capabilities of LMMs in multi-video retrieval andtemporal grounding task. Additionally, we propose a model-agnostic, multi-agentframework MAGNET to address this challenge, achieving up to 89% and 65%relative improvements over baseline methods on BLEU@4 and GPT evaluationscores in QA task on our proposed AVHaystacks. To enable robust evaluation ofmulti-video retrieval and temporal grounding for optimal response generation, weintroduce two new metrics, STEM, which captures alignment errors between aground truth and a predicted step sequence and MTGS, to facilitate balanced andinterpretable evaluation of segment-level grounding performance. Our code anddataset will be released publicly