Aha! - Predicting What Matters Next: Online Highlight Detection Without Looking Ahead
Aiden Chang, Celso de Melo, Stephanie Lukin
Real-time understanding of continuous video streams is essential for intelligent agents operating in high-stakes environments, including autonomous vehicles, surveillance drones, and disaster response robots. Yet, most existing video understanding and highlight detection methods assume access to the entire video during inference, making them unsuitable for online or streaming scenarios. In particular, current models optimize for offline summarization, failing to support step-by-step reasoning needed for real-time decision-making. We introduce Aha!, an autoregressive highlight detection framework that predicts the relevance of each video frame against a task described in natural language. Without accessing future video frames, Aha! utilizes a multimodal language-vision model and lightweight, decoupled heads trained on a large, curated dataset of human-centric video labels. To enable scalability, we adopt a fixed-size SinkCache mechanism that achieves constant memory usage across infinite-length streams without degrading performance on standard benchmarks. This encourages the hidden representation to capture high-level task objectives, enabling effective frame-level rankings for informativeness, relevance, and uncertainty with respect to the natural language task. Aha! achieves state-of-the-art performance on highlight detection benchmarks, surpassing prior full-context and video-language models by +5.5\% on TVSum and +8.3\% on Mr. HiSum in mAP. We explore Aha!’s potential for real-world robotics applications given a task-oriented natural language input and a continuous, robot-centric video. Both experiments demonstrate Aha!'s potential effectiveness as a real-time reasoning module for downstream planning and long-horizon understanding.