Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos
Weifeng Lin, Xinyu Wei, Ruichuan An, Tianhe Ren, Tingwei Chen, Renrui Zhang, Ziyu Guo, Wentao Zhang, Lei Zhang, Hongsheng Li
While powerful vision foundation models like Segment Anything Model 2 (SAM 2) excel at object segmentation in images and videos, how to achieve deep semantic understanding of these regions remains a critical challenge.  To address this, we introduce Perceive Anything Model (PAM), an end-to-end vision-language model that efficiently integrates SAM 2 and Large Language Models (LLMs), enabling the simultaneous segmentation of objects while generating diverse semantic outputs for each region in both images and videos.  Specifically, PAM introduces a Semantic Perceiver that acts as a crucial bridge. This component efficiently utilizes rich intermediate features from the SAM 2 backbone, thereby incorporating general vision, localization, and semantic priors into the visual tokens, which are subsequently fed into LLMs for understanding. To ensure PAM's robustness in understanding multi-dimensional semantic granularity, we develop a dedicated data augmentation and refinement pipeline, which yields 1.8M high-quality image data and 0.6M video data. Experimental results demonstrate that, even with a lightweight 1.5/3B LLM as the semantic decoder, PAM achieves strong performance across a diverse range of tasks, including category prediction, brief and detailed regional captioning, video captioning, and streaming region captioning. Furthermore, PAM exhibits significant inference efficiency, running 1.2$-$2.4$\times$ faster while consuming less GPU memory compared to prior approaches, marking a significant advancement for real-world applications.