Visual Multi-Agent System: Mitigating Hallucination Snowballing via
  Visual Flow
Multi-Agent System (MAS) powered by Visual Language Models (VLMs) enables
challenging tasks but suffers from a novel failure term, multi-agent visual
hallucination snowballing, where hallucinations are seeded in a single agent
and amplified by following ones due to the over-reliance on textual flow to
relay visual information. Through turn-, layer-, and token-wise attention
analyses, we provide detailed insights into the essence of hallucination
snowballing regarding the reduction of visual attention allocation. It leads us
to identify a subset of vision tokens with a unimodal attention peak in middle
layers that best preserve visual evidence but gradually diminish in deeper
agent turns, resulting in the visual hallucination snowballing in MAS. Thus, we
propose ViF, a lightweight, plug-and-play mitigation paradigm that relays
inter-agent messages with Visual Flow powered by the selected visual relay
tokens and applies attention reallocation to amplify this pattern. The
experiment results demonstrate that our method markedly reduces hallucination
snowballing, consistently improving the performance across eight benchmarks
based on four common MAS structures and ten base models. The source code will
be available at: https://github.com/YU-deep/ViF.git.
68df677ddf49fb0df1e03dab
Xinlei Yu
68df677ddf49fb0df1e03dac
Chengming Xu
68df677ddf49fb0df1e03dad
Guibin Zhang
68df677ddf49fb0df1e03dae
Yongbo He
68df677ddf49fb0df1e03daf
Zhangquan Chen
claimed_verified
2025-10-22T15:46:54.245Z
6458a99c3b81018d6b93aecb
/avatars/d90f16e8d3e49e095fea4cbd899837df.svg
jankin
user
jankin123
68df677ddf49fb0df1e03db0
Zhucun Xue
68df677ddf49fb0df1e03db1
Jiangning Zhang
68df677ddf49fb0df1e03db2
Yue Liao
68df677ddf49fb0df1e03db3
Xiaobin Hu
68df677ddf49fb0df1e03db4
Yu-Gang Jiang
68df677ddf49fb0df1e03db5
Shuicheng Yan
Visual Multi-Agent System: Mitigating Hallucination Snowballing via
  Visual Flow
Multi-Agent System (MAS) powered by Visual Language Models (VLMs) enables
challenging tasks but suffers from a novel failure term, multi-agent visual
hallucination snowballing, where hallucinations are seeded in a single agent
and amplified by following ones due to the over-reliance on textual flow to
relay visual information. Through turn-, layer-, and token-wise attention
analyses, we provide detailed insights into the essence of hallucination
snowballing regarding the reduction of visual attention allocation. It leads us
to identify a subset of vision tokens with a unimodal attention peak in middle
layers that best preserve visual evidence but gradually diminish in deeper
agent turns, resulting in the visual hallucination snowballing in MAS. Thus, we
propose ViF, a lightweight, plug-and-play mitigation paradigm that relays
inter-agent messages with Visual Flow powered by the selected visual relay
tokens and applies attention reallocation to amplify this pattern. The
experiment results demonstrate that our method markedly reduces hallucination
snowballing, consistently improving the performance across eight benchmarks
based on four common MAS structures and ten base models. The source code will
be available at: https://github.com/YU-deep/ViF.git.
ViF mitigates visual hallucination snowballing in Multi-Agent Systems by enhancing visual attention and message relay through selected visual tokens.
Multi-Agent System Visual Language Models visual hallucination snowballing turn-wise attention layer-wise attention token-wise attention visual attention allocation vision tokens unimodal attention peak Visual Flow attention reallocation