GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
Graphical user interface (GUI) agents built on vision-language models have
emerged as a promising approach to automate human-computer workflows. However,
they also face the inefficiency challenge as they process long sequences of
high-resolution screenshots and solving long-horizon tasks, making inference
slow, costly and memory-bound. While key-value (KV) caching can mitigate this,
storing the full cache is prohibitive for image-heavy contexts. Existing
cache-compression methods are sub-optimal as they do not account for the
spatial and temporal redundancy of GUIs. In this work, we first analyze
attention patterns in GUI agent workloads and find that, unlike in natural
images, attention sparsity is uniformly high across all transformer layers.
This insight motivates a simple uniform budget allocation strategy, which we
show empirically outperforms more complex layer-varying schemes. Building on
this, we introduce GUI-KV, a plug-and-play KV cache compression method for GUI
agents that requires no retraining. GUI-KV combines two novel techniques: (i)
spatial saliency guidance, which augments attention scores with the L2 norm of
hidden states to better preserve semantically important visual tokens, and (ii)
temporal redundancy scoring, which projects previous frames' keys onto the
current frame's key subspace to preferentially prune redundant history. Across
standard GUI agent benchmarks and models, GUI-KV outperforms competitive KV
compression baselines, closely matching full-cache accuracy at modest budgets.
Notably, in a 5-screenshot setting on the AgentNetBench benchmark, GUI-KV
reduces decoding FLOPs by 38.9% while increasing step accuracy by 4.1% over the
full-cache baseline. These results demonstrate that exploiting GUI-specific
redundancies enables efficient and reliable agent performance.
68ddddfc6024653e8a3ed0a2
Kung-Hsiang Huang
68ddddfc6024653e8a3ed0a3
Haoyi Qiu
68ddddfc6024653e8a3ed0a4
Yutong Dai
68ddddfc6024653e8a3ed0a5
Caiming Xiong
68ddddfc6024653e8a3ed0a6
Chien-Sheng Wu
GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
Graphical user interface (GUI) agents built on vision-language models have
emerged as a promising approach to automate human-computer workflows. However,
they also face the inefficiency challenge as they process long sequences of
high-resolution screenshots and solving long-horizon tasks, making inference
slow, costly and memory-bound. While key-value (KV) caching can mitigate this,
storing the full cache is prohibitive for image-heavy contexts. Existing
cache-compression methods are sub-optimal as they do not account for the
spatial and temporal redundancy of GUIs. In this work, we first analyze
attention patterns in GUI agent workloads and find that, unlike in natural
images, attention sparsity is uniformly high across all transformer layers.
This insight motivates a simple uniform budget allocation strategy, which we
show empirically outperforms more complex layer-varying schemes. Building on
this, we introduce GUI-KV, a plug-and-play KV cache compression method for GUI
agents that requires no retraining. GUI-KV combines two novel techniques: (i)
spatial saliency guidance, which augments attention scores with the L2 norm of
hidden states to better preserve semantically important visual tokens, and (ii)
temporal redundancy scoring, which projects previous frames' keys onto the
current frame's key subspace to preferentially prune redundant history. Across
standard GUI agent benchmarks and models, GUI-KV outperforms competitive KV
compression baselines, closely matching full-cache accuracy at modest budgets.
Notably, in a 5-screenshot setting on the AgentNetBench benchmark, GUI-KV
reduces decoding FLOPs by 38.9% while increasing step accuracy by 4.1% over the
full-cache baseline. These results demonstrate that exploiting GUI-specific
redundancies enables efficient and reliable agent performance.
GUI-KV, a KV cache compression method for GUI agents, improves efficiency by exploiting spatial and temporal redundancies, reducing computational cost while maintaining accuracy.
vision-language models GUI agents attention patterns transformer layers attention sparsity KV cache compression spatial saliency guidance temporal redundancy scoring AgentNetBench benchmark decoding FLOPs step accuracy