Improving GUI Grounding with Explicit Position-to-Coordinate Mapping
GUI grounding, the task of mapping natural-language instructions to pixel
coordinates, is crucial for autonomous agents, yet remains difficult for
current VLMs. The core bottleneck is reliable patch-to-pixel mapping, which
breaks when extrapolating to high-resolution displays unseen during training.
Current approaches generate coordinates as text tokens directly from visual
features, forcing the model to infer complex position-to-pixel mappings
implicitly; as a result, accuracy degrades and failures proliferate on new
resolutions. We address this with two complementary innovations. First, RULER
tokens serve as explicit coordinate markers, letting the model reference
positions similar to gridlines on a map and adjust rather than generate
coordinates from scratch. Second, Interleaved MRoPE (I-MRoPE) improves spatial
encoding by ensuring that width and height dimensions are represented equally,
addressing the asymmetry of standard positional schemes. Experiments on
ScreenSpot, ScreenSpot-V2, and ScreenSpot-Pro show consistent gains in
grounding accuracy, with the largest improvements on high-resolution
interfaces. By providing explicit spatial guidance rather than relying on
implicit learning, our approach enables more reliable GUI automation across
diverse resolutions and platforms.
68e31e7c73e20ab577841f9f
Suyuchen Wang
claimed_verified
2025-10-07T12:28:34.718Z
62bb1e0f3ff437e49a3088e5
https://cdn-avatars.huggingface.co/v1/production/uploads/62bb1e0f3ff437e49a3088e5/MWNanci3x5g780xh-704U.png
Suyuchen Wang
user
sheryc
68e31e7c73e20ab577841fa0
Tianyu Zhang
68e31e7c73e20ab577841fa1
Ahmed Masry
68e31e7c73e20ab577841fa2
Christopher Pal
68e31e7c73e20ab577841fa3
Spandana Gella
68e31e7c73e20ab577841fa4
Bang Liu
68e31e7c73e20ab577841fa5
Perouz Taslakian
Improving GUI Grounding with Explicit Position-to-Coordinate Mapping
GUI grounding, the task of mapping natural-language instructions to pixel
coordinates, is crucial for autonomous agents, yet remains difficult for
current VLMs. The core bottleneck is reliable patch-to-pixel mapping, which
breaks when extrapolating to high-resolution displays unseen during training.
Current approaches generate coordinates as text tokens directly from visual
features, forcing the model to infer complex position-to-pixel mappings
implicitly; as a result, accuracy degrades and failures proliferate on new
resolutions. We address this with two complementary innovations. First, RULER
tokens serve as explicit coordinate markers, letting the model reference
positions similar to gridlines on a map and adjust rather than generate
coordinates from scratch. Second, Interleaved MRoPE (I-MRoPE) improves spatial
encoding by ensuring that width and height dimensions are represented equally,
addressing the asymmetry of standard positional schemes. Experiments on
ScreenSpot, ScreenSpot-V2, and ScreenSpot-Pro show consistent gains in
grounding accuracy, with the largest improvements on high-resolution
interfaces. By providing explicit spatial guidance rather than relying on
implicit learning, our approach enables more reliable GUI automation across
diverse resolutions and platforms.
Explicit coordinate markers and improved spatial encoding enhance GUI grounding accuracy across diverse resolutions and platforms.
RULER tokens Interleaved MRoPE I-MRoPE spatial encoding GUI grounding ScreenSpot ScreenSpot-V2 ScreenSpot-Pro