MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human
  Perception of User Interfaces
In an ideal design pipeline, user interface (UI) design is intertwined with
user research to validate decisions, yet studies are often resource-constrained
during early exploration. Recent advances in multimodal large language models
(MLLMs) offer a promising opportunity to act as early evaluators, helping
designers narrow options before formal testing. Unlike prior work that
emphasizes user behavior in narrow domains such as e-commerce with metrics like
clicks or conversions, we focus on subjective user evaluations across varied
interfaces. We investigate whether MLLMs can mimic human preferences when
evaluating individual UIs and comparing them. Using data from a crowdsourcing
platform, we benchmark GPT-4o, Claude, and Llama across 30 interfaces and
examine alignment with human judgments on multiple UI factors. Our results show
that MLLMs approximate human preferences on some dimensions but diverge on
others, underscoring both their potential and limitations in supplementing
early UX research.
68ef3a71486b78128f0e3571
Reuben A. Luera
68ef3a71486b78128f0e3572
Ryan Rossi
68ef3a71486b78128f0e3573
Franck Dernoncourt
claimed_verified
2025-10-15T07:07:09.832Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
68ef3a71486b78128f0e3574
Samyadeep Basu
68ef3a71486b78128f0e3575
Sungchul Kim
68ef3a71486b78128f0e3576
Subhojyoti Mukherjee
68ef3a71486b78128f0e3577
Puneet Mathur
68ef3a71486b78128f0e3578
Ruiyi Zhang
68ef3a71486b78128f0e3579
Jihyung Kil
68ef3a71486b78128f0e357a
Nedim Lipka
68ef3a71486b78128f0e357b
Seunghyun Yoon
68ef3a71486b78128f0e357c
Jiuxiang Gu
68ef3a71486b78128f0e357d
Zichao Wang
68ef3a71486b78128f0e357e
Cindy Xiong Bearfield
68ef3a71486b78128f0e357f
Branislav Kveton
MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human
  Perception of User Interfaces
In an ideal design pipeline, user interface (UI) design is intertwined with
user research to validate decisions, yet studies are often resource-constrained
during early exploration. Recent advances in multimodal large language models
(MLLMs) offer a promising opportunity to act as early evaluators, helping
designers narrow options before formal testing. Unlike prior work that
emphasizes user behavior in narrow domains such as e-commerce with metrics like
clicks or conversions, we focus on subjective user evaluations across varied
interfaces. We investigate whether MLLMs can mimic human preferences when
evaluating individual UIs and comparing them. Using data from a crowdsourcing
platform, we benchmark GPT-4o, Claude, and Llama across 30 interfaces and
examine alignment with human judgments on multiple UI factors. Our results show
that MLLMs approximate human preferences on some dimensions but diverge on
others, underscoring both their potential and limitations in supplementing
early UX research.