Vibe Checker: Aligning Code Evaluation with Human Preference
Large Language Models (LLMs) have catalyzed vibe coding, where users leverage
LLMs to generate and iteratively refine code through natural language
interactions until it passes their vibe check. Vibe check is tied to real-world
human preference and goes beyond functionality: the solution should feel right,
read cleanly, preserve intent, and remain correct. However, current code
evaluation remains anchored to pass@k and captures only functional correctness,
overlooking the non-functional instructions that users routinely apply. In this
paper, we hypothesize that instruction following is the missing piece
underlying vibe check that represents human preference in coding besides
functional correctness. To quantify models' code instruction following
capabilities with measurable signals, we present VeriCode, a taxonomy of 30
verifiable code instructions together with corresponding deterministic
verifiers. We use the taxonomy to augment established evaluation suites,
resulting in Vibe Checker, a testbed to assess both code instruction following
and functional correctness. Upon evaluating 31 leading LLMs, we show that even
the strongest models struggle to comply with multiple instructions and exhibit
clear functional regression. Most importantly, a composite score of functional
correctness and instruction following correlates the best with human
preference, with the latter emerging as the primary differentiator on
real-world programming tasks. Our work identifies core factors of the vibe
check, providing a concrete path for benchmarking and developing models that
better align with user preferences in coding.
68e714517ae125f9582e6903
Ming Zhong
claimed_verified
2025-10-09T06:50:34.588Z
61d53df2062444ea769d3b79
/avatars/fa771202368b6b2626a8fdf1c4369239.svg
Ming Zhong
user
MingZhong
68e714517ae125f9582e6904
Xiang Zhou
68e714517ae125f9582e6905
Ting-Yun Chang
68e714517ae125f9582e6906
Qingze Wang
claimed_verified
2025-10-10T09:32:31.234Z
68e7d4e151f01cfb9dde07fb
/avatars/2297371204cc33cc7afd7cb785ed9ab9.svg
Qingze Wang
user
wqzscgy
68e714517ae125f9582e6907
Nan Xu
68e714517ae125f9582e6908
Xiance Si
68e714517ae125f9582e6909
Dan Garrette
68e714517ae125f9582e690a
Shyam Upadhyay
68e714517ae125f9582e690b
Jeremiah Liu
68e714517ae125f9582e690c
Jiawei Han
68e714517ae125f9582e690d
Benoit Schillings
68e714517ae125f9582e690e
Jiao Sun
Vibe Checker: Aligning Code Evaluation with Human Preference
Large Language Models (LLMs) have catalyzed vibe coding, where users leverage
LLMs to generate and iteratively refine code through natural language
interactions until it passes their vibe check. Vibe check is tied to real-world
human preference and goes beyond functionality: the solution should feel right,
read cleanly, preserve intent, and remain correct. However, current code
evaluation remains anchored to pass@k and captures only functional correctness,
overlooking the non-functional instructions that users routinely apply. In this
paper, we hypothesize that instruction following is the missing piece
underlying vibe check that represents human preference in coding besides
functional correctness. To quantify models' code instruction following
capabilities with measurable signals, we present VeriCode, a taxonomy of 30
verifiable code instructions together with corresponding deterministic
verifiers. We use the taxonomy to augment established evaluation suites,
resulting in Vibe Checker, a testbed to assess both code instruction following
and functional correctness. Upon evaluating 31 leading LLMs, we show that even
the strongest models struggle to comply with multiple instructions and exhibit
clear functional regression. Most importantly, a composite score of functional
correctness and instruction following correlates the best with human
preference, with the latter emerging as the primary differentiator on
real-world programming tasks. Our work identifies core factors of the vibe
check, providing a concrete path for benchmarking and developing models that
better align with user preferences in coding.
Vibe Checker evaluates LLMs by combining functional correctness and instruction following to better align with human coding preferences.
Large Language Models LLMs vibe coding Vibe Checker VeriCode verifiable code instructions deterministic verifiers instruction following functional correctness human preference coding preferences