Beyond Correctness: Evaluating Subjective Writing Preferences Across
  Cultures
Current preference learning methods achieve high accuracy on standard
benchmarks but exhibit significant performance degradation when objective
quality signals are removed. We introduce WritingPreferenceBench, a dataset of
1,800 human-annotated preference pairs (1,200 English, 600 Chinese) across 8
creative writing genres, where responses are matched for objective correctness,
factual accuracy, and length. On this benchmark, sequence-based reward
models--the standard architecture for RLHF--achieve only 52.7% mean accuracy,
while zero-shot language model judges perform at 53.9%. In contrast, generative
reward models that produce explicit reasoning chains achieve 81.8% accuracy. We
observe high within-model variance across genres: individual models range from
18.2% to 81.8% accuracy across different writing categories, with standard
deviations averaging 10.1%. This variance persists regardless of model scale,
with 27B parameter models showing no consistent improvement over 8B variants.
Our results suggest that current RLHF methods primarily learn to detect
objective errors rather than capture subjective quality preferences (e.g.,
creativity, stylistic flair, and emotional resonance), and that successful
preference modeling may require intermediate reasoning representations rather
than direct classification.
68f1b22d6e0bef323a68fdf4
Shuangshuang Ying
68f1b22d6e0bef323a68fdf5
Yunwen Li
68f1b22d6e0bef323a68fdf6
Xingwei Qu
68f1b22d6e0bef323a68fdf7
Xin Li
claimed_verified
2025-10-18T04:49:24.978Z
64a948d9723beceb2f13f5eb
https://cdn-avatars.huggingface.co/v1/production/uploads/64a948d9723beceb2f13f5eb/UQMO7LczJpabtbcoNt1zQ.jpeg
XinLi
user
XINLI1997
68f1b22d6e0bef323a68fdf8
Sheng Jin
68f1b22d6e0bef323a68fdf9
Minghao Liu
claimed_verified
2025-10-24T16:14:29.874Z
6417d9ea8f689506e7148417
https://cdn-avatars.huggingface.co/v1/production/uploads/6417d9ea8f689506e7148417/bAYcruWNw4WvmuQcGgcwC.jpeg
minghao
user
Liam-Liu
68f1b22d6e0bef323a68fdfa
Zhoufutu Wen
68f1b22d6e0bef323a68fdfb
Xeron Du
68f1b22d6e0bef323a68fdfc
Tianyu Zheng
68f1b22d6e0bef323a68fdfd
Yichi Zhang
68f1b22d6e0bef323a68fdfe
Letian Ni
68f1b22d6e0bef323a68fdff
Yuyang Cheng
68f1b22d6e0bef323a68fe00
Qiguang Chen
claimed_verified
2025-10-24T16:14:32.073Z
636f526a6cd69d9a36ff2b53
/avatars/8f2271a193fcac609d9be270552b5afa.svg
Qiguang Chen
user
LightChen2333
68f1b22d6e0bef323a68fe01
Jingzhe Ding
68f1b22d6e0bef323a68fe02
Shengda Long
68f1b22d6e0bef323a68fe03
Wangchunshu Zhou
68f1b22d6e0bef323a68fe04
Jiazhan Feng
68f1b22d6e0bef323a68fe05
Wanjun Zhong
68f1b22d6e0bef323a68fe06
Libo Qin
68f1b22d6e0bef323a68fe07
Ge Zhang
68f1b22d6e0bef323a68fe08
Wenhao Huang
68f1b22d6e0bef323a68fe09
Wanxiang Che
68f1b22d6e0bef323a68fe0a
Chenghua Lin
Beyond Correctness: Evaluating Subjective Writing Preferences Across
  Cultures
Current preference learning methods achieve high accuracy on standard
benchmarks but exhibit significant performance degradation when objective
quality signals are removed. We introduce WritingPreferenceBench, a dataset of
1,800 human-annotated preference pairs (1,200 English, 600 Chinese) across 8
creative writing genres, where responses are matched for objective correctness,
factual accuracy, and length. On this benchmark, sequence-based reward
models--the standard architecture for RLHF--achieve only 52.7% mean accuracy,
while zero-shot language model judges perform at 53.9%. In contrast, generative
reward models that produce explicit reasoning chains achieve 81.8% accuracy. We
observe high within-model variance across genres: individual models range from
18.2% to 81.8% accuracy across different writing categories, with standard
deviations averaging 10.1%. This variance persists regardless of model scale,
with 27B parameter models showing no consistent improvement over 8B variants.
Our results suggest that current RLHF methods primarily learn to detect
objective errors rather than capture subjective quality preferences (e.g.,
creativity, stylistic flair, and emotional resonance), and that successful
preference modeling may require intermediate reasoning representations rather
than direct classification.
Generative reward models with explicit reasoning chains outperform sequence-based reward models and zero-shot language models in preference learning for creative writing, indicating the need for intermediate reasoning in capturing subjective quality.
sequence-based reward models zero-shot language models generative reward models explicit reasoning chains RLHF preference learning creative writing genres objective correctness factual accuracy subjective quality preferences creativity stylistic flair emotional resonance