Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form
  Generation
Evaluating open-ended long-form generation is challenging because it is hard
to define what clearly separates good from bad outputs. Existing methods often
miss key aspects like coherence, style, or relevance, or are biased by
pretraining data, making open-ended long-form evaluation an underexplored
problem. To address this gap, we propose PrefBERT, a scoring model for
evaluating open-ended long-form generation in GRPO and guiding its training
with distinct rewards for good and bad outputs. Trained on two response
evaluation datasets with diverse long-form styles and Likert-rated quality,
PrefBERT effectively supports GRPO by offering better semantic reward feedback
than traditional metrics ROUGE-L and BERTScore do. Through comprehensive
evaluations, including LLM-as-a-judge, human ratings, and qualitative analysis,
we show that PrefBERT, trained on multi-sentence and paragraph-length
responses, remains reliable across varied long passages and aligns well with
the verifiable rewards GRPO needs. Human evaluations confirm that using
PrefBERT as the reward signal to train policy models yields responses better
aligned with human preferences than those trained with traditional metrics. Our
code is available at https://github.com/zli12321/long_form_rl.
68536bf399bf39f9665c7958
Zongxia Li
68536bf399bf39f9665c7959
Yapei Chang
68536bf399bf39f9665c795a
Yuhang Zhou
68536bf399bf39f9665c795b
Xiyang Wu
68536bf399bf39f9665c795c
Zichao Liang
68536bf399bf39f9665c795d
Yoo Yeon Sung
68536bf399bf39f9665c795e
Jordan Lee Boyd-Graber
Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form
  Generation
Evaluating open-ended long-form generation is challenging because it is hard
to define what clearly separates good from bad outputs. Existing methods often
miss key aspects like coherence, style, or relevance, or are biased by
pretraining data, making open-ended long-form evaluation an underexplored
problem. To address this gap, we propose PrefBERT, a scoring model for
evaluating open-ended long-form generation in GRPO and guiding its training
with distinct rewards for good and bad outputs. Trained on two response
evaluation datasets with diverse long-form styles and Likert-rated quality,
PrefBERT effectively supports GRPO by offering better semantic reward feedback
than traditional metrics ROUGE-L and BERTScore do. Through comprehensive
evaluations, including LLM-as-a-judge, human ratings, and qualitative analysis,
we show that PrefBERT, trained on multi-sentence and paragraph-length
responses, remains reliable across varied long passages and aligns well with
the verifiable rewards GRPO needs. Human evaluations confirm that using
PrefBERT as the reward signal to train policy models yields responses better
aligned with human preferences than those trained with traditional metrics. Our
code is available at https://github.com/zli12321/long_form_rl.
PrefBERT, a scoring model, improves open-ended long-form generation by providing better semantic reward feedback than traditional metrics.
PrefBERT GRPO multi-sentence responses paragraph-length responses Likert-rated quality LLM-as-a-judge human ratings qualitative analysis verifiable rewards