HelpSteer2: Open-source dataset for training top-performing reward
  models
High-quality preference datasets are essential for training reward models
that can effectively guide large language models (LLMs) in generating
high-quality responses aligned with human preferences. As LLMs become stronger
and better aligned, permissively licensed preference datasets, such as Open
Assistant, HH-RLHF, and HelpSteer need to be updated to remain effective for
reward modeling. Methods that distil preference data from proprietary LLMs such
as GPT-4 have restrictions on commercial usage imposed by model providers. To
improve upon both generated responses and attribute labeling quality, we
release HelpSteer2, a permissively licensed preference dataset (CC-BY-4.0).
Using a powerful internal base model trained on HelpSteer2, we are able to
achieve the SOTA score (92.0%) on Reward-Bench's primary dataset, outperforming
currently listed open and proprietary models, as of June 12th, 2024. Notably,
HelpSteer2 consists of only ten thousand response pairs, an order of magnitude
fewer than existing preference datasets (e.g., HH-RLHF), which makes it highly
efficient for training reward models. Our extensive experiments demonstrate
that reward models trained with HelpSteer2 are effective in aligning LLMs. In
particular, we propose SteerLM 2.0, a model alignment approach that can
effectively make use of the rich multi-attribute score predicted by our reward
models. HelpSteer2 is available at
https://huggingface.co/datasets/nvidia/HelpSteer2 and code is available at
https://github.com/NVIDIA/NeMo-Aligner
666ba265fba6c458fcee3cf6
Zhilin Wang
admin_assigned
2024-06-14T12:09:11.245Z
635865c8aff68f72ac039a4e
/avatars/3fa9bee2e9e211723fcf44bddc6110c2.svg
Zhilin Wang
user
zhilinw
666ba265fba6c458fcee3cf7
Yi Dong
666ba265fba6c458fcee3cf8
Olivier Delalleau
admin_assigned
2024-06-14T12:09:45.540Z
6556379e10428134ff235afd
/avatars/ec569729870d7392e806e59a02f37d0c.svg
Olivier Delalleau
user
odelalleau
666ba265fba6c458fcee3cf9
Jiaqi Zeng
claimed_verified
2024-06-15T07:46:01.951Z
6511cb8d254ae1f96174a330
/avatars/525c88f6c146fa83e0954d8ed97ee99e.svg
jiaqiz
user
jiaqiz
666ba265fba6c458fcee3cfa
Gerald Shen
admin_assigned
2024-06-14T12:10:12.778Z
6635187e27cf4d1a75f27a40
/avatars/e36f11c962f52e783b4677ca5460c41e.svg
Gerald Shen
user
gshennvm
666ba265fba6c458fcee3cfb
Daniel Egert
admin_assigned
2024-06-14T12:10:24.243Z
63d2f2bc2727d7888cb9f3cd
/avatars/f26eb927e803f1ea05d8ae9abfdd2846.svg
Daniel Egert
user
trias702
666ba265fba6c458fcee3cfc
Jimmy J. Zhang
666ba265fba6c458fcee3cfd
Makesh Narsimhan Sreedhar
admin_assigned
2024-06-14T12:11:05.118Z
6643782e17bbed56d0b9932b
/avatars/9f20fcf2b8bfc75c56c510e7bda4dbd7.svg
Makesh Sreedhar
user
makeshn
666ba265fba6c458fcee3cfe
Oleksii Kuchaiev
admin_assigned
2024-06-14T12:11:10.896Z
622937a4acd5bef90e55c49d
https://cdn-avatars.huggingface.co/v1/production/uploads/1649475932211-622937a4acd5bef90e55c49d.jpeg
Oleksii Kuchaiev
user
okuchaiev
HelpSteer2: Open-source dataset for training top-performing reward
  models
High-quality preference datasets are essential for training reward models
that can effectively guide large language models (LLMs) in generating
high-quality responses aligned with human preferences. As LLMs become stronger
and better aligned, permissively licensed preference datasets, such as Open
Assistant, HH-RLHF, and HelpSteer need to be updated to remain effective for
reward modeling. Methods that distil preference data from proprietary LLMs such
as GPT-4 have restrictions on commercial usage imposed by model providers. To
improve upon both generated responses and attribute labeling quality, we
release HelpSteer2, a permissively licensed preference dataset (CC-BY-4.0).
Using a powerful internal base model trained on HelpSteer2, we are able to
achieve the SOTA score (92.0%) on Reward-Bench's primary dataset, outperforming
currently listed open and proprietary models, as of June 12th, 2024. Notably,
HelpSteer2 consists of only ten thousand response pairs, an order of magnitude
fewer than existing preference datasets (e.g., HH-RLHF), which makes it highly
efficient for training reward models. Our extensive experiments demonstrate
that reward models trained with HelpSteer2 are effective in aligning LLMs. In
particular, we propose SteerLM 2.0, a model alignment approach that can
effectively make use of the rich multi-attribute score predicted by our reward
models. HelpSteer2 is available at
https://huggingface.co/datasets/nvidia/HelpSteer2 and code is available at
https://github.com/NVIDIA/NeMo-Aligner
NVIDIA's HelpSteer2 dataset and SteerLM 2.0 model improve LLM response quality and alignment using permissively licensed preference data.
preference datasets reward models large language models (LLMs) Open Assistant HH-RLHF HelpSteer HuRLHF HelpSteer2 SOTA score Reward-Bench SteerLM model alignment multi-attribute score