ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to
  Answer and Judge
Evaluating progress in large language models (LLMs) is often constrained by
the challenge of verifying responses, limiting assessments to tasks like
mathematics, programming, and short-form question-answering. However, many
real-world applications require evaluating LLMs in processing professional
documents, synthesizing information, and generating comprehensive reports in
response to user queries. We introduce ProfBench: a set of over 7000
response-criterion pairs as evaluated by human-experts with professional
knowledge across Physics PhD, Chemistry PhD, Finance MBA and Consulting MBA. We
build robust and affordable LLM-Judges to evaluate ProfBench rubrics, by
mitigating self-enhancement bias and reducing the cost of evaluation by 2-3
orders of magnitude, to make it fair and accessible to the broader community.
Our findings reveal that ProfBench poses significant challenges even for
state-of-the-art LLMs, with top-performing models like GPT-5-high achieving
only 65.9\% overall performance. Furthermore, we identify notable performance
disparities between proprietary and open-weight models and provide insights
into the role that extended thinking plays in addressing complex,
professional-domain tasks. Data:
https://huggingface.co/datasets/nvidia/ProfBench and Code:
https://github.com/NVlabs/ProfBench
68f972efb9b2e4ae04673706
Zhilin Wang
68f972efb9b2e4ae04673707
Jaehun Jung
68f972efb9b2e4ae04673708
Ximing Lu
68f972efb9b2e4ae04673709
Shizhe Diao
68f972efb9b2e4ae0467370a
Ellie Evans
68f972efb9b2e4ae0467370b
Jiaqi Zeng
68f972efb9b2e4ae0467370c
Pavlo Molchanov
68f972efb9b2e4ae0467370d
Yejin Choi
68f972efb9b2e4ae0467370e
Jan Kautz
68f972efb9b2e4ae0467370f
Yi Dong
ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to
  Answer and Judge
Evaluating progress in large language models (LLMs) is often constrained by
the challenge of verifying responses, limiting assessments to tasks like
mathematics, programming, and short-form question-answering. However, many
real-world applications require evaluating LLMs in processing professional
documents, synthesizing information, and generating comprehensive reports in
response to user queries. We introduce ProfBench: a set of over 7000
response-criterion pairs as evaluated by human-experts with professional
knowledge across Physics PhD, Chemistry PhD, Finance MBA and Consulting MBA. We
build robust and affordable LLM-Judges to evaluate ProfBench rubrics, by
mitigating self-enhancement bias and reducing the cost of evaluation by 2-3
orders of magnitude, to make it fair and accessible to the broader community.
Our findings reveal that ProfBench poses significant challenges even for
state-of-the-art LLMs, with top-performing models like GPT-5-high achieving
only 65.9\% overall performance. Furthermore, we identify notable performance
disparities between proprietary and open-weight models and provide insights
into the role that extended thinking plays in addressing complex,
professional-domain tasks. Data:
https://huggingface.co/datasets/nvidia/ProfBench and Code:
https://github.com/NVlabs/ProfBench
ProfBench evaluates large language models in professional domains using human-expert criteria, revealing challenges and performance disparities between proprietary and open-weight models.
large language models LLMs ProfBench response-criterion pairs human-experts LLM-Judges self-enhancement bias GPT-5-high extended thinking professional-domain tasks