Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health
  Information
As Large Language Models (LLMs) become widely accessible, a detailed
understanding of their knowledge within specific domains becomes necessary for
successful real world use. This is particularly critical in public health,
where failure to retrieve relevant, accurate, and current information could
significantly impact UK residents. However, currently little is known about LLM
knowledge of UK Government public health information. To address this issue,
this paper introduces a new benchmark, PubHealthBench, with over 8000 questions
for evaluating LLMs' Multiple Choice Question Answering (MCQA) and free form
responses to public health queries, created via an automated pipeline. We also
release a new dataset of the extracted UK Government public health guidance
documents used as source text for PubHealthBench. Assessing 24 LLMs on
PubHealthBench we find the latest private LLMs (GPT-4.5, GPT-4.1 and o1) have a
high degree of knowledge, achieving >90% in the MCQA setup, and outperform
humans with cursory search engine use. However, in the free form setup we see
lower performance with no model scoring >75%. Therefore, whilst there are
promising signs that state of the art (SOTA) LLMs are an increasingly accurate
source of public health information, additional safeguards or tools may still
be needed when providing free form responses on public health topics.
6821af48696b63e207ae8474
Joshua Harris
claimed_verified
2025-05-12T09:03:48.631Z
64cb98c6f103036e23c69b1d
/avatars/7ee33880ad39f5335b618dc53554124a.svg
Harris
user
Joshua-Harris
6821af48696b63e207ae8475
Fan Grayson
admin_assigned
2025-05-12T13:17:30.555Z
6422f276320bd601bff05691
/avatars/7f43ca6cb485a73e3de8a0653f109735.svg
Fan Grayson
user
fangrayson
6821af48696b63e207ae8476
Felix Feldman
admin_assigned
2025-05-12T13:17:36.880Z
681205b04119e4ecc947632d
/avatars/94f18aeed6fef7764252e06244e1e7c9.svg
Felix Feldman
user
felixFeldman
6821af48696b63e207ae8477
Timothy Laurence
admin_assigned
2025-05-12T13:17:44.371Z
64b9323691876a8d16d2dae5
/avatars/ea583af4ec8e5ad18fefbec2281ae50e.svg
Timothy Laurence
user
TRLaurence
6821af48696b63e207ae8478
Toby Nonnenmacher
6821af48696b63e207ae8479
Oliver Higgins
6821af48696b63e207ae847a
Leo Loman
6821af48696b63e207ae847b
Selina Patel
6821af48696b63e207ae847c
Thomas Finnie
6821af48696b63e207ae847d
Samuel Collins
6821af48696b63e207ae847e
Michael Borowitz
Healthy LLMs? Benchmarking LLM Knowledge of UK Government Public Health
  Information
As Large Language Models (LLMs) become widely accessible, a detailed
understanding of their knowledge within specific domains becomes necessary for
successful real world use. This is particularly critical in public health,
where failure to retrieve relevant, accurate, and current information could
significantly impact UK residents. However, currently little is known about LLM
knowledge of UK Government public health information. To address this issue,
this paper introduces a new benchmark, PubHealthBench, with over 8000 questions
for evaluating LLMs' Multiple Choice Question Answering (MCQA) and free form
responses to public health queries, created via an automated pipeline. We also
release a new dataset of the extracted UK Government public health guidance
documents used as source text for PubHealthBench. Assessing 24 LLMs on
PubHealthBench we find the latest private LLMs (GPT-4.5, GPT-4.1 and o1) have a
high degree of knowledge, achieving >90% in the MCQA setup, and outperform
humans with cursory search engine use. However, in the free form setup we see
lower performance with no model scoring >75%. Therefore, whilst there are
promising signs that state of the art (SOTA) LLMs are an increasingly accurate
source of public health information, additional safeguards or tools may still
be needed when providing free form responses on public health topics.
A new benchmark, PubHealthBench, evaluates LLMs' knowledge of UK Government public health information, finding high accuracy in MCQA but lower performance in free form responses.
Large Language Models (LLMs) Multiple Choice Question Answering (MCQA) public health PubHealthBench GPT-4.5 GPT-4.1 o1 state of the art (SOTA) free form responses