OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented
  LMs
Scientific progress depends on researchers' ability to synthesize the growing
body of literature. Can large language models (LMs) assist scientists in this
task? We introduce OpenScholar, a specialized retrieval-augmented LM that
answers scientific queries by identifying relevant passages from 45 million
open-access papers and synthesizing citation-backed responses. To evaluate
OpenScholar, we develop ScholarQABench, the first large-scale multi-domain
benchmark for literature search, comprising 2,967 expert-written queries and
208 long-form answers across computer science, physics, neuroscience, and
biomedicine. On ScholarQABench, OpenScholar-8B outperforms GPT-4o by 5% and
PaperQA2 by 7% in correctness, despite being a smaller, open model. While GPT4o
hallucinates citations 78 to 90% of the time, OpenScholar achieves citation
accuracy on par with human experts. OpenScholar's datastore, retriever, and
self-feedback inference loop also improves off-the-shelf LMs: for instance,
OpenScholar-GPT4o improves GPT-4o's correctness by 12%. In human evaluations,
experts preferred OpenScholar-8B and OpenScholar-GPT4o responses over
expert-written ones 51% and 70% of the time, respectively, compared to GPT4o's
32%. We open-source all of our code, models, datastore, data and a public demo.
673ff653b86fd98d952b2afa
Akari Asai
claimed_verified
2024-11-22T10:02:01.251Z
6266e0cb7a1f5a1562c4e86e
https://cdn-avatars.huggingface.co/v1/production/uploads/6266e0cb7a1f5a1562c4e86e/kkJZPssa76mdRPnKH-q3e.jpeg
Akari Asai
user
akariasai
673ff653b86fd98d952b2afb
Jacqueline He
claimed_verified
2024-11-22T10:01:59.353Z
627c52e51b02d7b5bf671dca
https://cdn-avatars.huggingface.co/v1/production/uploads/1666240522285-627c52e51b02d7b5bf671dca.png
Jacqueline He
user
jacquelinehe
673ff653b86fd98d952b2afc
Rulin Shao
673ff653b86fd98d952b2afd
Weijia Shi
673ff653b86fd98d952b2afe
Amanpreet Singh
673ff653b86fd98d952b2aff
Joseph Chee Chang
673ff653b86fd98d952b2b00
Kyle Lo
admin_assigned
2024-11-22T13:10:05.639Z
670c92d0b6e0a119d335e315
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/hSDQBRzrbGtZawSb-LcGx.png
Kyle Lo
user
kylelo
673ff653b86fd98d952b2b01
Luca Soldaini
admin_assigned
2024-11-22T13:10:11.852Z
5f04d8c45d08220171a0ad32
https://cdn-avatars.huggingface.co/v1/production/uploads/5f04d8c45d08220171a0ad32/uXEta6nqBabrUlAOXnS5g.jpeg
Luca Soldaini
user
soldni
673ff653b86fd98d952b2b02
Sergey Feldman
admin_assigned
2024-11-22T13:10:17.173Z
6272affe9824228a117849fe
https://cdn-avatars.huggingface.co/v1/production/uploads/6272affe9824228a117849fe/m83mFnu_O8LPem3SmSwRL.jpeg
Sergey Feldman
user
sergeyfeldman
673ff653b86fd98d952b2b03
Mike D'arcy
admin_assigned
2024-11-22T13:10:22.527Z
659c9bc896d8c1e3dbea8554
/avatars/0f88ac6e937fcdfad4fc97224d6dd76f.svg
Mike D'Arcy
user
miked-ai
673ff653b86fd98d952b2b04
David Wadden
admin_assigned
2024-11-22T13:10:27.652Z
5ff4e2a1463be69ae4bd42bd
https://cdn-avatars.huggingface.co/v1/production/uploads/1609884433338-5ff4e2a1463be69ae4bd42bd.jpeg
David Wadden
user
dwadden
673ff653b86fd98d952b2b05
Matt Latzke
admin_assigned
2024-11-22T13:10:33.621Z
65f9dd02d3aad0fbb6dc6b83
/avatars/8d2d4c5500c13c0631754dd71ae7721f.svg
Matt Latzke
user
MLatzke
673ff653b86fd98d952b2b06
Minyang Tian
admin_assigned
2024-11-22T13:10:39.284Z
64ead5f1213a0415bd22d0e4
/avatars/2797e96d93b999a3e5f816935eb43673.svg
Minyang Tian
user
mtian8
673ff653b86fd98d952b2b07
Pan Ji
673ff653b86fd98d952b2b08
Shengyan Liu
admin_assigned
2024-11-22T13:10:53.076Z
661551375e8f1eaaba4c53b3
/avatars/1d41150d06ce4e4726269b9c48cbbc10.svg
LiuShengyang
user
LSY3579
673ff653b86fd98d952b2b09
Hao Tong
673ff653b86fd98d952b2b0a
Bohao Wu
admin_assigned
2024-11-22T13:11:03.097Z
6717b1a510c78be6e4335dd4
/avatars/368e4af45151d2756726f45ec9cb0b64.svg
Bohao Wu
user
nlogn-27
673ff653b86fd98d952b2b0b
Yanyu Xiong
admin_assigned
2024-11-22T13:11:08.422Z
645862428eef89b5c3c25ba1
/avatars/b46887a16d17c6e552d26a5a140e81e0.svg
Yanyu Xiong
user
yxiong5
673ff653b86fd98d952b2b0c
Luke Zettlemoyer
673ff653b86fd98d952b2b0d
Graham Neubig
admin_assigned
2024-11-22T13:11:22.628Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
673ff653b86fd98d952b2b0e
Dan Weld
673ff653b86fd98d952b2b0f
Doug Downey
673ff653b86fd98d952b2b10
Wen-tau Yih
673ff653b86fd98d952b2b11
Pang Wei Koh
admin_assigned
2024-11-22T13:11:37.747Z
641b4263abfce26bcf7b27de
/avatars/e91b4205e4f74b0dd8c333c23203a924.svg
Pang Wei Koh
user
pangwei
673ff653b86fd98d952b2b12
Hannaneh Hajishirzi
OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented
  LMs
Scientific progress depends on researchers' ability to synthesize the growing
body of literature. Can large language models (LMs) assist scientists in this
task? We introduce OpenScholar, a specialized retrieval-augmented LM that
answers scientific queries by identifying relevant passages from 45 million
open-access papers and synthesizing citation-backed responses. To evaluate
OpenScholar, we develop ScholarQABench, the first large-scale multi-domain
benchmark for literature search, comprising 2,967 expert-written queries and
208 long-form answers across computer science, physics, neuroscience, and
biomedicine. On ScholarQABench, OpenScholar-8B outperforms GPT-4o by 5% and
PaperQA2 by 7% in correctness, despite being a smaller, open model. While GPT4o
hallucinates citations 78 to 90% of the time, OpenScholar achieves citation
accuracy on par with human experts. OpenScholar's datastore, retriever, and
self-feedback inference loop also improves off-the-shelf LMs: for instance,
OpenScholar-GPT4o improves GPT-4o's correctness by 12%. In human evaluations,
experts preferred OpenScholar-8B and OpenScholar-GPT4o responses over
expert-written ones 51% and 70% of the time, respectively, compared to GPT4o's
32%. We open-source all of our code, models, datastore, data and a public demo.
OpenScholar, a specialized language model, enhances scientific query synthesis by retrieving relevant passages and providing citation-backed answers, outperforming existing models in correctness and citation accuracy.
large language models retrieval-augmented LM passage identification citation-backed responses large-scale benchmark ScholarQABench multi-domain benchmark computer science physics neuroscience biomedicine citation accuracy inference loop off-the-shelf LMs human evaluations