INDUS: Effective and Efficient Language Models for Scientific
  Applications
Large language models (LLMs) trained on general domain corpora showed
remarkable results on natural language processing (NLP) tasks. However,
previous research demonstrated LLMs trained using domain-focused corpora
perform better on specialized tasks. Inspired by this pivotal insight, we
developed INDUS, a comprehensive suite of LLMs tailored for the Earth science,
biology, physics, heliophysics, planetary sciences and astrophysics domains and
trained using curated scientific corpora drawn from diverse data sources. The
suite of models include: (1) an encoder model trained using domain-specific
vocabulary and corpora to address natural language understanding tasks, (2) a
contrastive-learning-based general text embedding model trained using a diverse
set of datasets drawn from multiple sources to address information retrieval
tasks and (3) smaller versions of these models created using knowledge
distillation techniques to address applications which have latency or resource
constraints. We also created three new scientific benchmark datasets namely,
CLIMATE-CHANGE-NER (entity-recognition), NASA-QA (extractive QA) and NASA-IR
(IR) to accelerate research in these multi-disciplinary fields. Finally, we
show that our models outperform both general-purpose encoders (RoBERTa) and
existing domain-specific encoders (SciBERT) on these new tasks as well as
existing benchmark tasks in the domains of interest.
664b68b7bfd9b93ba4ac0026
Bishwaranjan Bhattacharjee
664b68b7bfd9b93ba4ac0027
Aashka Trivedi
admin_assigned
2024-05-21T09:23:50.141Z
6620044150fab5fc315c5457
https://cdn-avatars.huggingface.co/v1/production/uploads/6620044150fab5fc315c5457/WaUpGHLpS0BWY2Cjr5K4S.jpeg
Aashka Trivedi
user
aashkaaa
664b68b7bfd9b93ba4ac0028
Masayasu Muraoka
664b68b7bfd9b93ba4ac0029
Muthukumaran Ramasubramanian
claimed_verified
2024-07-15T09:01:27.127Z
61099e5d86580d4580767226
/avatars/76ab955b5388c8bd732a19c0516d72c5.svg
Ramasubramanian
user
Muthukumaran
664b68b7bfd9b93ba4ac002a
Takuma Udagawa
admin_assigned
2024-05-21T09:24:38.671Z
652f684161ce812084940d19
/avatars/21a233344781e6a6f60b17a7f38f006c.svg
Takuma Udagawa
user
uddie
664b68b7bfd9b93ba4ac002b
Iksha Gurung
admin_assigned
2024-05-21T09:24:45.485Z
6499a2d9b9a0112f18e15a09
/avatars/1576e485309c45cbf1e574354667670e.svg
Iksha Gurung
user
xhagrg
664b68b7bfd9b93ba4ac002c
Rong Zhang
admin_assigned
2024-05-21T09:24:54.102Z
62bc74b71cb22b7f9bb17e45
/avatars/5d2e9f98ee36c8ebc68911a8c2369646.svg
Rong Zhang
user
rongzhangibm
664b68b7bfd9b93ba4ac002d
Bharath Dandala
664b68b7bfd9b93ba4ac002e
Rahul Ramachandran
664b68b7bfd9b93ba4ac002f
Manil Maskey
admin_assigned
2024-05-21T09:25:29.849Z
64ba94321e53e10faf0c14ca
https://cdn-avatars.huggingface.co/v1/production/uploads/64ba94321e53e10faf0c14ca/leAHIQoy8nk74Mdut3MJ7.jpeg
Manil Maskey
user
manilmaskey
664b68b7bfd9b93ba4ac0030
Kayleen Bugbee
664b68b7bfd9b93ba4ac0031
Mike Little
664b68b7bfd9b93ba4ac0032
Elizabeth Fancher
admin_assigned
2024-05-21T09:25:55.994Z
64bec6b3b1a6188809b71efd
/avatars/a42e4aa0f014287c23d0445f3459e4b2.svg
Elizabeth Fancher
user
efancher
664b68b7bfd9b93ba4ac0033
Lauren Sanders
664b68b7bfd9b93ba4ac0034
Sylvain Costes
664b68b7bfd9b93ba4ac0035
Sergi Blanco-Cuaresma
admin_assigned
2024-05-21T09:26:24.255Z
627a6840f652717251c77628
/avatars/0be9a72c5efe835bbfbc7730d819dede.svg
Sergi Blanco-Cuaresma
user
marblestation
664b68b7bfd9b93ba4ac0036
Kelly Lockhart
admin_assigned
2024-05-21T09:26:30.619Z
646258c3d9b49021880f724e
/avatars/dfefd298a9973328671a3dca6e2cb04a.svg
Kelly Lockhart
user
kelockhart
664b68b7bfd9b93ba4ac0037
Thomas Allen
664b68b7bfd9b93ba4ac0038
Felix Grazes
664b68b7bfd9b93ba4ac0039
Megan Ansdel
664b68b7bfd9b93ba4ac003a
Alberto Accomazzi
admin_assigned
2024-05-21T09:27:07.546Z
65083c0018f9c580cde6494c
https://cdn-avatars.huggingface.co/v1/production/uploads/65083c0018f9c580cde6494c/W9sBT0xvexHcwDkTaxpgA.jpeg
Alberto Accomazzi
user
aaccomazzi
664b68b7bfd9b93ba4ac003b
Yousef El-Kurdi
664b68b7bfd9b93ba4ac003c
Davis Wertheimer
admin_assigned
2024-05-21T09:27:22.285Z
6614072e2938f840baacfa11
/avatars/894fc389c32b13a6f6a894dab4fdd65b.svg
Davis Wertheimer
user
daviswer
664b68b7bfd9b93ba4ac003d
Birgit Pfitzmann
664b68b7bfd9b93ba4ac003e
Cesar Berrospi Ramis
admin_assigned
2024-05-21T09:27:33.952Z
661e328c271f493e58b1f2ad
/avatars/0146ffdad7565b28687645c1dbdd4ea7.svg
Cesar Berrospi Ramis
user
ceberam
664b68b7bfd9b93ba4ac003f
Michele Dolfi
admin_assigned
2024-05-21T09:27:41.816Z
63c64dd877caf00391004e20
/avatars/6350443da6405ea7beccf47a80d8710e.svg
Michele Dolfi
user
dolfim-ibm
664b68b7bfd9b93ba4ac0040
Rafael Teixeira de Lima
admin_assigned
2024-05-21T09:27:48.919Z
65ce0dbef3bbc55ca4d0528f
/avatars/ae19424457dbec49579fe5a4ae7177b0.svg
Rafael Teixeira de Lima
user
rtdl
664b68b7bfd9b93ba4ac0041
Panos Vegenas
664b68b7bfd9b93ba4ac0042
S. Karthik Mukkavilli
admin_assigned
2024-05-21T09:27:59.672Z
64cd0861b16aabccbe34553b
https://cdn-avatars.huggingface.co/v1/production/uploads/64cd0861b16aabccbe34553b/T-IAqBIG6SzXnUOdagAii.jpeg
S. Karthik Mukkavilli
user
drmukkavilli
664b68b7bfd9b93ba4ac0043
Peter Staar
admin_assigned
2024-05-21T09:28:06.674Z
63c6522d8fddd19431f48bcc
/avatars/e2ec9c76656b7520472718595f9dafe1.svg
Peter W. J. Staar
user
PeterStaar
664b68b7bfd9b93ba4ac0044
Sanaz Vahidinia
664b68b7bfd9b93ba4ac0045
Ryan McGranaghan
664b68b7bfd9b93ba4ac0046
Armin Mehrabian
admin_assigned
2024-05-21T09:28:22.329Z
62d0696a485856cd710e549b
https://cdn-avatars.huggingface.co/v1/production/uploads/62d0696a485856cd710e549b/HaOWjdJyVImULLykvI_ez.jpeg
Armin Mehrabian
user
arminmehrabian
664b68b7bfd9b93ba4ac0047
Tsendgar Lee
INDUS: Effective and Efficient Language Models for Scientific
  Applications
Large language models (LLMs) trained on general domain corpora showed
remarkable results on natural language processing (NLP) tasks. However,
previous research demonstrated LLMs trained using domain-focused corpora
perform better on specialized tasks. Inspired by this pivotal insight, we
developed INDUS, a comprehensive suite of LLMs tailored for the Earth science,
biology, physics, heliophysics, planetary sciences and astrophysics domains and
trained using curated scientific corpora drawn from diverse data sources. The
suite of models include: (1) an encoder model trained using domain-specific
vocabulary and corpora to address natural language understanding tasks, (2) a
contrastive-learning-based general text embedding model trained using a diverse
set of datasets drawn from multiple sources to address information retrieval
tasks and (3) smaller versions of these models created using knowledge
distillation techniques to address applications which have latency or resource
constraints. We also created three new scientific benchmark datasets namely,
CLIMATE-CHANGE-NER (entity-recognition), NASA-QA (extractive QA) and NASA-IR
(IR) to accelerate research in these multi-disciplinary fields. Finally, we
show that our models outperform both general-purpose encoders (RoBERTa) and
existing domain-specific encoders (SciBERT) on these new tasks as well as
existing benchmark tasks in the domains of interest.
INDUS, a suite of domain-specific LLMs trained on curated scientific corpora, surpasses general-purpose and existing domain-specific models on specialized NLP tasks and new scientific benchmarks.
LLMs natural language processing domain-specific models encoder model contrastive-learning-based general text embedding model knowledge distillation CLIMATE-CHANGE-NER NASA-QA NASA-IR RoBERTa SciBERT