Training Task Experts through Retrieval Based Distillation
One of the most reliable ways to create deployable models for specialized
tasks is to obtain an adequate amount of high-quality task-specific data.
However, for specialized tasks, often such datasets do not exist. Existing
methods address this by creating such data from large language models (LLMs)
and then distilling such knowledge into smaller models. However, these methods
are limited by the quality of the LLMs output, and tend to generate repetitive
or incorrect data. In this work, we present Retrieval Based Distillation
(ReBase), a method that first retrieves data from rich online sources and then
transforms them into domain-specific data. This method greatly enhances data
diversity. Moreover, ReBase generates Chain-of-Thought reasoning and distills
the reasoning capacity of LLMs. We test our method on 4 benchmarks and results
show that our method significantly improves performance by up to 7.8% on SQuAD,
1.37% on MNLI, and 1.94% on BigBench-Hard.
668d62cdc04604b679487253
Jiaxin Ge
admin_assigned
2024-07-09T21:36:12.396Z
64bdc1e8d05a97d722b2c1f6
/avatars/b04cbe17e7cd0a1f5bf0e8e008640a1d.svg
Jiaxin Ge
user
para-lost
668d62cdc04604b679487254
Xueying Jia
668d62cdc04604b679487255
Vijay Viswanathan
admin_assigned
2024-07-09T21:37:02.268Z
603c44e677a2a899efe25828
https://cdn-avatars.huggingface.co/v1/production/uploads/1614562504843-noauth.jpeg
Vijay Viswanathan
user
viswavi
668d62cdc04604b679487256
Hongyin Luo
claimed_verified
2024-07-09T21:09:05.252Z
62bc495016897fa76cdcda18
https://cdn-avatars.huggingface.co/v1/production/uploads/1656506661693-noauth.jpeg
Hongyin Luo
user
luohy
668d62cdc04604b679487257
Graham Neubig
admin_assigned
2024-07-09T21:37:09.136Z
60de14638bedd2315529d43f
https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
Graham Neubig
user
gneubig
Training Task Experts through Retrieval Based Distillation
One of the most reliable ways to create deployable models for specialized
tasks is to obtain an adequate amount of high-quality task-specific data.
However, for specialized tasks, often such datasets do not exist. Existing
methods address this by creating such data from large language models (LLMs)
and then distilling such knowledge into smaller models. However, these methods
are limited by the quality of the LLMs output, and tend to generate repetitive
or incorrect data. In this work, we present Retrieval Based Distillation
(ReBase), a method that first retrieves data from rich online sources and then
transforms them into domain-specific data. This method greatly enhances data
diversity. Moreover, ReBase generates Chain-of-Thought reasoning and distills
the reasoning capacity of LLMs. We test our method on 4 benchmarks and results
show that our method significantly improves performance by up to 7.8% on SQuAD,
1.37% on MNLI, and 1.94% on BigBench-Hard.
Retrieval Based Distillation enhances data diversity and distills reasoning capacity from online sources, significantly improving performance on benchmarks.
Retrieval Based Distillation Chain-of-Thought reasoning LLMs SQuAD MNLI BigBench-Hard