SearchInstruct: Enhancing Domain Adaptation via Retrieval-Based
  Instruction Dataset Creation
Supervised Fine-Tuning (SFT) is essential for training large language models
(LLMs), significantly enhancing critical capabilities such as instruction
following and in-context learning. Nevertheless, creating suitable training
datasets tailored for specific domains remains challenging due to unique domain
constraints and data scarcity. In this paper, we propose SearchInstruct, an
innovative method explicitly designed to construct high quality instruction
datasets for SFT. Our approach begins with a limited set of domain specific,
human generated questions, which are systematically expanded using a large
language model. Subsequently, domain relevant resources are dynamically
retrieved to generate accurate and contextually appropriate answers for each
augmented question. Experimental evaluation demonstrates that SearchInstruct
enhances both the diversity and quality of SFT datasets, leading to measurable
improvements in LLM performance within specialized domains. Additionally, we
show that beyond dataset generation, the proposed method can also effectively
facilitate tasks such as model editing, enabling efficient updates to existing
models. To facilitate reproducibility and community adoption, we provide full
implementation details, the complete set of generated instruction response
pairs, and the source code in a publicly accessible Git repository:
[https://github.com/mostafaamiri/SearchInstruct](https://github.com/mostafaamiri/SearchInstruct)
68c8f911733e345e52ac1f20
Iman Barati
claimed_verified
2025-09-16T09:41:39.161Z
654a90ab55ecd2d37ac99965
https://cdn-avatars.huggingface.co/v1/production/uploads/654a90ab55ecd2d37ac99965/8lmXmiY0feZ9ORbkMrjJm.jpeg
Iman Barati
user
Iman998
68c8f911733e345e52ac1f21
Mostafa Amiri
68c8f911733e345e52ac1f22
Heshaam Faili
SearchInstruct: Enhancing Domain Adaptation via Retrieval-Based
  Instruction Dataset Creation
Supervised Fine-Tuning (SFT) is essential for training large language models
(LLMs), significantly enhancing critical capabilities such as instruction
following and in-context learning. Nevertheless, creating suitable training
datasets tailored for specific domains remains challenging due to unique domain
constraints and data scarcity. In this paper, we propose SearchInstruct, an
innovative method explicitly designed to construct high quality instruction
datasets for SFT. Our approach begins with a limited set of domain specific,
human generated questions, which are systematically expanded using a large
language model. Subsequently, domain relevant resources are dynamically
retrieved to generate accurate and contextually appropriate answers for each
augmented question. Experimental evaluation demonstrates that SearchInstruct
enhances both the diversity and quality of SFT datasets, leading to measurable
improvements in LLM performance within specialized domains. Additionally, we
show that beyond dataset generation, the proposed method can also effectively
facilitate tasks such as model editing, enabling efficient updates to existing
models. To facilitate reproducibility and community adoption, we provide full
implementation details, the complete set of generated instruction response
pairs, and the source code in a publicly accessible Git repository:
[https://github.com/mostafaamiri/SearchInstruct](https://github.com/mostafaamiri/SearchInstruct)
SearchInstruct enhances supervised fine-tuning datasets for large language models by expanding domain-specific questions and retrieving accurate answers, improving model performance and enabling efficient model editing.
supervised fine-tuning large language models instruction datasets domain-specific human-generated questions large language model domain relevant resources instruction response pairs model editing