OpenBezoar: Small, Cost-Effective and Open Models Trained on Mixes of
  Instruction Data
Instruction fine-tuning pretrained LLMs for diverse downstream tasks has
demonstrated remarkable success and has captured the interest of both academics
and practitioners. To ensure such fine-tuned LLMs align with human preferences,
techniques such as RLHF and DPO have emerged. At the same time, there is
increasing interest in smaller parameter counts for models. In this work, using
OpenLLaMA 3Bv2 as a base model, we describe the recipe used to fine-tune the
OpenBezoar family of models. In this recipe: We first generate synthetic
instruction fine-tuning data using an open and commercially non-restrictive
instruction fine-tuned variant of the Falcon-40B model under three schemes
based on: LaMini-LM, WizardLM/Evol-Instruct (with databricks-dolly-15k as a
seed dataset) and Orca (with the Flan Collection as a seed dataset), then
filter these generations using GPT-4 as a human proxy. We then perform
cost-effective QLoRA-based supervised fine-tuning sequentially with each
scheme. The resulting checkpoint is further fine-tuned with a subset of the
HH-RLHF dataset to minimize distribution shift prior to using the DPO loss to
obtain the final checkpoint. Evaluation is done with the LM Eval Harness
tasks/metrics as well as on MT-Bench using the "LLM-as-a-judge" framework with
Claude 2.1, with the finding that the final checkpoint,
"OpenBezoar-HH-RLHF-DPO", demonstrates superior performance over many models at
the 3B parameter scale, even outperforming the top model in one of the
categories on the Huggingface Open LLM Leaderboard. We release
"OpenBezoar-SFT", "OpenBezoar-HH-RLHF-SFT", "OpenBezoar-HH-RLHF-DPO"
checkpoints, alongside our generated datasets on HuggingFace at
https://huggingface.co/collections/SurgeGlobal/open-bezoar-6620a24923e12127e9e2b9cc
and our codebase at
https://bitbucket.org/paladinanalytics/workspace/projects/OP.
6621c753d057cd6d668bad28
Chandeepa Dissanayake
claimed_verified
2024-04-19T07:38:48.032Z
64b65956de30803f4e9a6e46
https://cdn-avatars.huggingface.co/v1/production/uploads/64b65956de30803f4e9a6e46/044T6a04Ekx0fFStOaKiB.jpeg
Chandeepa Dissanayake
user
chansurgeplus
6621c753d057cd6d668bad29
Lahiru Lowe
claimed_verified
2024-04-24T18:23:45.338Z
66293de6fa14eaccf04382cf
https://cdn-avatars.huggingface.co/v1/production/uploads/66293de6fa14eaccf04382cf/SjNe5po9O2hWKzoL94UGc.png
Lahiru Lowe
user
LahiruTLowe
6621c753d057cd6d668bad2a
Sachith Gunasekara
claimed_verified
2024-04-19T07:38:45.078Z
647eff9aaa8c04bbf9365219
https://cdn-avatars.huggingface.co/v1/production/uploads/647eff9aaa8c04bbf9365219/lu8O13Fgsferal-yhAtih.jpeg
Sachith Gunasekara
user
sachithgunasekara
6621c753d057cd6d668bad2b
Yasiru Ratnayake
claimed_verified
2024-04-19T11:17:25.664Z
6440fa0b66a62c605c91d739
/avatars/3b941c8d2a7941bb77f46167907df704.svg
Yasiru Dhanika Ratnayake
user
yasiru89
OpenBezoar: Small, Cost-Effective and Open Models Trained on Mixes of
  Instruction Data
Instruction fine-tuning pretrained LLMs for diverse downstream tasks has
demonstrated remarkable success and has captured the interest of both academics
and practitioners. To ensure such fine-tuned LLMs align with human preferences,
techniques such as RLHF and DPO have emerged. At the same time, there is
increasing interest in smaller parameter counts for models. In this work, using
OpenLLaMA 3Bv2 as a base model, we describe the recipe used to fine-tune the
OpenBezoar family of models. In this recipe: We first generate synthetic
instruction fine-tuning data using an open and commercially non-restrictive
instruction fine-tuned variant of the Falcon-40B model under three schemes
based on: LaMini-LM, WizardLM/Evol-Instruct (with databricks-dolly-15k as a
seed dataset) and Orca (with the Flan Collection as a seed dataset), then
filter these generations using GPT-4 as a human proxy. We then perform
cost-effective QLoRA-based supervised fine-tuning sequentially with each
scheme. The resulting checkpoint is further fine-tuned with a subset of the
HH-RLHF dataset to minimize distribution shift prior to using the DPO loss to
obtain the final checkpoint. Evaluation is done with the LM Eval Harness
tasks/metrics as well as on MT-Bench using the "LLM-as-a-judge" framework with
Claude 2.1, with the finding that the final checkpoint,
"OpenBezoar-HH-RLHF-DPO", demonstrates superior performance over many models at
the 3B parameter scale, even outperforming the top model in one of the
categories on the Huggingface Open LLM Leaderboard. We release
"OpenBezoar-SFT", "OpenBezoar-HH-RLHF-SFT", "OpenBezoar-HH-RLHF-DPO"
checkpoints, alongside our generated datasets on HuggingFace at
https://huggingface.co/collections/SurgeGlobal/open-bezoar-6620a24923e12127e9e2b9cc
and our codebase at
https://bitbucket.org/paladinanalytics/workspace/projects/OP.
A fine-tuning recipe using OpenLLaMA 3Bv2, synthetic data generation, QLoRA, HH-RLHF, and DPO yields superior performance for the OpenBezoar family of models.
instruction fine-tuning RLHF DPO OpenLLaMA LaMini-LM WizardLM Evol-Instruct databricks-dolly-15k Orca Flan Collection GPT-4 QLoRA supervised fine-tuning HH-RLHF LM Eval Harness MT-Bench LLM-as-a-judge OpenBezoar