FarisET/tinyllama-sft-dpo-alpaca-orca

TinyLlama-1.1B — SFT + DPO Fine-Tuned (NLP Assignment)

This is a fully merged fine-tuned version of TinyLlama-1.1B-intermediate-step-1431k-3T trained in two stages as part of an NLP course assignment.

Training Pipeline

Stage	Method	Dataset	Config
1 — SFT	LoRA (r=8, α=16, q+v proj)	tatsu-lab/alpaca (10k)	lr=2e-4, 2 epochs
2 — DPO	LoRA on SFT model (β=0.3)	Intel/orca_dpo_pairs (3k)	lr=1e-5, 1 epoch

Both adapters were merged into the base model weights before upload — no adapter loading needed at inference.

Evaluation Results (10-prompt test set, GPT-4o references)

Model	Mean BLEU	Mean BERTScore F1
Base TinyLlama-1.1B	2.60	−0.016
After SFT (T2)	6.45	0.222
After DPO (D3) — this model	2.998	0.051

SFT produced the largest metric gains. DPO improved alignment over the base model but showed some regression on BLEU due to distributional shift between the Orca training format and the Alpaca-style evaluation prompts.

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "FarisET/tinyllama-sft-dpo-alpaca-orca",
    torch_dtype=torch.float16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("FarisET/tinyllama-sft-dpo-alpaca-orca")

prompt = (
    "Below is an instruction that describes a task. "
    "Write a response that appropriately completes the request.\n\n"
    "### Instruction:\n"
    "Explain the concept of recursion in programming with a simple example.\n\n"
    "### Response:\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
input_len = inputs["input_ids"].shape[1]

with torch.inference_mode():
    out = model.generate(
        **inputs,
        max_new_tokens=200,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(out[0][input_len:], skip_special_tokens=True))

Framework Versions

• transformers 4.44.2
• peft 0.12.0
• trl 0.9.6
• bitsandbytes 0.46.0 (used during training only)
• torch 2.4.x (Google Colab T4 GPU)