Akash-nath29/tinyllamashakespeare

TinyLlama Shakespeare 🎭

A fine-tuned version of TinyLlama-1.1B-Chat trained on Shakespeare's complete works to generate Shakespearean-style text.

Model Details

• Base Model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Quantization: 4-bit (NF4) via bitsandbytes
• Developed by: Akash Nath
• License: MIT
• Language: English

Training Details

Training Data

The model was fine-tuned on Shakespeare's complete works including sonnets, plays, and poems (~42,000 lines of text).

Training Configuration

Parameter	Value
LoRA Rank (r)	16
LoRA Alpha	32
Target Modules	q_proj, v_proj
LoRA Dropout	0.05
Batch Size	2
Gradient Accumulation	4
Learning Rate	2e-4
Epochs	2
Precision	FP16
Max Sequence Length	256

Hardware

• GPU: NVIDIA GeForce RTX 3050 Laptop GPU
• Training Time: ~4 hours

Usage

Quick Start

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model and tokenizer
base_model = AutoModelForCausalLM.from_pretrained(
    "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    load_in_4bit=True,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")

# Load LoRA adapters
model = PeftModel.from_pretrained(base_model, "Akash-nath29/tinyllamashakespeare")

# Generate text
prompt = "To be or not to be"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_length=200, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using the Training Repository

git clone https://github.com/Akash-nath29/TinyLlamaShakespeare
cd TinyLlamaShakespeare
pip install -r requirements.txt

# Run inference
python scripts/inference.py --model_path Akash-nath29/tinyllamashakespeare --prompt "Shall I compare thee"

Example Outputs

Prompt: "To be or not to be"

Output: "To be or not to be, that's the question. I have seen you with the duke, and I know he is a man..."

Limitations

• The model is trained primarily on Shakespearean text and may not perform well on modern language tasks
• Output quality varies based on prompts and generation parameters
• The model inherits biases present in the base TinyLlama model and Shakespeare's original texts

Repository

• GitHub: https://github.com/Akash-nath29/TinyLlamaShakespeare

Citation

@misc{tinyllamashakespeare,
  author = {Akash Nath},
  title = {TinyLlama Shakespeare: Fine-tuned TinyLlama on Shakespeare's Works},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/Akash-nath29/tinyllamashakespeare}
}

Framework Versions

• PEFT 0.18.0
• Transformers 4.x
• PyTorch 2.6.0+cu124
• bitsandbytes 0.41+