---
language: en
datasets: karpathy/tiny_shakespeare
library_name: pytorch
tags:
  - transformer
  - decoder-only
  - character-level
  - text-generation
  - shakespeare
license: mit
model_name: decoder-shakespeare-gpt
pipeline_tag: text-generation
---

# Decoder-Only Shakespeare GPT

This is a lightweight GPT-style decoder-only transformer model trained on the Tiny Shakespeare dataset (`karpathy/tiny_shakespeare`). It uses a custom implementation in PyTorch and supports character-level text generation.

## Model Details

- **Architecture**: Decoder-only Transformer
- **Layers**: 2
- **Embedding Size**: 128
- **Heads**: 4
- **Sequence Length**: 64
- **Training Epochs**: 4
- **Tokenizer**: GPT-2 tokenizer (character-level)

## Training

Trained on the full Tiny Shakespeare dataset for 4 epochs using Adam optimizer and cross-entropy loss. Validation loss is tracked and logged using Weights & Biases (wandb).

## Usage

```python
from transformers import AutoTokenizer
import torch
from model import DecoderOnlyTransformer  # custom model class

tokenizer = AutoTokenizer.from_pretrained("NataliiaM15/decoder-shakespeare-gpt")
model = DecoderOnlyTransformer(
    vocab_size=tokenizer.vocab_size,
    embed_dim=128,
    num_heads=4,
    num_layers=2,
    seq_len=64
)
model.load_state_dict(torch.load("pytorch_model.bin"))
model.eval()

# Generate text
prompt = "ROMEO:"
input_ids = tokenizer.encode(prompt, return_tensors="pt")
# generation loop would go here...