tinyMind
This is a small transformer language model trained from scratch with approximately 17,731,328 parameters.
Model Details
- Architecture: GPT-style transformer
- Parameters: ~17M
- Layers: 6
- Attention Heads: 8
- Embedding Dimension: 256
- Max Sequence Length: 512
- Vocabulary Size: 50257
Training Data
The model was trained on a diverse mixture of high-quality text data including:
- OpenWebText
- Wikipedia articles
- BookCorpus
- Other curated text sources
Usage
from transformers import GPT2TokenizerFast, AutoModelForCausalLM
tokenizer = GPT2TokenizerFast.from_pretrained("HenrySentinel/tinyMind")
model = AutoModelForCausalLM.from_pretrained("HenrySentinel/tinyMind")
# Generate text
input_text = "The key to artificial intelligence is"
input_ids = tokenizer.encode(input_text, return_tensors="pt")
output = model.generate(input_ids, max_length=100, temperature=0.8, do_sample=True)
generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)
Training Details
- Optimizer: AdamW with cosine learning rate scheduling
- Learning Rate: 0.001
- Batch Size: 8
- Sequence Length: 512
- Epochs: 3
- Gradient Clipping: 1.0
Limitations
This is a small model designed for experimentation and learning. It may:
- Generate inconsistent or factually incorrect content
- Have limited knowledge compared to larger models
- Require careful prompt engineering for best results
License
Apache 2.0
- Downloads last month
- 8