metadata
license: apache-2.0
base_model: llm-semantic-router/mmbert-32k-yarn
tags:
- text-classification
- intent-classification
- modernbert
- lora
- peft
- mmlu-pro
datasets:
- TIGER-Lab/MMLU-Pro
- LLM-Semantic-Router/category-classifier-supplement
language:
- en
- multilingual
metrics:
- accuracy
- f1
pipeline_tag: text-classification
mmBERT-32K Intent Classifier (LoRA Adapter)
LoRA adapter for intent classification based on mmBERT-32K-YaRN (32K context, multilingual).
Model Details
- Base Model: llm-semantic-router/mmbert-32k-yarn
- Training Method: LoRA (Low-Rank Adaptation)
- LoRA Rank: 32
- LoRA Alpha: 64
- Trainable Parameters: 6.8M (2.2% of base model)
- Adapter Size: 27 MB
Training Data
- Primary: TIGER-Lab/MMLU-Pro (~12K academic questions)
- Supplement: LLM-Semantic-Router/category-classifier-supplement (653 samples including casual "other" examples)
Categories (14 classes)
biology, business, chemistry, computer science, economics, engineering, health, history, law, math, other, philosophy, physics, psychology
Performance
| Metric | Score |
|---|---|
| Test Accuracy | 80.0% |
| Adapter Size | 27 MB |
Usage
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel
# Load base model and LoRA adapter
base_model = AutoModelForSequenceClassification.from_pretrained(
"llm-semantic-router/mmbert-32k-yarn", num_labels=14
)
model = PeftModel.from_pretrained(base_model, "llm-semantic-router/mmbert32k-intent-classifier-lora")
tokenizer = AutoTokenizer.from_pretrained("llm-semantic-router/mmbert32k-intent-classifier-lora")
# Inference
inputs = tokenizer("How do neural networks learn?", return_tensors="pt")
outputs = model(**inputs)
predicted_class = outputs.logits.argmax().item()
Training Configuration
- Epochs: 5
- Batch Size: 16
- Learning Rate: 2e-4
- Weight Decay: 0.1
- Optimizer: AdamW with cosine LR scheduler