---
library_name: transformers
license: apache-2.0
base_model:
- allenai/specter2_base
pipeline_tag: text-classification
datasets:
- nicolauduran45/horizon_clusters_annotated
---

# SPECTER2-base Multilabel Horizon Clusters Classifier

This model is based on **[SPECTER2-base](https://huggingface.co/allenai/specter2_base)**, fine-tuned for multilabel classification of scientific publications into Horizon Europe clusters.

---

## Model Description

- **Base model:** [allenai/specter2_base](https://huggingface.co/allenai/specter2_base)
- **Task:** Multilabel classification (assigns one or more clusters per document)
- **Labels:** 6 Horizon Europe clusters (see below)
- **Languages:** English
- **Input:** Title and abstract concatenated

---

## Training Details

- **Training framework:** Hugging Face Transformers (`Trainer`)
- **Batch size:** 4
- **Learning rate:** 2e-5
- **Epochs:** 6
- **Optimizer:** AdamW with weight decay 0.01
- **Loss:** Binary Cross-Entropy with Logits
- **Best model selection:** F1-score on validation set

---

## Clusters (Labels)

- Civil Security for Society
- Climate, Energy and Mobility
- Culture, Creativity and Inclusive Society
- Digital, Industry and Space
- Food, Bioeconomy, Natural Resources, Agriculture and Environment
- Health

---

## Evaluation Metrics

| Epoch | Training Loss | Validation Loss | F1       | ROC AUC   | Accuracy |
|-------|--------------|----------------|----------|-----------|----------|
| 1     | No log       | 0.1774         | 0.910    | 0.9368    | 0.766    |
| 2     | 0.0606       | 0.1849         | 0.921    | 0.9454    | 0.787    |
| 3     | 0.0351       | 0.2071         | 0.919    | 0.9434    | 0.787    |
| 4     | 0.0180       | 0.2191         | 0.921    | 0.9451    | 0.793    |
| 5     | 0.0093       | 0.2295         | 0.921    | 0.9451    | 0.793    |
| 6     | 0.0060       | 0.2307         | 0.921    | 0.9451    | 0.793    |

**Best epoch:** 6 (highest F1 and accuracy, last improvement at epoch 4)

- **Final validation loss:** 0.2307
- **Final F1:** 0.9212
- **Final ROC AUC:** 0.9451
- **Final Accuracy:** 0.7927

---

## Per-Category Classification Report

| Label                                                               | Precision | Recall | F1-score | Support |
|---------------------------------------------------------------------|-----------|--------|----------|---------|
| Civil Security for Society                                          | 0.97      | 0.79   | 0.87     | 39      |
| Climate, Energy and Mobility                                       | 0.94      | 0.91   | 0.93     | 91      |
| Culture, Creativity and Inclusive Society                           | 0.89      | 0.88   | 0.88     | 96      |
| Digital, Industry and Space                                         | 0.93      | 0.92   | 0.93     | 214     |
| Food, Bioeconomy, Natural Resources, Agriculture and Environment    | 0.89      | 0.97   | 0.93     | 75      |
| Health                                                             | 0.96      | 0.96   | 0.96     | 73      |
| **micro avg**                                                      | 0.93      | 0.91   | 0.92     | 588     |
| **macro avg**                                                      | 0.93      | 0.91   | 0.92     | 588     |
| **weighted avg**                                                   | 0.93      | 0.91   | 0.92     | 588     |
| **samples avg**                                                    | 0.91      | 0.92   | 0.90     | 588     |

---

## License

This model is licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).