Initial commit of WimBERT-synth-v0

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

MODEL_CARD.md

@@ -0,0 +1,121 @@
+# WimBERT v0
+
+WimBERT is a dual‑head, multi‑label classifier for Dutch municipal complaint messages.
+The model uses a shared mmBERT‑base encoder with two MLP heads:
+- Onderwerp (topics): 96 labels
+- Beleving (experience): 26 labels
+
+Trained with a combined objective: alpha · (1 − Soft‑F1) + (1 − alpha) · BCE.
+
+## Overview
+- Encoder: mmBERT‑base (multilingual)
+- Heads: 2× MLP (Linear → Dropout → ReLU → Linear)
+- Labels: 96 onderwerp, 26 beleving
+- Task: Multi‑label classification (sigmoid per class)
+- Thresholds: Disabled (fixed 0.5 used for evaluation/inference)
+
+## Intended Use
+- Classify incoming Dutch complaint messages into topical (onderwerp) and experiential (beleving) labels.
+- Useful for analytics, routing, and trend insights. Not intended for legal or benefit decisions without human review.
+
+## Training Data
+- Source: `UWV/wim-synthetic-data-rd` (train split)
+- Samples: 9,351
+- Labels: 96 onderwerp, 26 beleving
+- Avg labels per sample: onderwerp 1.75, beleving 1.89
+- Shapes: onderwerp (9351, 96), beleving (9351, 26)
+- Train/Val split: 7,480 / 1,871 (80/20)
+
+## Training Setup
+- Date: 2025‑10‑20
+- Hardware: NVIDIA A100 GPU
+- Epochs: 15
+- Batch size: 16
+- Sequence length: 1,408 tokens
+- Optimizer: AdamW
+- Scheduler: Linear warmup (10%) → cosine annealing, `min_lr=1e‑6`
+- Gradient clipping: max_norm 1.0
+- Random seed: 42
+
+### Hyperparameters
+- alpha (F1 weight): 0.15
+- dropout: 0.20
+- encoder peak LR: 8e‑5
+- temperature (Soft‑F1): 2.0
+- learnable thresholds: false
+- initial_threshold: 0.565 (not used, thresholds disabled)
+- threshold LR mult: 5.0 (not used because thresholds disabled)
+
+## Metrics
+Final validation (500 samples):
+- Onderwerp:
+  - Accuracy: 99.8%
+  - Precision: 0.960
+  - Recall: 0.905
+  - F1: 0.932
+- Beleving:
+  - Accuracy: 97.1%
+  - Precision: 0.859
+  - Recall: 0.730
+  - F1: 0.789
+- Combined:
+  - Average Accuracy: 98.4%
+  - Average F1: 0.861
+
+## Saved Artifacts
+- HF‑compatible files:
+  - `model.safetensors` — encoder weights
+  - `config.json` — encoder config
+  - `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json` — tokenizer
+  - `dual_head_state.pt` — classification heads + metadata (no thresholds included when disabled)
+  - `label_names.json` — label names for both heads
+- `inference_mmbert_hf_example.py` — example inference script (CLI)
+
+## Inference
+Quick start (script):
+- `python inference_mmbert_hf_example.py [model_dir=. ] "Uw voorbeeldzin hier"`
+
+Minimal code (probabilities + top‑k):
+```python
+import os, json, torch, torch.nn as nn
+from transformers import AutoModel, AutoTokenizer
+
+model_dir = "."
+tok = AutoTokenizer.from_pretrained(model_dir)
+enc = AutoModel.from_pretrained(model_dir).eval()
+state = torch.load(os.path.join(model_dir, "dual_head_state.pt"), map_location="cpu")
+with open(os.path.join(model_dir, "label_names.json")) as f:
+    labels = json.load(f)
+
+hidden = enc.config.hidden_size
+drop = float(state.get("dropout", 0.1))
+n_on, n_be = int(state["num_onderwerp"]), int(state["num_beleving"])
+on_head = nn.Sequential(nn.Linear(hidden, hidden), nn.Dropout(drop), nn.ReLU(), nn.Linear(hidden, n_on)).eval()
+be_head = nn.Sequential(nn.Linear(hidden, hidden), nn.Dropout(drop), nn.ReLU(), nn.Linear(hidden, n_be)).eval()
+on_head.load_state_dict(state["onderwerp_head_state"])
+be_head.load_state_dict(state["beleving_head_state"])
+
+text = "Goedemiddag, ik heb al drie keer gebeld over mijn uitkering ..."
+enc_inputs = tok(text, truncation=True, padding="max_length", max_length=int(state.get("max_length", 512)), return_tensors="pt")
+pooled = enc(**enc_inputs).last_hidden_state[:, 0, :]
+on_probs = torch.sigmoid(on_head(pooled))[0]
+be_probs = torch.sigmoid(be_head(pooled))[0]
+
+topk = lambda p, names, k=5: [(names[i], float(p[i])) for i in torch.topk(p, k=min(k, len(p))).indices]
+print("Onderwerp:", topk(on_probs, labels["onderwerp"]))
+print("Beleving:",  topk(be_probs, labels["beleving"]))
+```
+
+## Limitations & Risks
+- Domain: Dutch complaint messages; performance may degrade out‑of‑domain or in other languages.
+- Thresholding: No learned thresholds; 0.5 cutoff is a simple heuristic.
+- Label imbalance and multi‑label ambiguity can affect precision/recall trade‑offs.
+
+## Reproduction
+- Script: `train_mmbert_dual_soft_f1_simplified.py`
+- Env: see `requirements.txt` (PyTorch, Transformers, Datasets, wandb)
+- Key config: seed 42, batch size 16, epochs 13, max_length 1408, α=0.15, encoder_peak_lr=8e‑5, warmup_ratio=0.1, min_lr=1e‑6.
+
+## Acknowledgements
+- UWV WIM synthetic RD dataset
+- Hugging Face Transformers/Datasets

config.json

@@ -0,0 +1,46 @@
+{
+  "architectures": [
+    "ModernBertModel"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 1,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 1,
+  "global_attn_every_n_layers": 3,
+  "global_rope_theta": 160000,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 768,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "layer_norm_eps": 1e-05,
+  "local_attention": 128,
+  "local_rope_theta": 160000,
+  "mask_token_id": 4,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 22,
+  "pad_token_id": 0,
+  "position_embedding_type": "sans_pos",
+  "repad_logits_with_grad": false,
+  "sep_token_id": 1,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "transformers_version": "4.57.1",
+  "vocab_size": 256000
+}

dual_head_state.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1823023519e9bf2f730708ef03835144a46fd1624557c6c63af6efd305e9e818
+size 5103997

inference_mmbert_hf_example.py

@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+"""
+Concise inference: load HF bundle and predict on one text.
+Usage:
+  python inference_mmbert_hf_example.py [model_dir] [text]
+Defaults:
+  model_dir = .
+  text      = simple Dutch example
+"""
+
+import os, sys, json, torch
+import torch.nn as nn
+from transformers import AutoModel, AutoTokenizer
+
+
+def main():
+    model_dir = sys.argv[1] if len(sys.argv) > 1 else "."
+    text = sys.argv[2] if len(sys.argv) > 2 else (
+        "Het is echt NIET te doen hier!!! Door dat hele filmfestival zijn er elke avond mensen aan het schreeuwen en harde muziek tot laat Ik kan gewoon niet meer slapen  Hoe is dit ooit goedgekeurd zo vlak na de feestdagen?????? Heb al beelden gemaakt als bewijs, kan ik die ergens heen sturen?? Het moet toch snellre opgelost kunnen worden dan dit, het duurt allemaal veel te lang Kunnen jullie dr ff naar kijken????"
+    )
+
+    device = torch.device("cuda" if torch.cuda.is_available() else ("mps" if getattr(torch.backends, "mps", None) and torch.backends.mps.is_available() else "cpu"))
+
+    # Load encoder + tokenizer + heads metadata
+    tokenizer = AutoTokenizer.from_pretrained(model_dir)
+    encoder = AutoModel.from_pretrained(model_dir).to(device).eval()
+    state = torch.load(os.path.join(model_dir, "dual_head_state.pt"), map_location="cpu")
+    with open(os.path.join(model_dir, "label_names.json")) as f:
+        labels = json.load(f)
+
+    hidden = encoder.config.hidden_size
+    n_on, n_be = int(state["num_onderwerp"]), int(state["num_beleving"]) 
+    drop = float(state.get("dropout", 0.1))
+    max_len = int(state.get("max_length", 512))
+
+    # Rebuild heads and load weights
+    onderwerp_head = nn.Sequential(nn.Linear(hidden, hidden), nn.Dropout(drop), nn.ReLU(), nn.Linear(hidden, n_on)).to(device).eval()
+    beleving_head  = nn.Sequential(nn.Linear(hidden, hidden), nn.Dropout(drop), nn.ReLU(), nn.Linear(hidden, n_be)).to(device).eval()
+    onderwerp_head.load_state_dict(state["onderwerp_head_state"], strict=True)
+    beleving_head.load_state_dict(state["beleving_head_state"], strict=True)
+
+    # Encode and predict
+    with torch.inference_mode():
+        enc = tokenizer(text, truncation=True, padding="max_length", max_length=max_len, return_tensors="pt")
+        input_ids, attn = enc["input_ids"].to(device), enc["attention_mask"].to(device)
+        pooled = encoder(input_ids=input_ids, attention_mask=attn).last_hidden_state[:, 0, :]
+        on_probs = torch.sigmoid(onderwerp_head(pooled))[0].cpu()
+        be_probs = torch.sigmoid(beleving_head(pooled))[0].cpu()
+
+    # Top-5 per head (probability)
+    def topk(probs, names, k=5):
+        idx = torch.topk(probs, k=min(k, len(probs))).indices.tolist()
+        return [(names[i], float(probs[i])) for i in idx]
+
+    print(f"Onderwerp top-5: {[f'{n}: {p:.3f}' for n,p in topk(on_probs, labels['onderwerp'])]}")
+    print(f"Beleving top-5:  {[f'{n}: {p:.3f}' for n,p in topk(be_probs, labels['beleving'])]}")
+    print(f"Device: {device} | max_length: {max_len} | model_dir: {model_dir}")
+
+
+if __name__ == "__main__":
+    main()

label_names.json

@@ -0,0 +1,128 @@
+{
+  "onderwerp": [
+    "Advies",
+    "Algemene veiligheid",
+    "Begeleiding",
+    "Begeleiding bij- en naar werk",
+    "Bereikbaarheid afvalcontainers",
+    "Bereikbaarheid/beschikbaarheid parkeerplek",
+    "Betalingsregeling",
+    "Bijstand",
+    "Bouwoverlast",
+    "Bruikbaarheid/beschikbaarheid afvalcontainers",
+    "Burgemeester",
+    "Buurt ambiance",
+    "COVID-19",
+    "Criminaliteit",
+    "Documentaanvraag",
+    "Echtscheiding",
+    "Energiekosten",
+    "Evenementen, feestdagen en herdenkingen",
+    "Financiële regelingen",
+    "Gebedsoproepen",
+    "Geboorte/erkenning",
+    "Geluidsoverlast",
+    "Gemeentelijke heffingen",
+    "Geuroverlast",
+    "Hangjongeren",
+    "Huisdierenoverlast",
+    "Huishoudelijke hulp",
+    "Hulp aan dak- en thuislozen",
+    "Hulpmiddelen en woonvoorzieningen",
+    "Huur en verhuur",
+    "Huwelijk/geregistreerd partnerschap",
+    "Inbraak",
+    "Infrastructuur",
+    "Koop en verkoop",
+    "Kwijtschelding",
+    "Locatie: A20 richting Utrecht",
+    "Locatie: Achterzijde Museum Boijmans",
+    "Locatie: Afvalcontainer Schiedamseweg",
+    "Locatie: Appartement Blaak",
+    "Locatie: Appartement Oude Binnenweg",
+    "Locatie: Beursplein uitgang parkeergarage",
+    "Locatie: Bouwmarkt Alexandrium",
+    "Locatie: Bouwmarkt Zuidplein",
+    "Locatie: Eendrachtsplein",
+    "Locatie: Erasmusbrug",
+    "Locatie: Gemeente Rotterdam - Stadhuis",
+    "Locatie: Grensovergang Hazeldonk",
+    "Locatie: Maastunnel ingang noord",
+    "Locatie: Marathon route - Coolsingel",
+    "Locatie: Museum - Achteringang",
+    "Locatie: Museumpark - Meerdere locaties",
+    "Locatie: Oldegaarde 45-89",
+    "Locatie: Oldegaarde complex",
+    "Locatie: Parkeergarage Beursplein",
+    "Locatie: Politiebureau Boezemsingel",
+    "Locatie: Station Rotterdam Centraal",
+    "Locatie: Tankstation Kleinpolderplein",
+    "Locatie: Verhuurbedrijf Rotterdam Zuid",
+    "Locatie: Voor Oldegaarde complex",
+    "Locatie: Waalhaven Zuid - Industrieterrein",
+    "Locatie: Westersingel - Tegenover museum",
+    "Locatie: Westersingel/Museumpark",
+    "Luchtkwaliteit",
+    "Migratie",
+    "Milieupark",
+    "No subtopic found",
+    "Omgevingsvergunning",
+    "Onderhoud omgeving",
+    "Ophalen huiscontainers/grofvuil",
+    "Overlast personen",
+    "Overlijden",
+    "Parkeerapp",
+    "Parkeerboetes",
+    "Parkeergarage",
+    "Parkeerkosten",
+    "Parkeeroverlast",
+    "Parkeerterminal",
+    "Parkeervergunning/abonnement",
+    "Parkeren bezoekers",
+    "Rotterdampas",
+    "Schade en claims",
+    "Urgent - directe actie nodig",
+    "Vaccinatie",
+    "Verdacht voertuig",
+    "Verdachte situatie",
+    "Verhuizen",
+    "Verkeersmaatregelen",
+    "Verkeersovertreding",
+    "Verkeersveiligheid",
+    "Verlichting",
+    "Verloren voorwerpen",
+    "Vervoer",
+    "Vervoersorganisaties",
+    "Vuil/ongedierte overlast",
+    "Wijkteam",
+    "Wijzigen officiële gegevens"
+  ],
+  "beleving": [
+    "Afspraakmogelijkheden",
+    "Algemene ervaring",
+    "Behulpzaamheid",
+    "Bereikbaarheid van medewerker",
+    "Bezwaar & bewijs",
+    "Communicatie",
+    "Duidelijkheid",
+    "Efficiëntie van het proces",
+    "Functionaliteiten web & app",
+    "Gebruiksgemak web & app",
+    "Gemak van het proces",
+    "Informatievoorziening web & app",
+    "Integriteit & Afspraken nakomen",
+    "Juiste persoon te spreken krijgen",
+    "Juistheid van afhandeling",
+    "Kwaliteit v/d informatie",
+    "Op de hoogte houden",
+    "Oprechte interesse",
+    "Persoonlijk",
+    "Snelheid van afhandeling",
+    "Statusinformatie",
+    "Vragen student",
+    "Vriendelijkheid",
+    "Wachttijd",
+    "ongerust",
+    "verdacht"
+  ]
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7a0c1d139dedea006dc8c862788a2ed33601dc7c2edcf1729ea2372bc1ea33f
+size 1227771776

requirements.txt

@@ -0,0 +1,93 @@
+absl-py==2.3.1
+accelerate==1.10.1
+aiohappyeyeballs==2.6.1
+aiohttp==3.13.1
+aiosignal==1.4.0
+annotated-types==0.7.0
+anyio==4.11.0
+attrs==25.4.0
+certifi==2025.10.5
+charset-normalizer==3.4.4
+click==8.3.0
+datasets==4.2.0
+dill==0.4.0
+filelock==3.19.1
+frozenlist==1.8.0
+fsspec==2025.9.0
+gitdb==4.0.12
+GitPython==3.1.45
+grpcio==1.75.1
+h11==0.16.0
+hf-xet==1.1.10
+httpcore==1.0.9
+httpx==0.28.1
+huggingface-hub==0.35.3
+idna==3.11
+Jinja2==3.1.6
+Markdown==3.9
+MarkupSafe==2.1.5
+mpmath==1.3.0
+multidict==6.7.0
+multiprocess==0.70.16
+networkx==3.5
+numpy==2.3.4
+nvidia-cublas-cu12==12.8.4.1
+nvidia-cuda-cupti-cu12==12.8.90
+nvidia-cuda-nvrtc-cu12==12.8.93
+nvidia-cuda-runtime-cu12==12.8.90
+nvidia-cudnn-cu12==9.10.2.21
+nvidia-cufft-cu12==11.3.3.83
+nvidia-cufile-cu12==1.13.1.3
+nvidia-curand-cu12==10.3.9.90
+nvidia-cusolver-cu12==11.7.3.90
+nvidia-cusparse-cu12==12.5.8.93
+nvidia-cusparselt-cu12==0.7.1
+nvidia-ml-py==13.580.82
+nvidia-nccl-cu12==2.27.5
+nvidia-nvjitlink-cu12==12.8.93
+nvidia-nvshmem-cu12==3.3.20
+nvidia-nvtx-cu12==12.8.90
+nvitop==1.5.3
+ollama==0.6.0
+packaging==25.0
+pandas==2.3.3
+peft==0.17.1
+pillow==12.0.0
+platformdirs==4.5.0
+propcache==0.4.1
+protobuf==6.33.0
+psutil==7.1.1
+pyarrow==21.0.0
+pydantic==2.12.3
+pydantic_core==2.41.4
+pyparsing==3.2.5
+python-dateutil==2.9.0.post0
+pytz==2025.2
+PyYAML==6.0.3
+rdflib==7.2.1
+regex==2025.9.18
+requests==2.32.5
+safetensors==0.6.2
+sentry-sdk==2.42.1
+setuptools==70.2.0
+six==1.17.0
+smmap==5.0.2
+sniffio==1.3.1
+sympy==1.14.0
+tensorboard==2.20.0
+tensorboard-data-server==0.7.2
+termcolor==3.1.0
+tokenizers==0.22.1
+torch==2.9.0+cu128
+tqdm==4.67.1
+transformers==4.57.1
+triton==3.5.0
+trl==0.24.0
+typing-inspection==0.4.2
+typing_extensions==4.15.0
+tzdata==2025.2
+urllib3==2.5.0
+wandb==0.22.2
+Werkzeug==3.1.3
+xxhash==3.6.0
+yarl==1.22.0

special_tokens_map.json

@@ -0,0 +1,55 @@
+{
+  "additional_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "bos_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<bos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "<eos>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5e0d8cfefee1eb9d19b4a6f301d5ae937cb866ff3d9a3af101595aab000c627
+size 34363455

tokenizer_config.json

@@ -0,0 +1,2018 @@
+{
+  "add_bos_token": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<eos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<bos>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<2mass>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "6": {
+      "content": "[@BOS@]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "7": {
+      "content": "<unused0>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "8": {
+      "content": "<unused1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "9": {
+      "content": "<unused2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "10": {
+      "content": "<unused3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "11": {
+      "content": "<unused4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "12": {
+      "content": "<unused5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "13": {
+      "content": "<unused6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "14": {
+      "content": "<unused7>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "15": {
+      "content": "<unused8>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "16": {
+      "content": "<unused9>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "17": {
+      "content": "<unused10>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "18": {
+      "content": "<unused11>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "19": {
+      "content": "<unused12>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "20": {
+      "content": "<unused13>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "21": {
+      "content": "<unused14>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "22": {
+      "content": "<unused15>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "23": {
+      "content": "<unused16>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "24": {
+      "content": "<unused17>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "25": {
+      "content": "<unused18>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "26": {
+      "content": "<unused19>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "27": {
+      "content": "<unused20>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "28": {
+      "content": "<unused21>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "29": {
+      "content": "<unused22>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "30": {
+      "content": "<unused23>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "31": {
+      "content": "<unused24>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "32": {
+      "content": "<unused25>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "33": {
+      "content": "<unused26>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "34": {
+      "content": "<unused27>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "35": {
+      "content": "<unused28>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "36": {
+      "content": "<unused29>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "37": {
+      "content": "<unused30>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "38": {
+      "content": "<unused31>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "39": {
+      "content": "<unused32>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "40": {
+      "content": "<unused33>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "41": {
+      "content": "<unused34>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "42": {
+      "content": "<unused35>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "43": {
+      "content": "<unused36>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "44": {
+      "content": "<unused37>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "45": {
+      "content": "<unused38>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "46": {
+      "content": "<unused39>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "47": {
+      "content": "<unused40>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "48": {
+      "content": "<unused41>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "49": {
+      "content": "<unused42>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50": {
+      "content": "<unused43>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "51": {
+      "content": "<unused44>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "52": {
+      "content": "<unused45>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "53": {
+      "content": "<unused46>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "54": {
+      "content": "<unused47>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "55": {
+      "content": "<unused48>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "56": {
+      "content": "<unused49>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "57": {
+      "content": "<unused50>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "58": {
+      "content": "<unused51>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "59": {
+      "content": "<unused52>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "60": {
+      "content": "<unused53>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "61": {
+      "content": "<unused54>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "62": {
+      "content": "<unused55>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "63": {
+      "content": "<unused56>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "64": {
+      "content": "<unused57>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "65": {
+      "content": "<unused58>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "66": {
+      "content": "<unused59>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "67": {
+      "content": "<unused60>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "68": {
+      "content": "<unused61>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "69": {
+      "content": "<unused62>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "70": {
+      "content": "<unused63>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "71": {
+      "content": "<unused64>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "72": {
+      "content": "<unused65>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "73": {
+      "content": "<unused66>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "74": {
+      "content": "<unused67>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "75": {
+      "content": "<unused68>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "76": {
+      "content": "<unused69>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "77": {
+      "content": "<unused70>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "78": {
+      "content": "<unused71>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "79": {
+      "content": "<unused72>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "80": {
+      "content": "<unused73>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "81": {
+      "content": "<unused74>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "82": {
+      "content": "<unused75>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "83": {
+      "content": "<unused76>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "84": {
+      "content": "<unused77>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "85": {
+      "content": "<unused78>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "86": {
+      "content": "<unused79>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "87": {
+      "content": "<unused80>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "88": {
+      "content": "<unused81>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "89": {
+      "content": "<unused82>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "90": {
+      "content": "<unused83>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "91": {
+      "content": "<unused84>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "92": {
+      "content": "<unused85>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "93": {
+      "content": "<unused86>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "94": {
+      "content": "<unused87>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "95": {
+      "content": "<unused88>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "96": {
+      "content": "<unused89>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "97": {
+      "content": "<unused90>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "98": {
+      "content": "<unused91>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "99": {
+      "content": "<unused92>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "100": {
+      "content": "<unused93>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "101": {
+      "content": "<unused94>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "102": {
+      "content": "<unused95>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "103": {
+      "content": "<unused96>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "104": {
+      "content": "<unused97>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "105": {
+      "content": "<unused98>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "106": {
+      "content": "<start_of_turn>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "107": {
+      "content": "<end_of_turn>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "108": {
+      "content": "\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "109": {
+      "content": "\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "110": {
+      "content": "\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "111": {
+      "content": "\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "112": {
+      "content": "\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "113": {
+      "content": "\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "114": {
+      "content": "\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "115": {
+      "content": "\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "116": {
+      "content": "\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "117": {
+      "content": "\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "118": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "119": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "120": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "121": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "122": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "123": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "124": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "125": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "126": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "127": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "129": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "130": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "131": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "132": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "133": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "134": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "135": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "136": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "137": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "138": {
+      "content": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "139": {
+      "content": "▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "140": {
+      "content": "▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "141": {
+      "content": "▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "142": {
+      "content": "▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "143": {
+      "content": "▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "144": {
+      "content": "▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "145": {
+      "content": "▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "146": {
+      "content": "▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "147": {
+      "content": "▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "148": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "149": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "150": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "152": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "153": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "154": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "155": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "156": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "157": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "158": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "159": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "160": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "161": {
+      "content": "▁▁▁���▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "162": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "163": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "164": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "165": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "166": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "167": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "168": {
+      "content": "▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "169": {
+      "content": "<table>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "170": {
+      "content": "<caption>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "171": {
+      "content": "<thead>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "172": {
+      "content": "<tbody>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "173": {
+      "content": "<tfoot>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "174": {
+      "content": "<tr>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "175": {
+      "content": "<th>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "176": {
+      "content": "<td>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "177": {
+      "content": "</table>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "178": {
+      "content": "</caption>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "179": {
+      "content": "</thead>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "180": {
+      "content": "</tbody>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "181": {
+      "content": "</tfoot>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "182": {
+      "content": "</tr>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "183": {
+      "content": "</th>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "184": {
+      "content": "</td>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "185": {
+      "content": "<h1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "186": {
+      "content": "<h2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "187": {
+      "content": "<h3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "188": {
+      "content": "<h4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "189": {
+      "content": "<h5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "190": {
+      "content": "<h6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "191": {
+      "content": "<blockquote>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "192": {
+      "content": "</h1>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "193": {
+      "content": "</h2>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "194": {
+      "content": "</h3>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "195": {
+      "content": "</h4>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "196": {
+      "content": "</h5>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "197": {
+      "content": "</h6>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "198": {
+      "content": "</blockquote>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "199": {
+      "content": "<strong>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "200": {
+      "content": "<em>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "201": {
+      "content": "<b>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "202": {
+      "content": "<i>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "203": {
+      "content": "<u>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "204": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "205": {
+      "content": "<sub>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "206": {
+      "content": "<sup>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "207": {
+      "content": "<code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "208": {
+      "content": "</strong>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "209": {
+      "content": "</em>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "210": {
+      "content": "</b>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "211": {
+      "content": "</i>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "212": {
+      "content": "</u>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "213": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "214": {
+      "content": "</sub>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "215": {
+      "content": "</sup>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "216": {
+      "content": "</code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255968": {
+      "content": "[toxicity=0]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255969": {
+      "content": "\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255970": {
+      "content": "\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255971": {
+      "content": "\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255972": {
+      "content": "\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255973": {
+      "content": "\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255974": {
+      "content": "\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255975": {
+      "content": "\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255976": {
+      "content": "\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255977": {
+      "content": "\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255978": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255979": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255980": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255981": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255982": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255983": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255984": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255985": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255986": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255987": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255988": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255989": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255990": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255991": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255992": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255993": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255994": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255995": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255996": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255997": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255998": {
+      "content": "\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "255999": {
+      "content": "<unused99>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "<start_of_turn>",
+    "<end_of_turn>"
+  ],
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<bos>",
+  "eos_token": "<eos>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "<pad>",
+  "padding_side": "right",
+  "sep_token": "<eos>",
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<unk>"
+}

train/rd_dataset_loader.py

@@ -0,0 +1,138 @@
+#!/usr/bin/env python3
+"""
+UWV/wim-synthetic-data-rd dataset loader for multi-label classification.
+Carmack-style: minimal abstraction, direct data flow, fast operations.
+"""
+
+import numpy as np
+from datasets import load_dataset
+
+
+def load_rd_wim_dataset(max_samples=None, split='train'):
+    """
+    Load UWV/wim-synthetic-data-rd dataset and encode multi-labels.
+
+    Dataset contains Dutch municipal complaint conversations with two types of labels:
+    - onderwerp: What the message is about (96 unique labels)
+    - beleving: How the citizen experienced the interaction (26 unique labels)
+
+    Args:
+        max_samples: Limit number of samples (None = all 9,351 samples)
+        split: Dataset split to load (default: 'train')
+
+    Returns:
+        texts: List of conversation strings
+        onderwerp_encoded: numpy array [n_samples, 96] - multi-hot encoded topics
+        beleving_encoded: numpy array [n_samples, 26] - multi-hot encoded experiences
+        onderwerp_labels: List of 96 onderwerp label names (sorted alphabetically)
+        beleving_labels: List of 26 beleving label names (sorted alphabetically)
+    """
+
+    # Load dataset from HuggingFace
+    print(f"Loading UWV/wim-synthetic-data-rd dataset (split={split})...")
+    ds = load_dataset('UWV/wim-synthetic-data-rd', split=split)
+
+    # Limit samples if requested
+    if max_samples is not None:
+        ds = ds.select(range(min(max_samples, len(ds))))
+
+    print(f"Loaded {len(ds)} samples")
+
+    # Extract all unique labels from the entire dataset
+    onderwerp_set = set()
+    beleving_set = set()
+
+    for sample in ds:
+        for label in sample['onderwerp_labels']:
+            onderwerp_set.add(label)
+        for label in sample['beleving_labels']:
+            beleving_set.add(label)
+
+    # Sort labels alphabetically for consistent indexing across runs
+    onderwerp_labels = sorted(onderwerp_set)
+    beleving_labels = sorted(beleving_set)
+
+    print(f"Found {len(onderwerp_labels)} unique onderwerp labels")
+    print(f"Found {len(beleving_labels)} unique beleving labels")
+
+    # Create label -> index mappings
+    onderwerp_to_idx = {label: idx for idx, label in enumerate(onderwerp_labels)}
+    beleving_to_idx = {label: idx for idx, label in enumerate(beleving_labels)}
+
+    # Encode labels to multi-hot vectors
+    n_samples = len(ds)
+    n_onderwerp = len(onderwerp_labels)
+    n_beleving = len(beleving_labels)
+
+    # Preallocate arrays (faster than appending)
+    texts = []
+    onderwerp_encoded = np.zeros((n_samples, n_onderwerp), dtype=np.float32)
+    beleving_encoded = np.zeros((n_samples, n_beleving), dtype=np.float32)
+
+    # Fill arrays
+    for i, sample in enumerate(ds):
+        texts.append(sample['text'])
+
+        # Encode onderwerp labels (multi-hot)
+        for label in sample['onderwerp_labels']:
+            idx = onderwerp_to_idx[label]
+            onderwerp_encoded[i, idx] = 1.0
+
+        # Encode beleving labels (multi-hot)
+        for label in sample['beleving_labels']:
+            idx = beleving_to_idx[label]
+            beleving_encoded[i, idx] = 1.0
+
+    print(f"Encoded {n_samples} samples")
+    print(f"  onderwerp shape: {onderwerp_encoded.shape}")
+    print(f"  beleving shape: {beleving_encoded.shape}")
+
+    return texts, onderwerp_encoded, beleving_encoded, onderwerp_labels, beleving_labels
+
+
+def print_sample_info(texts, onderwerp_encoded, beleving_encoded,
+                     onderwerp_labels, beleving_labels, sample_idx=0):
+    """
+    Print information about a specific sample (useful for debugging).
+
+    Args:
+        All outputs from load_uwv_wim_dataset()
+        sample_idx: Which sample to print (default: 0)
+    """
+    print(f"\n{'='*60}")
+    print(f"SAMPLE {sample_idx}")
+    print(f"{'='*60}")
+    print(f"Text: {texts[sample_idx][:200]}...")
+    print()
+
+    # Get active onderwerp labels
+    onderwerp_active = [onderwerp_labels[i] for i, val in enumerate(onderwerp_encoded[sample_idx]) if val == 1]
+    print(f"Onderwerp labels ({len(onderwerp_active)}):")
+    for label in onderwerp_active:
+        print(f"  - {label}")
+    print()
+
+    # Get active beleving labels
+    beleving_active = [beleving_labels[i] for i, val in enumerate(beleving_encoded[sample_idx]) if val == 1]
+    print(f"Beleving labels ({len(beleving_active)}):")
+    for label in beleving_active:
+        print(f"  - {label}")
+    print(f"{'='*60}\n")
+
+
+if __name__ == "__main__":
+    # Test the loader
+    print("Testing UWV dataset loader...\n")
+
+    # Load small subset for testing
+    texts, onderwerp, beleving, onderwerp_names, beleving_names = load_rd_wim_dataset(max_samples=10)
+
+    # Print first sample
+    print_sample_info(texts, onderwerp, beleving, onderwerp_names, beleving_names, sample_idx=0)
+
+    # Print statistics
+    print("\nDataset Statistics:")
+    print(f"  Total samples: {len(texts)}")
+    print(f"  Avg onderwerp labels per sample: {onderwerp.sum(axis=1).mean():.2f}")
+    print(f"  Avg beleving labels per sample: {beleving.sum(axis=1).mean():.2f}")
+    print(f"  Text length range: {min(len(t) for t in texts)} - {max(len(t) for t in texts)} chars")

train/train_mmbert_dual_soft_f1_simplified.py

@@ -0,0 +1,953 @@
+#!/usr/bin/env python3
+"""
+Dual-head multi-label PyTorch training script for mmBERT-base.
+Two classification heads: onderwerp (96 labels) and beleving (26 labels).
+Uses combined F1+BCE loss with weight α (configurable balance).
+Features: learnable thresholds, warmup + cosine LR, gradient clipping.
+mmBERT: Modern multilingual encoder (1800+ languages, 2x faster than XLM-R).
+"""
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.utils.data import Dataset, DataLoader
+from torch.optim.lr_scheduler import LinearLR, CosineAnnealingLR, SequentialLR
+from transformers import AutoTokenizer, AutoModel
+import os
+import json
+import numpy as np
+import random
+import wandb
+from rd_dataset_loader import load_rd_wim_dataset
+
+
+# Threshold helpers: logit ↔ probability conversions
+def prob_to_logit(p: torch.Tensor, eps: float = 1e-7) -> torch.Tensor:
+    """Convert probabilities to logits (inverse sigmoid). Numerically stable."""
+    p = torch.clamp(p, eps, 1 - eps)
+    return torch.log(p / (1 - p))
+
+
+def logit_to_prob(l: torch.Tensor) -> torch.Tensor:
+    """Convert logits to probabilities using sigmoid."""
+    return torch.sigmoid(l)
+
+
+# Set device - MPS for Apple Silicon, fallback to CPU
+def get_device():
+    if torch.backends.mps.is_available():
+        device = torch.device("mps")
+        print("Using MPS (Apple Silicon) for acceleration")
+    elif torch.cuda.is_available():
+        device = torch.device("cuda")
+        print("Using CUDA GPU")
+    else:
+        device = torch.device("cpu")
+        print("Using CPU")
+    return device
+
+
+def set_seed(seed):
+    """Set random seeds for reproducibility across torch, numpy, and Python random."""
+    torch.manual_seed(seed)
+    np.random.seed(seed)
+    random.seed(seed)
+    if torch.cuda.is_available():
+        torch.cuda.manual_seed_all(seed)
+
+
+class mmBERTDualHead(nn.Module):
+    """
+    mmBERT with two classification heads for multi-task learning.
+    Shared encoder with separate heads for onderwerp and beleving.
+    Optionally includes learnable thresholds for each head.
+    """
+    def __init__(self, model_name, num_onderwerp, num_beleving, dropout, initial_threshold, use_thresholds: bool = True):
+        super().__init__()
+        self.use_thresholds = use_thresholds
+
+        # Shared mmBERT encoder (22 layers, 768 hidden, supports up to 8192 tokens)
+        self.encoder = AutoModel.from_pretrained(model_name)
+        hidden_size = self.encoder.config.hidden_size  # 768 for mmBERT-base
+
+        # Classification head for onderwerp (topics)
+        self.onderwerp_head = nn.Sequential(
+            nn.Linear(hidden_size, hidden_size),
+            nn.Dropout(dropout),
+            nn.ReLU(),
+            nn.Linear(hidden_size, num_onderwerp)
+        )
+
+        # Classification head for beleving (experiences)
+        self.beleving_head = nn.Sequential(
+            nn.Linear(hidden_size, hidden_size),
+            nn.Dropout(dropout),
+            nn.ReLU(),
+            nn.Linear(hidden_size, num_beleving)
+        )
+
+        # Thresholds are optionally parameterized in **logit space** (tau_logit).
+        # Why: (1) avoids prob clamping and keeps grads healthy, (2) matches the space of logits,
+        # (3) lets Soft-F1 express per-class decision boundaries independent of BCE calibration.
+        self.onderwerp_tau_logit = None
+        self.beleving_tau_logit = None
+        if self.use_thresholds:
+            init_logit = prob_to_logit(torch.tensor(initial_threshold))
+            self.onderwerp_tau_logit = nn.Parameter(torch.full((num_onderwerp,), init_logit))
+            self.beleving_tau_logit = nn.Parameter(torch.full((num_beleving,), init_logit))
+
+    def forward(self, input_ids, attention_mask):
+        # Get shared representation from mmBERT encoder
+        outputs = self.encoder(
+            input_ids=input_ids,
+            attention_mask=attention_mask
+        )
+
+        # mmBERT doesn't have pooler_output, use CLS token from last_hidden_state
+        # Extract [CLS] token representation (first token in sequence)
+        pooled_output = outputs.last_hidden_state[:, 0, :]
+
+        # Generate predictions from both heads
+        onderwerp_logits = self.onderwerp_head(pooled_output)
+        beleving_logits = self.beleving_head(pooled_output)
+
+        return onderwerp_logits, beleving_logits
+
+
+class DutchDualLabelDataset(Dataset):
+    """Dataset for dual-label classification (onderwerp + beleving)."""
+
+    def __init__(self, texts, onderwerp_labels, beleving_labels, tokenizer, max_length):
+        self.texts = texts
+        self.onderwerp_labels = onderwerp_labels
+        self.beleving_labels = beleving_labels
+        self.tokenizer = tokenizer
+        self.max_length = max_length
+
+    def __len__(self):
+        return len(self.texts)
+
+    def __getitem__(self, idx):
+        text = self.texts[idx]
+
+        # Tokenize text
+        encoding = self.tokenizer(
+            text,
+            truncation=True,
+            padding='max_length',
+            max_length=self.max_length,
+            return_tensors='pt'
+        )
+
+        return {
+            'input_ids': encoding['input_ids'].squeeze(),
+            'attention_mask': encoding['attention_mask'].squeeze(),
+            'onderwerp_labels': torch.tensor(self.onderwerp_labels[idx], dtype=torch.float),
+            'beleving_labels': torch.tensor(self.beleving_labels[idx], dtype=torch.float)
+        }
+
+
+def calculate_soft_f1(logits, labels, logit_threshold=None, temperature=1.0):
+    """
+    Calculate differentiable F1 score using sigmoid approximation.
+
+    If logit_threshold is None: y_soft = sigmoid(logits * T)
+    Else:                       y_soft = sigmoid((logits - logit_threshold) * T)
+
+    Rationale:
+      - With thresholds ON, Soft-F1 learns per-class decision boundaries in logit space.
+      - With thresholds OFF, we follow POLA: a single, obvious source (head logits).
+
+    Args:
+        logits: Model predictions (before sigmoid)
+        labels: True labels (multi-hot encoded)
+        logit_threshold: Optional decision threshold in LOGIT space (None = no shift)
+        temperature: Sharpness of sigmoid approximation
+
+    Returns:
+        soft_f1: Differentiable F1 score
+    """
+    # Compute shifted logits (or raw logits if threshold is None)
+    if logit_threshold is None:
+        shifted = logits * temperature
+    else:
+        shifted = (logits - logit_threshold) * temperature
+
+    # Soft predictions using sigmoid
+    y_pred_soft = torch.sigmoid(shifted)
+
+    # Soft confusion matrix elements
+    TP = (y_pred_soft * labels).sum(dim=-1)  # True Positives
+    FP = (y_pred_soft * (1 - labels)).sum(dim=-1)  # False Positives
+    FN = ((1 - y_pred_soft) * labels).sum(dim=-1)  # False Negatives
+
+    # Differentiable F1 score
+    eps = 1e-8
+    precision = TP / (TP + FP + eps)
+    recall = TP / (TP + FN + eps)
+    f1 = 2 * precision * recall / (precision + recall + eps)
+
+    return f1.mean()  # Average across batch
+
+
+def evaluate(model, val_texts, val_onderwerp, val_beleving, tokenizer, device,
+             onderwerp_names, beleving_names, num_samples, max_length):
+    """
+    Evaluate model on validation set and return metrics.
+
+    Args:
+        model: The trained model
+        val_texts: List of validation texts
+        val_onderwerp: Validation onderwerp labels
+        val_beleving: Validation beleving labels
+        tokenizer: Tokenizer for encoding text
+        device: Device to run evaluation on
+        onderwerp_names: List of onderwerp label names
+        beleving_names: List of beleving label names
+        num_samples: Number of samples to evaluate (None = all)
+        max_length: Max sequence length
+
+    Returns:
+        dict: Dictionary containing all evaluation metrics
+    """
+    model.eval()
+
+    # Determine number of samples to evaluate
+    if num_samples is None:
+        num_samples = len(val_texts)
+    else:
+        num_samples = min(num_samples, len(val_texts))
+
+    # Track metrics
+    onderwerp_correct = np.zeros(len(onderwerp_names))
+    onderwerp_total = np.zeros(len(onderwerp_names))
+    beleving_correct = np.zeros(len(beleving_names))
+    beleving_total = np.zeros(len(beleving_names))
+
+    # Track F1 components
+    onderwerp_tp = 0
+    onderwerp_fp = 0
+    onderwerp_fn = 0
+    beleving_tp = 0
+    beleving_fp = 0
+    beleving_fn = 0
+
+    with torch.inference_mode():
+        for i in range(num_samples):
+            # Tokenize
+            encoding = tokenizer(
+                val_texts[i],
+                truncation=True,
+                padding='max_length',
+                max_length=max_length,
+                return_tensors='pt'
+            )
+
+            # Move to device
+            input_ids = encoding['input_ids'].to(device)
+            attention_mask = encoding['attention_mask'].to(device)
+
+            # Get predictions
+            onderwerp_logits, beleving_logits = model(input_ids, attention_mask)
+
+            # Convert to probabilities
+            onderwerp_probs = torch.sigmoid(onderwerp_logits)
+            beleving_probs = torch.sigmoid(beleving_logits)
+
+            # Apply learned per-class thresholds (if enabled) or fixed 0.5 cutoff
+            if model.use_thresholds:
+                tau_on = logit_to_prob(model.onderwerp_tau_logit)  # [C1]
+                tau_be = logit_to_prob(model.beleving_tau_logit)   # [C2]
+            else:
+                # Fixed probability cutoff (POLA-friendly)
+                tau_on = torch.full_like(onderwerp_probs[0], 0.5)
+                tau_be = torch.full_like(beleving_probs[0], 0.5)
+
+            onderwerp_pred = (onderwerp_probs > tau_on).squeeze().cpu().numpy()
+            beleving_pred = (beleving_probs > tau_be).squeeze().cpu().numpy()
+
+            # Get true labels
+            onderwerp_true = val_onderwerp[i]
+            beleving_true = val_beleving[i]
+
+            # Update F1 components
+            onderwerp_tp += ((onderwerp_pred == 1) & (onderwerp_true == 1)).sum()
+            onderwerp_fp += ((onderwerp_pred == 1) & (onderwerp_true == 0)).sum()
+            onderwerp_fn += ((onderwerp_pred == 0) & (onderwerp_true == 1)).sum()
+
+            beleving_tp += ((beleving_pred == 1) & (beleving_true == 1)).sum()
+            beleving_fp += ((beleving_pred == 1) & (beleving_true == 0)).sum()
+            beleving_fn += ((beleving_pred == 0) & (beleving_true == 1)).sum()
+
+            # Update accuracy metrics
+            for j in range(len(onderwerp_names)):
+                if onderwerp_pred[j] == onderwerp_true[j]:
+                    onderwerp_correct[j] += 1
+                onderwerp_total[j] += 1
+
+            for j in range(len(beleving_names)):
+                if beleving_pred[j] == beleving_true[j]:
+                    beleving_correct[j] += 1
+                beleving_total[j] += 1
+
+    # Calculate F1 scores
+    epsilon = 1e-8
+    onderwerp_precision = onderwerp_tp / (onderwerp_tp + onderwerp_fp + epsilon)
+    onderwerp_recall = onderwerp_tp / (onderwerp_tp + onderwerp_fn + epsilon)
+    onderwerp_f1_score = 2 * onderwerp_precision * onderwerp_recall / (onderwerp_precision + onderwerp_recall + epsilon)
+
+    beleving_precision = beleving_tp / (beleving_tp + beleving_fp + epsilon)
+    beleving_recall = beleving_tp / (beleving_tp + beleving_fn + epsilon)
+    beleving_f1_score = 2 * beleving_precision * beleving_recall / (beleving_precision + beleving_recall + epsilon)
+
+    # Calculate accuracies
+    onderwerp_acc = onderwerp_correct.sum() / onderwerp_total.sum()
+    beleving_acc = beleving_correct.sum() / beleving_total.sum()
+
+    # Get threshold statistics (convert to probability space for human readability)
+    if model.use_thresholds:
+        onderwerp_thresh_mean = logit_to_prob(model.onderwerp_tau_logit).mean().item()
+        onderwerp_thresh_min = logit_to_prob(model.onderwerp_tau_logit).min().item()
+        onderwerp_thresh_max = logit_to_prob(model.onderwerp_tau_logit).max().item()
+        onderwerp_thresh_std = logit_to_prob(model.onderwerp_tau_logit).std().item()
+        beleving_thresh_mean = logit_to_prob(model.beleving_tau_logit).mean().item()
+        beleving_thresh_min = logit_to_prob(model.beleving_tau_logit).min().item()
+        beleving_thresh_max = logit_to_prob(model.beleving_tau_logit).max().item()
+        beleving_thresh_std = logit_to_prob(model.beleving_tau_logit).std().item()
+    else:
+        # Fixed threshold values
+        onderwerp_thresh_mean = onderwerp_thresh_min = onderwerp_thresh_max = onderwerp_thresh_std = 0.5
+        beleving_thresh_mean = beleving_thresh_min = beleving_thresh_max = beleving_thresh_std = 0.5
+
+    # Return metrics dictionary
+    return {
+        'onderwerp_acc': onderwerp_acc,
+        'onderwerp_precision': onderwerp_precision,
+        'onderwerp_recall': onderwerp_recall,
+        'onderwerp_f1': onderwerp_f1_score,
+        'beleving_acc': beleving_acc,
+        'beleving_precision': beleving_precision,
+        'beleving_recall': beleving_recall,
+        'beleving_f1': beleving_f1_score,
+        'combined_acc': (onderwerp_acc + beleving_acc) / 2,
+        'combined_f1': (onderwerp_f1_score + beleving_f1_score) / 2,
+        'onderwerp_thresh_mean': onderwerp_thresh_mean,
+        'onderwerp_thresh_min': onderwerp_thresh_min,
+        'onderwerp_thresh_max': onderwerp_thresh_max,
+        'onderwerp_thresh_std': onderwerp_thresh_std,
+        'beleving_thresh_mean': beleving_thresh_mean,
+        'beleving_thresh_min': beleving_thresh_min,
+        'beleving_thresh_max': beleving_thresh_max,
+        'beleving_thresh_std': beleving_thresh_std,
+        'num_samples_evaluated': num_samples
+    }
+
+
+def grad_l2_norm(params):
+    """
+    Calculate L2 norm of gradients safely (avoids Python int→Tensor addition).
+
+    Args:
+        params: Iterator of parameters (e.g., model.parameters())
+
+    Returns:
+        float: L2 norm of all gradients, or 0.0 if no gradients exist
+    """
+    sq_sum = None
+    for p in params:
+        if p.grad is None:
+            continue
+        g = p.grad
+        val = g.pow(2).sum()
+        sq_sum = val if sq_sum is None else (sq_sum + val)
+    if sq_sum is None:
+        return 0.0
+    return sq_sum.sqrt().item()
+
+
+def make_opt_sched(model, enc_lr, thr_lr, total_steps, warmup_ratio, eta_min):
+    """
+    Create optimizer+scheduler for training.
+    Optimizer has 1-2 param groups: [0]=encoder+heads, [1]=thresholds (optional).
+    """
+    # Group 0: encoder + heads
+    encoder_params = [p for n, p in model.named_parameters()
+                      if not (model.use_thresholds and 'tau_logit' in n)]
+    param_groups = [{"params": encoder_params, "lr": enc_lr, "weight_decay": 0.0}]
+
+    # Group 1 (optional): thresholds
+    if model.use_thresholds:
+        thr_params = [model.onderwerp_tau_logit, model.beleving_tau_logit]
+        param_groups.append({"params": thr_params, "lr": thr_lr, "weight_decay": 0.0})
+
+    optimizer = torch.optim.AdamW(param_groups)
+
+    # Warmup → cosine schedule
+    warmup_steps = min(max(1, int(warmup_ratio * total_steps)), max(1, total_steps - 1))
+    warmup = LinearLR(optimizer, start_factor=1e-10, end_factor=1.0, total_iters=warmup_steps)
+    cosine = CosineAnnealingLR(optimizer, T_max=max(1, total_steps - warmup_steps), eta_min=eta_min)
+    scheduler = SequentialLR(optimizer, [warmup, cosine], milestones=[warmup_steps])
+
+    return optimizer, scheduler
+
+
+def run_epochs(model, tokenizer, train_loader, val_texts, val_onderwerp, val_beleving,
+               onderwerp_names, beleving_names, device,
+               *, start_epoch, end_epoch, phase_name="train",
+               optimizer, scheduler, temperature, alpha,
+               max_length, global_step):
+    """
+    Run training for a range of epochs.
+
+    Args:
+        model: The model to train
+        tokenizer: Tokenizer for text encoding
+        train_loader: DataLoader for training batches
+        val_texts, val_onderwerp, val_beleving: Validation data
+        onderwerp_names, beleving_names: Label names
+        device: Device to train on
+        start_epoch: Starting epoch (inclusive)
+        end_epoch: Ending epoch (exclusive)
+        phase_name: Name for logging (default: "train")
+        optimizer: Optimizer
+        scheduler: LR scheduler
+        temperature: Soft-F1 temperature
+        alpha: Loss weighting (F1 vs BCE)
+        max_length: Max sequence length
+        global_step: Starting global step counter
+
+    Returns:
+        Updated global_step
+    """
+    num_epochs = end_epoch - start_epoch
+    phase_total_steps = max(1, len(train_loader) * num_epochs)
+
+    model.train()
+
+    for epoch in range(start_epoch, end_epoch):
+        total_loss = 0
+        total_onderwerp_f1 = 0
+        total_beleving_f1 = 0
+        total_bce_loss = 0
+        total_f1_loss = 0
+        num_batches = 0
+
+        print(f"\n[{phase_name.upper()}] Epoch {epoch + 1}/{end_epoch}")
+        print("-" * 40)
+
+        for batch_idx, batch in enumerate(train_loader):
+            # Move batch to device
+            input_ids = batch['input_ids'].to(device)
+            attention_mask = batch['attention_mask'].to(device)
+            onderwerp_labels = batch['onderwerp_labels'].to(device)
+            beleving_labels = batch['beleving_labels'].to(device)
+
+            # Zero gradients
+            optimizer.zero_grad()
+
+            # Forward pass
+            onderwerp_logits, beleving_logits = model(input_ids, attention_mask)
+
+            # Calculate Soft-F1 for both heads (conditionally pass thresholds)
+            onderwerp_f1 = calculate_soft_f1(
+                onderwerp_logits, onderwerp_labels,
+                model.onderwerp_tau_logit if model.use_thresholds else None,
+                temperature
+            )
+            beleving_f1 = calculate_soft_f1(
+                beleving_logits, beleving_labels,
+                model.beleving_tau_logit if model.use_thresholds else None,
+                temperature
+            )
+
+            # Calculate BCE loss
+            # Design choice (POLA):
+            # - BCE is computed on raw logits to maintain probability calibration.
+            # - Soft-F1 may use a shifted logit (if thresholds ON) to learn F1-friendly boundaries.
+            # - If thresholds OFF, Soft-F1 acts directly on logits; there is a single "source of truth".
+            # This keeps behavior unsurprising: either (A) calibrated logits + separate boundary learning,
+            # or (B) no extra threshold machinery; F1 and BCE both reference the same logits.
+            bce_onderwerp = F.binary_cross_entropy_with_logits(onderwerp_logits, onderwerp_labels)
+            bce_beleving = F.binary_cross_entropy_with_logits(beleving_logits, beleving_labels)
+
+            # Combined loss
+            f1_loss = (1 - onderwerp_f1) + (1 - beleving_f1)
+            bce_loss = bce_onderwerp + bce_beleving
+            loss = alpha * (f1_loss / 2) + (1 - alpha) * (bce_loss / 2)
+
+            # Periodic logging
+            if batch_idx % 20 == 0:
+                with torch.no_grad():
+                    # Get predictions (convert thresholds from logit-space to prob-space if enabled)
+                    onderwerp_probs = torch.sigmoid(onderwerp_logits)
+                    beleving_probs = torch.sigmoid(beleving_logits)
+                    if model.use_thresholds:
+                        tau_on = logit_to_prob(model.onderwerp_tau_logit)
+                        tau_be = logit_to_prob(model.beleving_tau_logit)
+                    else:
+                        tau_on = torch.full_like(onderwerp_probs[0], 0.5)
+                        tau_be = torch.full_like(beleving_probs[0], 0.5)
+                    onderwerp_pred = (onderwerp_probs > tau_on).float()
+                    beleving_pred = (beleving_probs > tau_be).float()
+
+                    # Log actual optimizer param group LRs
+                    lrs = scheduler.get_last_lr()
+                    encoder_head_lr = lrs[0]   # Param group 0: encoder + heads
+                    threshold_lr = lrs[1] if len(lrs) > 1 else None  # Param group 1: thresholds (optional)
+
+                    # Threshold statistics (convert to probability space for readability)
+                    if model.use_thresholds:
+                        onderwerp_thresh_mean = logit_to_prob(model.onderwerp_tau_logit).mean().item()
+                        onderwerp_thresh_min = logit_to_prob(model.onderwerp_tau_logit).min().item()
+                        onderwerp_thresh_max = logit_to_prob(model.onderwerp_tau_logit).max().item()
+                        beleving_thresh_mean = logit_to_prob(model.beleving_tau_logit).mean().item()
+                        beleving_thresh_min = logit_to_prob(model.beleving_tau_logit).min().item()
+                        beleving_thresh_max = logit_to_prob(model.beleving_tau_logit).max().item()
+                    else:
+                        onderwerp_thresh_mean = onderwerp_thresh_min = onderwerp_thresh_max = 0.5
+                        beleving_thresh_mean = beleving_thresh_min = beleving_thresh_max = 0.5
+
+                    print(f"  Batch {batch_idx + 1} | Step {global_step + 1}/{phase_total_steps}:")
+                    if threshold_lr is not None:
+                        print(f"    Total loss: {loss.item():.4f} (α={alpha} F1 + {1-alpha} BCE) | LR: enc_head={encoder_head_lr:.2e} thresh={threshold_lr:.2e}")
+                    else:
+                        print(f"    Total loss: {loss.item():.4f} (α={alpha} F1 + {1-alpha} BCE) | LR: enc_head={encoder_head_lr:.2e}")
+                    print(f"    F1 loss: {(f1_loss/2).item():.4f} | BCE loss: {(bce_loss/2).item():.4f}")
+                    print(f"    Onderwerp F1: {onderwerp_f1.item():.4f} | BCE: {bce_onderwerp.item():.4f} | Thresh: {onderwerp_thresh_mean:.3f} [{onderwerp_thresh_min:.3f}-{onderwerp_thresh_max:.3f}]")
+                    print(f"    Beleving F1: {beleving_f1.item():.4f} | BCE: {bce_beleving.item():.4f} | Thresh: {beleving_thresh_mean:.3f} [{beleving_thresh_min:.3f}-{beleving_thresh_max:.3f}]")
+                    print(f"    Onderwerp preds: {int(onderwerp_pred.sum())} / {int(onderwerp_labels.sum())} true")
+                    print(f"    Beleving preds: {int(beleving_pred.sum())} / {int(beleving_labels.sum())} true")
+
+                    # Log to wandb
+                    log_dict = {
+                        "phase": phase_name,
+                        "train/loss": loss.item(),
+                        "train/f1_loss": (f1_loss / 2).item(),
+                        "train/bce_loss": (bce_loss / 2).item(),
+                        "train/onderwerp_f1": onderwerp_f1.item(),
+                        "train/onderwerp_bce": bce_onderwerp.item(),
+                        "train/beleving_f1": beleving_f1.item(),
+                        "train/beleving_bce": bce_beleving.item(),
+                        "train/encoder_head_lr": encoder_head_lr,
+                        "train/onderwerp_threshold_mean": onderwerp_thresh_mean,
+                        "train/onderwerp_threshold_min": onderwerp_thresh_min,
+                        "train/onderwerp_threshold_max": onderwerp_thresh_max,
+                        "train/beleving_threshold_mean": beleving_thresh_mean,
+                        "train/beleving_threshold_min": beleving_thresh_min,
+                        "train/beleving_threshold_max": beleving_thresh_max,
+                    }
+                    if threshold_lr is not None:
+                        log_dict["train/threshold_lr"] = threshold_lr
+                    wandb.log(log_dict, step=global_step)
+
+            # Backward pass
+            loss.backward()
+
+            # Calculate gradient norms
+            with torch.no_grad():
+                onderwerp_thresh_grad = (model.onderwerp_tau_logit.grad.abs().mean().item()
+                                         if model.use_thresholds and model.onderwerp_tau_logit.grad is not None else 0.0)
+                beleving_thresh_grad = (model.beleving_tau_logit.grad.abs().mean().item()
+                                        if model.use_thresholds and model.beleving_tau_logit.grad is not None else 0.0)
+
+                encoder_grad_norm = grad_l2_norm(model.encoder.parameters())
+                onderwerp_head_grad_norm = grad_l2_norm(model.onderwerp_head.parameters())
+                beleving_head_grad_norm = grad_l2_norm(model.beleving_head.parameters())
+                global_grad_norm = grad_l2_norm(model.parameters())
+
+            # Log gradient norms
+            wandb.log({
+                "phase": phase_name,
+                "grads/threshold_onderwerp": onderwerp_thresh_grad,
+                "grads/threshold_beleving": beleving_thresh_grad,
+                "grads/encoder": encoder_grad_norm,
+                "grads/onderwerp_head": onderwerp_head_grad_norm,
+                "grads/beleving_head": beleving_head_grad_norm,
+                "grads/global_norm": global_grad_norm,
+            }, step=global_step)
+
+            # Gradient clipping
+            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
+
+            # Update weights and LR
+            optimizer.step()
+            scheduler.step()
+
+            # Update counters
+            global_step += 1
+            total_loss += loss.item()
+            total_onderwerp_f1 += onderwerp_f1.item()
+            total_beleving_f1 += beleving_f1.item()
+            total_f1_loss += (f1_loss / 2).item()
+            total_bce_loss += (bce_loss / 2).item()
+            num_batches += 1
+
+        # Epoch summary
+        avg_loss = total_loss / max(1, num_batches)
+        avg_onderwerp_f1 = total_onderwerp_f1 / max(1, num_batches)
+        avg_beleving_f1 = total_beleving_f1 / max(1, num_batches)
+        avg_f1_loss = total_f1_loss / max(1, num_batches)
+        avg_bce_loss = total_bce_loss / max(1, num_batches)
+
+        # Get current LR for summary
+        lrs = scheduler.get_last_lr()
+        current_lr = lrs[0]  # Display first group LR
+
+        # Threshold statistics (convert to probability space for readability)
+        if model.use_thresholds:
+            onderwerp_thresh_mean = logit_to_prob(model.onderwerp_tau_logit).mean().item()
+            onderwerp_thresh_std = logit_to_prob(model.onderwerp_tau_logit).std().item()
+            beleving_thresh_mean = logit_to_prob(model.beleving_tau_logit).mean().item()
+            beleving_thresh_std = logit_to_prob(model.beleving_tau_logit).std().item()
+        else:
+            onderwerp_thresh_mean = onderwerp_thresh_std = 0.5
+            beleving_thresh_mean = beleving_thresh_std = 0.5
+
+        print(f"\n  [{phase_name.upper()}] Epoch {epoch + 1} Summary:")
+        print(f"    Average total loss: {avg_loss:.4f} (α={alpha} F1 + {1-alpha} BCE)")
+        print(f"    Average F1 loss: {avg_f1_loss:.4f} | Average BCE loss: {avg_bce_loss:.4f}")
+        print(f"    Average onderwerp F1: {avg_onderwerp_f1:.4f} | Threshold: {onderwerp_thresh_mean:.3f} (σ={onderwerp_thresh_std:.3f})")
+        print(f"    Average beleving F1: {avg_beleving_f1:.4f} | Threshold: {beleving_thresh_mean:.3f} (σ={beleving_thresh_std:.3f})")
+        print(f"    Average combined F1: {(avg_onderwerp_f1 + avg_beleving_f1) / 2:.4f}")
+        print(f"    Current learning rate: {current_lr:.2e}")
+
+        # Per-epoch validation
+        print(f"\n  Running validation on 200 samples...")
+        val_metrics = evaluate(
+            model, val_texts, val_onderwerp, val_beleving, tokenizer, device,
+            onderwerp_names, beleving_names, num_samples=200, max_length=max_length
+        )
+
+        # Log validation metrics
+        wandb.log({
+            "phase": phase_name,
+            "val/onderwerp_acc": val_metrics['onderwerp_acc'],
+            "val/onderwerp_precision": val_metrics['onderwerp_precision'],
+            "val/onderwerp_recall": val_metrics['onderwerp_recall'],
+            "val/onderwerp_f1": val_metrics['onderwerp_f1'],
+            "val/beleving_acc": val_metrics['beleving_acc'],
+            "val/beleving_precision": val_metrics['beleving_precision'],
+            "val/beleving_recall": val_metrics['beleving_recall'],
+            "val/beleving_f1": val_metrics['beleving_f1'],
+            "val/combined_acc": val_metrics['combined_acc'],
+            "val/combined_f1": val_metrics['combined_f1'],
+            "val/onderwerp_threshold_mean": val_metrics['onderwerp_thresh_mean'],
+            "val/beleving_threshold_mean": val_metrics['beleving_thresh_mean'],
+            "epoch": epoch + 1
+        }, step=global_step)
+
+        # Log threshold histograms (convert to probability space for readability)
+        if model.use_thresholds:
+            wandb.log({
+                "phase": phase_name,
+                "thresholds/onderwerp": wandb.Histogram(logit_to_prob(model.onderwerp_tau_logit).detach().cpu().numpy()),
+                "thresholds/beleving": wandb.Histogram(logit_to_prob(model.beleving_tau_logit).detach().cpu().numpy()),
+                "epoch": epoch + 1
+            }, step=global_step)
+
+        print(f"    Val onderwerp F1: {val_metrics['onderwerp_f1']:.4f} | Val beleving F1: {val_metrics['beleving_f1']:.4f}")
+        print(f"    Val combined F1: {val_metrics['combined_f1']:.4f}")
+
+        # Return to training mode
+        model.train()
+
+    return global_step
+
+
+def main():
+    # Enable TensorFloat32 for better performance on modern NVIDIA GPUs
+    if torch.cuda.is_available():
+        torch.set_float32_matmul_precision('high')
+
+    # Initialize device
+    device = get_device()
+
+    # ============== CONFIGURATION FOR WANDB SWEEPS ==============
+    # Fixed model configuration (not swept)
+    model_name = "jhu-clsp/mmBERT-base"
+
+    # Sweepable hyperparameters with defaults
+    default_config = dict(
+        # Reproducibility
+        seed=42,
+
+        # Model architecture
+        dropout=0.2,
+        initial_threshold=0.565,
+        max_length=1408,
+
+        # Training switches
+        use_thresholds=False,  # If False: no learnable thresholds; Soft-F1 uses raw logits
+
+        # Training
+        encoder_peak_lr=8e-5,
+        threshold_lr_mult=5.0,  # Threshold LR = encoder_peak_lr * threshold_lr_mult
+        num_epochs=15,
+        batch_size=16,
+
+        # Loss function
+        alpha=0.15,  # Weight for F1 loss in combined loss (0.5 = balanced)
+        temperature=2.0,  # Sigmoid smoothing (lower = softer, higher = sharper)
+
+        # LR schedule
+        warmup_ratio=0.1,  # 10% warmup
+        min_lr=1e-6,
+    )
+
+    # Initialize wandb and get config (allows sweep agent to override defaults)
+    wandb.init(project="wim-multilabel-mmbert", config=default_config)
+    cfg = wandb.config
+
+    # Set seed for reproducibility (before loading data)
+    set_seed(cfg.seed)
+
+    # Load RD dataset
+    print("\nLoading FULL RD dataset (9,351 samples)...")
+    texts, onderwerp, beleving, onderwerp_names, beleving_names = load_rd_wim_dataset(
+        max_samples=None  # Using full dataset for better training
+    )
+
+    print(f"\nDataset loaded:")
+    print(f"  Samples: {len(texts)}")
+    print(f"  Onderwerp labels: {len(onderwerp_names)}")
+    print(f"  Beleving labels: {len(beleving_names)}")
+    print(f"  Avg onderwerp per sample: {onderwerp.sum(axis=1).mean():.2f}")
+    print(f"  Avg beleving per sample: {beleving.sum(axis=1).mean():.2f}")
+
+    # Unpack hyperparameters from wandb.config
+    dropout = cfg.dropout
+    initial_threshold = cfg.initial_threshold
+    max_length = cfg.max_length
+    encoder_peak_lr = cfg.encoder_peak_lr
+    threshold_peak_lr = encoder_peak_lr * cfg.threshold_lr_mult  # Derived from multiplier
+    num_epochs = cfg.num_epochs
+    batch_size = cfg.batch_size
+    alpha = cfg.alpha
+    temperature = cfg.temperature
+    warmup_ratio = cfg.warmup_ratio
+    min_lr = cfg.min_lr
+    # ================================================================
+
+    # Load tokenizer and create model
+    print("\nLoading mmBERT-base tokenizer and creating dual-head model...")
+    tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+    model = mmBERTDualHead(
+        model_name=model_name,
+        num_onderwerp=len(onderwerp_names),
+        num_beleving=len(beleving_names),
+        dropout=dropout,
+        initial_threshold=initial_threshold,
+        use_thresholds=cfg.use_thresholds
+    )
+
+    # Move model to device
+    model = model.to(device)
+
+    # Ensure thresholds match encoder dtype for mixed precision safety
+    encoder_dtype = next(model.encoder.parameters()).dtype
+    with torch.no_grad():
+        if model.use_thresholds:
+            model.onderwerp_tau_logit.copy_(model.onderwerp_tau_logit.to(encoder_dtype))
+            model.beleving_tau_logit.copy_(model.beleving_tau_logit.to(encoder_dtype))
+
+    print(f"Model loaded and moved to {device}")
+    print(f"  Onderwerp head: {len(onderwerp_names)} outputs")
+    print(f"  Beleving head: {len(beleving_names)} outputs")
+
+    # Split data into train/val (80/20)
+    split_idx = int(0.8 * len(texts))
+    train_texts = texts[:split_idx]
+    train_onderwerp = onderwerp[:split_idx]
+    train_beleving = beleving[:split_idx]
+    val_texts = texts[split_idx:]
+    val_onderwerp = onderwerp[split_idx:]
+    val_beleving = beleving[split_idx:]
+
+    print(f"\nData split:")
+    print(f"  Train: {len(train_texts)} samples")
+    print(f"  Val: {len(val_texts)} samples")
+
+    # Create training dataset and dataloader
+    train_dataset = DutchDualLabelDataset(
+        train_texts, train_onderwerp, train_beleving, tokenizer, max_length
+    )
+
+    train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
+
+    steps_per_epoch = len(train_loader)
+    total_training_steps = steps_per_epoch * num_epochs
+
+    # Log derived/computed values to wandb (sweepable params already in config)
+    wandb.config.update({
+        # Fixed model configuration
+        "model_name": model_name,
+        "num_onderwerp": len(onderwerp_names),
+        "num_beleving": len(beleving_names),
+
+        # Derived training params
+        "threshold_peak_lr": threshold_peak_lr,
+        "total_training_steps": total_training_steps,
+
+        # Dataset info
+        "train_samples": len(train_texts),
+        "val_samples": len(val_texts),
+        "total_samples": len(texts),
+        "split_ratio": 0.8,
+
+        # Loss configuration (derived from alpha)
+        "loss_type": "combined_f1_bce",
+        "f1_weight": alpha,
+        "bce_weight": 1 - alpha,
+
+        # Fixed features
+        "learnable_thresholds": cfg.use_thresholds,
+        "per_class_thresholds": cfg.use_thresholds,
+        "gradient_clipping": True,
+        "max_grad_norm": 1.0,
+    }, allow_val_change=True)
+
+    # Print training info
+    print(f"\nStarting training for {num_epochs} total epochs with COMBINED F1+BCE LOSS...")
+    print(f"Loss formula: {alpha} * (1-F1) + {1-alpha} * BCE")
+    print(f"Temperature for Soft-F1: {temperature} | Initial thresholds: {initial_threshold}")
+    print(f"Batch size: {batch_size} | Total training batches: {steps_per_epoch}")
+    print(f"Learnable thresholds enabled for both onderwerp and beleving heads")
+    print("=" * 60)
+
+    # ===== SINGLE-PHASE TRAINING =====
+    print(f"\n{'='*60}")
+    print(f"TRAINING: {num_epochs} epoch(s)")
+    print(f"{'='*60}")
+
+    # Create optimizer and scheduler
+    optimizer, scheduler = make_opt_sched(
+        model,
+        enc_lr=encoder_peak_lr,
+        thr_lr=threshold_peak_lr,
+        total_steps=total_training_steps,
+        warmup_ratio=warmup_ratio,
+        eta_min=min_lr
+    )
+
+    # Run training
+    global_step = run_epochs(
+        model, tokenizer, train_loader,
+        val_texts, val_onderwerp, val_beleving,
+        onderwerp_names, beleving_names, device,
+        start_epoch=0, end_epoch=num_epochs,
+        phase_name="train",
+        optimizer=optimizer, scheduler=scheduler,
+        temperature=temperature, alpha=alpha,
+        max_length=max_length, global_step=0
+    )
+
+    # Training complete
+    print(f"\n{'='*60}")
+    print("TRAINING COMPLETE")
+    print(f"{'='*60}")
+
+    # Final evaluation on larger validation set
+    print("\n" + "=" * 60)
+    print("FINAL EVALUATION ON VALIDATION SET")
+    print("=" * 60)
+
+    print(f"\nEvaluating on 500 validation samples...")
+    final_metrics = evaluate(
+        model, val_texts, val_onderwerp, val_beleving, tokenizer, device,
+        onderwerp_names, beleving_names, num_samples=500, max_length=max_length
+    )
+
+    # Print overall metrics
+    print("\n" + "=" * 60)
+    print(f"FINAL METRICS (on {final_metrics['num_samples_evaluated']} validation samples)")
+    print("-" * 40)
+
+    print(f"  Onderwerp:")
+    print(f"    Accuracy: {final_metrics['onderwerp_acc']:.1%}")
+    print(f"    Precision: {final_metrics['onderwerp_precision']:.3f}")
+    print(f"    Recall: {final_metrics['onderwerp_recall']:.3f}")
+    print(f"    F1 Score: {final_metrics['onderwerp_f1']:.3f}")
+
+    print(f"\n  Beleving:")
+    print(f"    Accuracy: {final_metrics['beleving_acc']:.1%}")
+    print(f"    Precision: {final_metrics['beleving_precision']:.3f}")
+    print(f"    Recall: {final_metrics['beleving_recall']:.3f}")
+    print(f"    F1 Score: {final_metrics['beleving_f1']:.3f}")
+
+    print(f"\n  Combined:")
+    print(f"    Average Accuracy: {final_metrics['combined_acc']:.1%}")
+    print(f"    Average F1: {final_metrics['combined_f1']:.3f}")
+
+    # Log final metrics to wandb
+    wandb.log({
+        "final/onderwerp_acc": final_metrics['onderwerp_acc'],
+        "final/onderwerp_precision": final_metrics['onderwerp_precision'],
+        "final/onderwerp_recall": final_metrics['onderwerp_recall'],
+        "final/onderwerp_f1": final_metrics['onderwerp_f1'],
+        "final/beleving_acc": final_metrics['beleving_acc'],
+        "final/beleving_precision": final_metrics['beleving_precision'],
+        "final/beleving_recall": final_metrics['beleving_recall'],
+        "final/beleving_f1": final_metrics['beleving_f1'],
+        "final/combined_acc": final_metrics['combined_acc'],
+        "final/combined_f1": final_metrics['combined_f1'],
+    }, step=global_step)
+
+    print("\n" + "=" * 60)
+    print("Training complete! 🎉")
+    print("mmBERT-base dual-head architecture with balanced F1+BCE loss")
+    print(f"Loss formula: {alpha} * (1-F1) + {1-alpha} * BCE")
+    print(f"Temperature: {temperature}")
+    if cfg.use_thresholds:
+        print(f"Learned per-class thresholds:")
+        print(f"  Onderwerp ({len(onderwerp_names)} classes): mean={final_metrics['onderwerp_thresh_mean']:.3f} [{final_metrics['onderwerp_thresh_min']:.3f}-{final_metrics['onderwerp_thresh_max']:.3f}] σ={final_metrics['onderwerp_thresh_std']:.3f}")
+        print(f"  Beleving ({len(beleving_names)} classes): mean={final_metrics['beleving_thresh_mean']:.3f} [{final_metrics['beleving_thresh_min']:.3f}-{final_metrics['beleving_thresh_max']:.3f}] σ={final_metrics['beleving_thresh_std']:.3f}")
+    else:
+        print("Thresholds disabled (fixed cutoff τ=0.5 for both heads).")
+    print(f"With gradient clipping (max_norm=1.0) and warmup LR schedule")
+    print(f"Full dataset: {len(texts)} samples | Batch size: {batch_size} | Epochs: {num_epochs}")
+    print(f"mmBERT: Modern multilingual encoder (1800+ languages, max_length: {max_length})")
+
+    # Save final model weights (minimal model saving)
+    save_path = "mmbert_dual_head_final.pt"
+    torch.save(model.state_dict(), save_path)
+    print(f"\nModel weights saved to {save_path}")
+
+    # Save Hugging Face-compatible checkpoint (encoder + tokenizer + custom heads)
+    hf_dir = "mmbert_dual_head_hf"
+    os.makedirs(hf_dir, exist_ok=True)
+    # Save base encoder and tokenizer in HF format
+    model.encoder.save_pretrained(hf_dir)
+    tokenizer.save_pretrained(hf_dir)
+    # Save custom heads and metadata alongside
+    head_state = {
+        "onderwerp_head_state": model.onderwerp_head.state_dict(),
+        "beleving_head_state": model.beleving_head.state_dict(),
+        "use_thresholds": model.use_thresholds,
+        "num_onderwerp": len(onderwerp_names),
+        "num_beleving": len(beleving_names),
+        "dropout": dropout,
+        "max_length": max_length,
+        "alpha": alpha,
+        "temperature": temperature,
+        "model_name": model_name,
+    }
+    if model.use_thresholds:
+        head_state["onderwerp_tau_logit"] = model.onderwerp_tau_logit.detach().cpu()
+        head_state["beleving_tau_logit"] = model.beleving_tau_logit.detach().cpu()
+    torch.save(head_state, os.path.join(hf_dir, "dual_head_state.pt"))
+    # Save label names for convenience
+    with open(os.path.join(hf_dir, "label_names.json"), "w") as f:
+        json.dump({
+            "onderwerp": list(map(str, onderwerp_names)),
+            "beleving": list(map(str, beleving_names))
+        }, f, ensure_ascii=False, indent=2)
+    print(f"HF-compatible checkpoint saved to '{hf_dir}' (encoder+tokenizer), with heads in dual_head_state.pt")
+
+    # Finish wandb run
+    wandb.finish()
+    print("\nWandB logging completed and run finished.")
+
+
+if __name__ == "__main__":
+    main()