nym-pii-multilingual-small

Compact multilingual PII token-classification model for the nym anonymization CLI — a 16-layer, vocabulary-pruned, distilled student of Wismut/nym-pii-multilingual (mmBERT architecture). 429 MB fp32 / 133 MB compressed / 104 MB edge.

40 entity types / 81 BIO labels across ~23 languages and 6 scripts (Latin, Cyrillic, CJK, Korean, Arabic, Devanagari), trained with OCR-style corruption for robustness to scanned/mangled text, and with structured-document formats (JSON, CSV, tables, key-value, EDI) so field names and delimiters aren't mistaken for PII.

Files

File	Size	Use
model.onnx	429 MB	fp32, nym's default (accuracy-first)
int8/	133 MB	embedding-int8 + fp16 body weights (compute stays fp32). Near-lossless — see ladder below. token_model = "Wismut/nym-pii-multilingual-small/int8"
edge-int8/	104 MB	full dynamic int8 — smallest, pays real OOD accuracy (−4 ai4). For memory-constrained edge use only.
tokenizer.json	12 MB	vocabulary-pruned (199k of 256k tokens, 99.9% per-corpus coverage over 27 corpora incl. 25 languages of real text; byte-fallback keeps any input representable)

Compression is never free by assumption — every variant above was re-run on the full benchmark battery (v2's int8 shipped with a "99.9% agreement" claim that turned out to hide an 8-point OOD loss; that mistake is not repeated here).

Benchmarks

Same harness for every row (span-level, label-agnostic; OOD sets never trained on). v2 rows are the previously published artifacts, re-measured today:

Model	Size	Real-text F1	ai4privacy OOD	Non-Latin char-F1
v3 fp32 (this)	429 MB	76.4	67.7	71.8
v3 int8/	133 MB	76.4	67.2	71.7
v3 edge-int8/	104 MB	75.9	63.1	70.3
v2 fp32 (previous)	274 MB	72.9	55.6	68.6
v2 int8 (previous)	70 MB	70.0	47.2	67.3
teacher v3 (1.2 GB, fp32)	1.2 GB	79.1	69.8	73.1
OpenMed-PII-mSuperClinical (279M, their best)	1.1 GB	81.4	55.0	60.4
OpenMed-PII-SuperClinical-Small (44M)	172 MB	78.1	44.6	46.3
piiranha-v1	~600 MB	60.6	62.9	43.2

On the independent REDACT benchmark (25 languages / 9 scripts / 51 PII types, type-matched partial micro-F1, their harness): this model 0.507 — above the OpenAI Privacy Filter (0.512 → base tier 0.515 beats it, small ties), far above GLiNER-multi (0.320) and Presidio (0.195), below GPT-4.1 (0.597) and Claude Sonnet 4.6 (0.636).

Honest trade vs OpenMed: on curated real text their models win the F1 column — the 44M SuperClinical-Small by +1.7 over this model, their multilingual 279M mSuperClinical by +5.0. Off-distribution the picture inverts hard: this model is +23.1 ai4 over the 44M and +12.7 ai4 / +11.4 non-Latin over their multilingual best, at an 8× smaller compressed artifact (133 MB vs ~1.1 GB). Pick by whether your text looks like a clinical corpus or like the wild.

How it was trained

1. Init: jhu-clsp/mmBERT-small depth-pruned 22→16 layers.
2. Synthetic (724k): exactly-labeled data from Wismut/nym-pii-multilingual-data, including structured-format templates (JSON/CSV/table/key-value/EDI).
3. Real text (77.5k): Wikipedia passages auto-labeled by a large LLM teacher (gemma-4-26b); weak O tokens masked from the loss.
4. Knowledge distillation from the retrained mmBERT-base teacher (--distill-from, KL on all attended tokens, α=0.5, T=2) — sharpens calibration, worth ~+3 ai4.
5. Recall tilt (--o-weight 0.3): down-weights the O class; combined with KD it is super-additive (tilt alone costs precision; KD repairs it).
6. Compression: vocab prune (0.999 coverage) → embedding-only int8 → weight-only fp16 body. Each step individually re-benchmarked.

Pipeline: scripts/ (datagen/, prune/, train_ner.py, export_onnx.py).

Versions

• v3 (current) — retrained-teacher KD + recall tilt; corrected ONNX export (v2's export env silently ran a different transformers version whose ModernBERT forward differs — the shipped v2 artifact lost ~3.5 ai4 vs its own checkpoint; v3 exports are gate-verified torch-vs-ONNX ≤1e-3 at 10 lengths). Same-harness gains over shipped v2: +3.5 F1, +12.1 ai4, +3.2 non-Latin; the int8 tier gains +6.4 F1 / +20 ai4 over v2-int8.
• v2 — LLM-teacher-labeled real text + structured-format synthetic.
• v1 — synthetic + mmBERT-teacher-labeled real text.

Use with nym

[ner]
enabled = true
backend = "tokens"
token_model = "Wismut/nym-pii-multilingual-small"        # fp32, accuracy-first
# token_model = "Wismut/nym-pii-multilingual-small/int8" # 133 MB, near-lossless
# token_model = "Wismut/nym-pii-multilingual-small/edge-int8" # 104 MB, edge
threshold = 0.25
# recall_first = true   # flag on total entity mass (1-P(O)): +5-9 char-recall
#                       # at ~-3 precision. Right mode for redaction; keep off
#                       # when label accuracy matters more than coverage.

Limitations

• The pruned tokenizer + ONNX layout targets nym / onnxruntime; for transformers fine-tuning use the teacher repo (Wismut/nym-pii-multilingual).
• Trained partly on LLM-labeled Wikipedia text (CC-BY-SA source; not redistributed here). Those labels are machine-generated.
• Recall-leaning by design; expect over-flagging on ambiguous terms.
• Reference-style IDs (ticket/case numbers) are the weakest class on out-of-taxonomy benchmarks — pair with nym's regex patterns, which is how the full pipeline runs anyway.
• No claim of HIPAA/GDPR compliance by itself; it is a detection component.

License

MIT (inherited from mmBERT-small). The synthetic training data contains no real personal information; the real-text training corpus is not distributed with this model.