ViLLM-Tok: Vietnamese-English Code-Switching Tokenizer
A hybrid tokenizer for Vietnamese-English code-switching, powering the viLLM project. Combines a Vietnamese syllable/compound vocabulary with tiktoken byte-level BPE (cl100k_base) for English, delivering 100% roundtrip fidelity on a diverse 80-test benchmark.
| Property | Value |
|---|---|
| Vocab size | 203,470 |
| English BPE | tiktoken cl100k_base (100,256 merge ranks) |
| Vietnamese | Syllable-level + compound bigrams via Viterbi DP |
| Languages | Vietnamese, English, code-switching |
| Special tokens | [PAD], [UNK], [BOS], [EOS], [SEP], [MASK] |
| Code-switch markers | [VI→EN], [EN→VI] (optional) |
| Roundtrip accuracy | 80/80 (100%) — Unicode, CJK, Thai, Zalgo, emoji, code |
How it works
Tokenization proceeds in stages:
- Pretokenization — character-class grouping (alpha/digit/punct/other) with combining-mark preservation
- Language detection — per-word Vi/EN/Num/Code/Punct classification
- Vietnamese Viterbi — dynamic programming over syllable runs to merge frequent bigrams into compound tokens (
học_sinh,Việt_Nam,công_nghệ_thông_tin) - English tiktoken BPE — byte-level BPE (cl100k_base) for English words not in the direct vocab
- Byte fallback — any character not in vocab encoded as
<0xNN>byte tokens - Code-switch markers — optional
[VI→EN]/[EN→VI]insertion at language boundaries
Quick start
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(
"vlinhd11/villm-tokenizer",
trust_remote_code=True,
)
text = "Học sinh giỏi tiếng Việt và học lập trình Python"
enc = tok(text)
print(tok.convert_ids_to_tokens(enc["input_ids"]))
# ['Học_sinh', 'giỏi', 'tiếng_Việt', 'và_học', 'lập_trình', '[VI→EN]', ' Python']
Install Rust backend for faster tokenization
pip install villm-tok-rs
The Rust backend is auto-detected on next import — no code changes needed. It uses tiktoken byte-level BPE and runs the full pipeline natively.
# Same code — Rust is used automatically when installed
tok = AutoTokenizer.from_pretrained("vlinhd11/villm-tokenizer", trust_remote_code=True)
Disable code-switch markers
tok.add_code_switch_markers = False
Tokenization examples
| Input | Tokens |
|---|---|
Học sinh |
['Học_sinh'] |
xe máy Việt Nam |
['xe_máy', 'Việt_Nam'] |
công nghệ thông tin |
['công_nghệ', 'thông_tin'] |
Học_sinh giỏi tiếng_Việt... |
['Học_sinh', 'giỏi', 'tiếng_Việt', 'và_học', 'lập_trình', '[VI→EN]', ' Python'] |
Thủ tướng Phạm Minh Chính |
['Thủ_tướng', 'Phạm', 'Minh', 'Chính'] |
Hello, world! |
['Hello', ',', ' world', '!'] |
你好世界 |
['你', '好', '世', '界'] |
H̴e̷l̶l̶o̵ |
['H̴', 'e̷', 'l̶', 'l̶', 'o̵'] |
Roundtrip benchmark
The tokenizer guarantees 100% exact roundtrip (encode → decode returns the original text) across 80 diverse test cases:
- English text, punctuation, numbers, code
- Vietnamese syllables and compounds
- CJK (Chinese, Japanese, Korean)
- Thai with combining vowel marks
- Zalgo / combining diacritical marks
- Emoji and special symbols
- Mixed-script code-switching
Backend comparison
| Backend | Speed | BPE engine | Install |
|---|---|---|---|
Rust (villm-tok-rs) |
~8x faster | tiktoken cl100k_base | pip install villm-tok-rs |
| Pure Python | 1x | SentencePiece (fallback) | none needed |
Files on this repo
| File | Description |
|---|---|
vocab.json |
203,470 tokens (vi_syllables, en_bpe, en_sp, special) |
token_meta.json |
Per-token metadata (type, frequency) |
tiktoken_ranks_hex.json |
cl100k_base merge ranks (hex-encoded byte keys) |
en_subwords.json |
154,783 English subword tokens for smart join |
hf_tokenizer.py |
Main tokenizer implementation (AutoTokenizer-compatible) |
tokenization_villm.py |
Import shim for AutoTokenizer discovery |
Citation
@software{villm_tokenizer,
author = {vlinhd11},
title = {ViLLM-Tok: Vietnamese-English Code-Switching Tokenizer},
year = {2025},
url = {https://huggingface.co/vlinhd11/villm-tokenizer}
}