Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit
We present a training-free method to transplant tokenizers in pretrained
large language models (LLMs) by reconstructing unseen token embeddings via
Orthogonal Matching Pursuit (OMP). Specifically, we approximate each
out-of-vocabulary token as a sparse linear combination of shared tokens, in two
phases: first, compute each new token's representation in the donor embedding
space with a small dictionary of shared anchor tokens, then transfer these same
sparse coefficients back into the base model's embedding space.
  On two challenging cross-tokenizer tasks--LlamatoMistral NeMo (12B) and
QwentoLlama (1B)--we show that OMP achieves best zero-shot preservation of
the base model's performance across multiple benchmarks, while other zero-shot
approaches degrade significantly. Compared to baselines (zero-init, mean-init,
and existing approaches like WECHSEL, FOCUS, ZETT), OMP consistently achieves
the best overall performance, effectively bridging large tokenizer
discrepancies without gradient updates. Our analysis further identifies
mismatched numerical tokenization schemes as a critical challenge for
preserving mathematical reasoning capabilities. This technique enables direct
reuse of pretrained model weights with new tokenizers, facilitating
cross-tokenizer knowledge distillation, speculative decoding, ensembling,
merging, and domain-specific vocabulary adaptations. We integrate our method
into the open-source mergekit-tokensurgeon tool for post hoc vocabulary
realignment.
6848c79942e4f9106973f23e
Charles Goddard
claimed_verified
2025-06-11T08:35:12.039Z
630495b0ce6b12280b193c25
https://cdn-avatars.huggingface.co/v1/production/uploads/630495b0ce6b12280b193c25/bT61kBtQPhDYk00AI0o0g.png
Charles Goddard
user
chargoddard
6848c79942e4f9106973f23f
Fernando Fernandes Neto
Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit
We present a training-free method to transplant tokenizers in pretrained
large language models (LLMs) by reconstructing unseen token embeddings via
Orthogonal Matching Pursuit (OMP). Specifically, we approximate each
out-of-vocabulary token as a sparse linear combination of shared tokens, in two
phases: first, compute each new token's representation in the donor embedding
space with a small dictionary of shared anchor tokens, then transfer these same
sparse coefficients back into the base model's embedding space.
  On two challenging cross-tokenizer tasks--LlamatoMistral NeMo (12B) and
QwentoLlama (1B)--we show that OMP achieves best zero-shot preservation of
the base model's performance across multiple benchmarks, while other zero-shot
approaches degrade significantly. Compared to baselines (zero-init, mean-init,
and existing approaches like WECHSEL, FOCUS, ZETT), OMP consistently achieves
the best overall performance, effectively bridging large tokenizer
discrepancies without gradient updates. Our analysis further identifies
mismatched numerical tokenization schemes as a critical challenge for
preserving mathematical reasoning capabilities. This technique enables direct
reuse of pretrained model weights with new tokenizers, facilitating
cross-tokenizer knowledge distillation, speculative decoding, ensembling,
merging, and domain-specific vocabulary adaptations. We integrate our method
into the open-source mergekit-tokensurgeon tool for post hoc vocabulary
realignment.
A training-free method using Orthogonal Matching Pursuit (OMP) effectively transplants tokenizers in pretrained large language models, preserving performance across different tokenizers without gradient updates.
tokenizers pretrained large language models (LLMs) Orthogonal Matching Pursuit (OMP) token embeddings sparse linear combination anchor tokens cross-tokenizer tasks zero-shot performance baseline methods WECHSEL FOCUS ZETT numerical tokenization schemes knowledge distillation speculative decoding ensembling merging domain-specific vocabulary adaptations mergekit-tokensurgeon