Achieving Tokenizer Flexibility in Language Models through Heuristic
  Adaptation and Supertoken Learning
Pretrained language models (LLMs) are often constrained by their fixed
tokenization schemes, leading to inefficiencies and performance limitations,
particularly for multilingual or specialized applications. This tokenizer
lock-in presents significant challenges. standard methods to overcome this
often require prohibitive computational resources. Although tokenizer
replacement with heuristic initialization aims to reduce this burden, existing
methods often require exhaustive residual fine-tuning and still may not fully
preserve semantic nuances or adequately address the underlying compression
inefficiencies. Our framework introduces two innovations: first, Tokenadapt, a
model-agnostic tokenizer transplantation method, and second, novel
pre-tokenization learning for multi-word Supertokens to enhance compression and
reduce fragmentation. Tokenadapt initializes new unique token embeddings via a
hybrid heuristic that combines two methods: a local estimate based on subword
decomposition using the old tokenizer, and a global estimate utilizing the
top-k semantically similar tokens from the original vocabulary. This
methodology aims to preserve semantics while significantly minimizing
retraining requirements. Empirical investigations validate both contributions:
the transplantation heuristic successfully initializes unique tokens, markedly
outperforming conventional baselines and sophisticated methods including
Transtokenizer and ReTok, while our Supertokens achieve notable compression
gains. Our zero-shot perplexity results demonstrate that the TokenAdapt hybrid
initialization consistently yields lower perplexity ratios compared to both
ReTok and TransTokenizer baselines across different base models and newly
trained target tokenizers. TokenAdapt typically reduced the overall perplexity
ratio significantly compared to ReTok, yielding at least a 2-fold improvement
in these aggregate scores.
6826c755f032d814754cadfe
Shaurya Sharthak
6826c755f032d814754cadff
Vinayak Pahalwan
6826c755f032d814754cae00
Adithya Kamath
claimed_verified
2025-05-16T07:11:49.739Z
6523d85b27d1f3d84ab3a0a4
https://cdn-avatars.huggingface.co/v1/production/uploads/6523d85b27d1f3d84ab3a0a4/GF159gtSvxSWR9TAJIQby.jpeg
Adithya Kamath
user
adi-kmt
6826c755f032d814754cae01
Adarsh Shirawalmath
extracted_confirmed
2025-05-16T05:19:45.785Z
644bf6ef778ecbfb977e8e84
https://cdn-avatars.huggingface.co/v1/production/uploads/644bf6ef778ecbfb977e8e84/x2zFOkSIhDHf0LEZ1wzbU.jpeg
Adarsh AS
user
adarshxs
Achieving Tokenizer Flexibility in Language Models through Heuristic
  Adaptation and Supertoken Learning
Pretrained language models (LLMs) are often constrained by their fixed
tokenization schemes, leading to inefficiencies and performance limitations,
particularly for multilingual or specialized applications. This tokenizer
lock-in presents significant challenges. standard methods to overcome this
often require prohibitive computational resources. Although tokenizer
replacement with heuristic initialization aims to reduce this burden, existing
methods often require exhaustive residual fine-tuning and still may not fully
preserve semantic nuances or adequately address the underlying compression
inefficiencies. Our framework introduces two innovations: first, Tokenadapt, a
model-agnostic tokenizer transplantation method, and second, novel
pre-tokenization learning for multi-word Supertokens to enhance compression and
reduce fragmentation. Tokenadapt initializes new unique token embeddings via a
hybrid heuristic that combines two methods: a local estimate based on subword
decomposition using the old tokenizer, and a global estimate utilizing the
top-k semantically similar tokens from the original vocabulary. This
methodology aims to preserve semantics while significantly minimizing
retraining requirements. Empirical investigations validate both contributions:
the transplantation heuristic successfully initializes unique tokens, markedly
outperforming conventional baselines and sophisticated methods including
Transtokenizer and ReTok, while our Supertokens achieve notable compression
gains. Our zero-shot perplexity results demonstrate that the TokenAdapt hybrid
initialization consistently yields lower perplexity ratios compared to both
ReTok and TransTokenizer baselines across different base models and newly
trained target tokenizers. TokenAdapt typically reduced the overall perplexity
ratio significantly compared to ReTok, yielding at least a 2-fold improvement
in these aggregate scores.
A new framework introduces Tokenadapt for tokenizer transplantation and Supertokens for improved pre-tokenization to enhance compression and reduce inefficiencies in pretrained language models.
pretrained language models Tokenadapt tokenizer transplantation Supertokens pre-tokenization learning subword decomposition zero-shot perplexity