TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar
Large language models (LLMs) for code rely on subword tokenizers, such as
byte-pair encoding (BPE), learned from mixed natural language text and
programming language code but driven by statistics rather than grammar. As a
result, semantically identical code snippets can be tokenized differently
depending on superficial factors such as whitespace or identifier naming. To
measure the impact of this misalignment, we introduce TokDrift, a framework
that applies semantic-preserving rewrite rules to create code variants
differing only in tokenization. Across nine code LLMs, including large ones
with over 30B parameters, even minor formatting changes can cause substantial
shifts in model behavior. Layer-wise analysis shows that the issue originates
in early embeddings, where subword segmentation fails to capture grammar token
boundaries. Our findings identify misaligned tokenization as a hidden obstacle
to reliable code understanding and generation, highlighting the need for
grammar-aware tokenization for future code LLMs.
68f19f5f6e0bef323a68fc99
Yinxi Li
claimed_verified
2025-10-17T04:11:23.022Z
675764b1a55640463e079271
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/6EMvM7M8uP9K7EeVMqAz-.png
Yinxi Li
user
Yinxxx
68f19f5f6e0bef323a68fc9a
Yuntian Deng
claimed_verified
2025-10-17T05:56:56.808Z
63081e15a670ed10f9d44229
https://cdn-avatars.huggingface.co/v1/production/uploads/63081e15a670ed10f9d44229/w1b9uq-9774bMMgJbSPsS.jpeg
Yuntian Deng
user
yuntian-deng
68f19f5f6e0bef323a68fc9b
Pengyu Nie
claimed_verified
2025-10-17T04:11:25.906Z
6643bdd9bc86c9465b551357
/avatars/98f04dc2263e79f170e662a9ed113fc5.svg
Pengyu Nie
user
pengyunie
TokDrift: When LLM Speaks in Subwords but Code Speaks in Grammar
Large language models (LLMs) for code rely on subword tokenizers, such as
byte-pair encoding (BPE), learned from mixed natural language text and
programming language code but driven by statistics rather than grammar. As a
result, semantically identical code snippets can be tokenized differently
depending on superficial factors such as whitespace or identifier naming. To
measure the impact of this misalignment, we introduce TokDrift, a framework
that applies semantic-preserving rewrite rules to create code variants
differing only in tokenization. Across nine code LLMs, including large ones
with over 30B parameters, even minor formatting changes can cause substantial
shifts in model behavior. Layer-wise analysis shows that the issue originates
in early embeddings, where subword segmentation fails to capture grammar token
boundaries. Our findings identify misaligned tokenization as a hidden obstacle
to reliable code understanding and generation, highlighting the need for
grammar-aware tokenization for future code LLMs.
Misaligned tokenization in large language models for code leads to inconsistent model behavior, necessitating grammar-aware tokenization.
large language models code LLMs subword tokenizers byte-pair encoding BPE semantic-preserving rewrite rules TokDrift tokenization early embeddings grammar token boundaries code understanding code generation