Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs
LLMs process text as sequences of tokens that roughly correspond to words,
where less common words are represented by multiple tokens. However, individual
tokens are often semantically unrelated to the meanings of the words/concepts
they comprise. For example, Llama-2-7b's tokenizer splits the word
"northeastern" into the tokens ['_n', 'ort', 'he', 'astern'], none of which
correspond to semantically meaningful units like "north" or "east." Similarly,
the overall meanings of named entities like "Neil Young" and multi-word
expressions like "break a leg" cannot be directly inferred from their
constituent tokens. Mechanistically, how do LLMs convert such arbitrary groups
of tokens into useful higher-level representations? In this work, we find that
last token representations of named entities and multi-token words exhibit a
pronounced "erasure" effect, where information about previous and current
tokens is rapidly forgotten in early layers. Using this observation, we propose
a method to "read out" the implicit vocabulary of an autoregressive LLM by
examining differences in token representations across layers, and present
results of this method for Llama-2-7b and Llama-3-8B. To our knowledge, this is
the first attempt to probe the implicit vocabulary of an LLM.
668420ae3b6f1b196d0074ef
Sheridan Feucht
admin_assigned
2024-07-03T15:06:30.903Z
653bf2002c81c4adc8f99d70
/avatars/0bd74739cef2df5d4889f108de17a120.svg
Sheridan Feucht
user
sfeucht
668420ae3b6f1b196d0074f0
David Atkinson
admin_assigned
2024-07-03T15:06:36.877Z
64769e3c3d8d1cb791909f09
/avatars/c09f6f1df52a710db6a65e37a4545458.svg
David Atkinson
user
datkinson
668420ae3b6f1b196d0074f1
Byron Wallace
admin_assigned
2024-07-03T15:06:43.376Z
6474aa4433192631bace7ff0
/avatars/06b586fc65350800a9aeffacd7bbdf50.svg
Byron Wallace
user
byroncwallace
668420ae3b6f1b196d0074f2
David Bau
admin_assigned
2024-07-03T15:06:51.724Z
6214d6c01e35c843d42d1f77
/avatars/ac208cd180b4f3ed1ec367e581facfcf.svg
David Bau
user
davidbau
Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs
LLMs process text as sequences of tokens that roughly correspond to words,
where less common words are represented by multiple tokens. However, individual
tokens are often semantically unrelated to the meanings of the words/concepts
they comprise. For example, Llama-2-7b's tokenizer splits the word
"northeastern" into the tokens ['_n', 'ort', 'he', 'astern'], none of which
correspond to semantically meaningful units like "north" or "east." Similarly,
the overall meanings of named entities like "Neil Young" and multi-word
expressions like "break a leg" cannot be directly inferred from their
constituent tokens. Mechanistically, how do LLMs convert such arbitrary groups
of tokens into useful higher-level representations? In this work, we find that
last token representations of named entities and multi-token words exhibit a
pronounced "erasure" effect, where information about previous and current
tokens is rapidly forgotten in early layers. Using this observation, we propose
a method to "read out" the implicit vocabulary of an autoregressive LLM by
examining differences in token representations across layers, and present
results of this method for Llama-2-7b and Llama-3-8B. To our knowledge, this is
the first attempt to probe the implicit vocabulary of an LLM.
LLMs convert token sequences into higher-level representations by exhibiting an "erasure" effect in early layers, where information is rapidly forgotten; a method is proposed to probe the implicit vocabulary of LLMs by analyzing token representations across layers.
LLMs autoregressive LLMs token representations erasure effect implicit vocabulary