AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through
  Lightweight Vocabulary Adaptation
Large Language Models (LLMs) have shown impressive versatility as general
purpose models. However, their broad applicability comes at a high-cost
computational overhead, particularly in auto-regressive decoding where each
step requires a forward pass. In domain-specific settings, general-purpose
capabilities are unnecessary and can be exchanged for efficiency. In this work,
we take a novel perspective on domain adaptation, reducing latency and
computational costs by adapting the vocabulary to focused domains of interest.
We introduce AdaptiVocab, an end-to-end approach for vocabulary adaptation,
designed to enhance LLM efficiency in low-resource domains. AdaptiVocab can be
applied to any tokenizer and architecture, modifying the vocabulary by
replacing tokens with domain-specific n-gram-based tokens, thereby reducing the
number of tokens required for both input processing and output generation.
AdaptiVocab initializes new n-token embeddings using an exponentially weighted
combination of existing embeddings and employs a lightweight fine-tuning phase
that can be efficiently performed on a single GPU. We evaluate two 7B LLMs
across three niche domains, assessing efficiency, generation quality, and
end-task performance. Our results show that AdaptiVocab reduces token usage by
over 25% without compromising performance
67ea363dd13d75fc156ec498
Itay Nakash
admin_assigned
2025-03-31T08:24:09.634Z
671f8106d677d3a764a6f9a5
/avatars/90b4b00058aac30c060c5eac8debb1c7.svg
itay nakash
user
itaynakash
67ea363dd13d75fc156ec499
Nitay Calderon
admin_assigned
2025-03-31T08:24:15.760Z
62d6a0c18faee0ac953c51fa
/avatars/ca818cebdb089a8d853c5bc4d5e0987b.svg
Nitay Calderon
user
nitay
67ea363dd13d75fc156ec49a
Eyal Ben David
admin_assigned
2025-03-31T08:24:21.914Z
6645fc650e6706053171ce51
/avatars/54b03ac6939d4b8943606b12b979ce52.svg
Eyal Ben-David
user
eyalbd
67ea363dd13d75fc156ec49b
Elad Hoffer
admin_assigned
2025-03-31T08:24:27.895Z
630480fa6dbbb80f16352ee3
/avatars/f39ce2fe96a578f42a57e3bfe3a2d137.svg
Elad Hoffer
user
ehoffer
67ea363dd13d75fc156ec49c
Roi Reichart
AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through
  Lightweight Vocabulary Adaptation
Large Language Models (LLMs) have shown impressive versatility as general
purpose models. However, their broad applicability comes at a high-cost
computational overhead, particularly in auto-regressive decoding where each
step requires a forward pass. In domain-specific settings, general-purpose
capabilities are unnecessary and can be exchanged for efficiency. In this work,
we take a novel perspective on domain adaptation, reducing latency and
computational costs by adapting the vocabulary to focused domains of interest.
We introduce AdaptiVocab, an end-to-end approach for vocabulary adaptation,
designed to enhance LLM efficiency in low-resource domains. AdaptiVocab can be
applied to any tokenizer and architecture, modifying the vocabulary by
replacing tokens with domain-specific n-gram-based tokens, thereby reducing the
number of tokens required for both input processing and output generation.
AdaptiVocab initializes new n-token embeddings using an exponentially weighted
combination of existing embeddings and employs a lightweight fine-tuning phase
that can be efficiently performed on a single GPU. We evaluate two 7B LLMs
across three niche domains, assessing efficiency, generation quality, and
end-task performance. Our results show that AdaptiVocab reduces token usage by
over 25% without compromising performance
AdaptiVocab, a novel vocabulary adaptation method, enhances the efficiency of Large Language Models in domain-specific applications without sacrificing performance by reducing token usage.
Large Language Models LLMs domain adaptation auto-regressive decoding vocabulary adaptation AdaptiVocab n-gram-based tokens token embeddings fine-tuning end-to-end approach