Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided
  Language Modeling
Using large teacher models to guide the training of smaller student models
has become the prevailing paradigm for efficient and effective learning.
However, vocabulary mismatches between teacher and student language models pose
significant challenges in language modeling, resulting in divergent token
sequences and output distributions. To overcome these limitations, we propose
Vocabulary-agnostic Teacher Guided Language Modeling (VocAgnoLM), a novel
approach that bridges the gap caused by vocabulary mismatch through two key
methods: (1) Token-level Lexical Alignment, which aligns token sequences across
mismatched vocabularies, and (2) Teacher Guided Loss, which leverages the loss
of teacher model to guide effective student training. We demonstrate its
effectiveness in language modeling with 1B student model using various 7B
teacher models with different vocabularies. Notably, with
Qwen2.5-Math-Instruct, a teacher model sharing only about 6% of its vocabulary
with TinyLlama, VocAgnoLM achieves a 46% performance improvement compared to
naive continual pretraining. Furthermore, we demonstrate that VocAgnoLM
consistently benefits from stronger teacher models, providing a robust solution
to vocabulary mismatches in language modeling.
67e3fcb76323be71222fce35
Haebin Shin
67e3fcb76323be71222fce36
Lei Ji
extracted_pending
2025-03-26T13:10:17.004Z
664bed934bea570e25a8dc8c
/avatars/4e5a69db9005759b8812a9b54faf14c1.svg
Lei Ji
user
jilei03111
67e3fcb76323be71222fce37
Xiao Liu
extracted_confirmed
2025-03-26T13:10:33.192Z
63fb6e281b4b1bd4e7ffc5be
https://cdn-avatars.huggingface.co/v1/production/uploads/63fb6e281b4b1bd4e7ffc5be/aiRu_bulgnxvEMrjipGoQ.jpeg
Xiao Liu
user
lx865712528
67e3fcb76323be71222fce38
Yeyun Gong
extracted_pending
2025-03-26T13:10:17.004Z
643f615aa16cd6d1f4c581de
/avatars/47753a3e82b44f81881600c52e1e8495.svg
Yeyun Gong
user
yegong
Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided
  Language Modeling
Using large teacher models to guide the training of smaller student models
has become the prevailing paradigm for efficient and effective learning.
However, vocabulary mismatches between teacher and student language models pose
significant challenges in language modeling, resulting in divergent token
sequences and output distributions. To overcome these limitations, we propose
Vocabulary-agnostic Teacher Guided Language Modeling (VocAgnoLM), a novel
approach that bridges the gap caused by vocabulary mismatch through two key
methods: (1) Token-level Lexical Alignment, which aligns token sequences across
mismatched vocabularies, and (2) Teacher Guided Loss, which leverages the loss
of teacher model to guide effective student training. We demonstrate its
effectiveness in language modeling with 1B student model using various 7B
teacher models with different vocabularies. Notably, with
Qwen2.5-Math-Instruct, a teacher model sharing only about 6% of its vocabulary
with TinyLlama, VocAgnoLM achieves a 46% performance improvement compared to
naive continual pretraining. Furthermore, we demonstrate that VocAgnoLM
consistently benefits from stronger teacher models, providing a robust solution
to vocabulary mismatches in language modeling.
VocAgnoLM addresses vocabulary mismatches between teacher and student models through token-level alignment and teacher-guided loss, improving performance in language modeling.
Token-level Lexical Alignment Teacher Guided Loss VocAgnoLM language modeling student model teacher model vocabulary mismatches Qwen2.5-Math-Instruct TinyLlama