Patch-Level Training for Large Language Models
As Large Language Models (LLMs) achieve remarkable progress in language
understanding and generation, their training efficiency has become a critical
concern. Traditionally, LLMs are trained to predict the next token in a
sequence. Despite the success of token-level training, it suffers from
considerable computational costs due to the need to process an extensive number
of tokens. To mitigate this issue, this paper introduces patch-level training
for LLMs, which reduces the sequence length by compressing multiple tokens into
a single patch. During patch-level training, we feed the language model shorter
sequences of patches and train it to predict the next patch, thereby processing
the majority of the training data at a significantly reduced computational
cost. Following this, the model continues token-level training on the remaining
training data to align with the inference mode. Experiments on a diverse range
of models (370M-2.7B parameters) demonstrate that patch-level training can
reduce overall computational costs to 0.5times, without compromising the
model performance compared to token-level training. Source code:
https://github.com/shaochenze/PatchTrain.
6698800ebd704c8fb2d7d5f7
Chenze Shao
6698800ebd704c8fb2d7d5f8
Fandong Meng
6698800ebd704c8fb2d7d5f9
Jie Zhou
Patch-Level Training for Large Language Models
As Large Language Models (LLMs) achieve remarkable progress in language
understanding and generation, their training efficiency has become a critical
concern. Traditionally, LLMs are trained to predict the next token in a
sequence. Despite the success of token-level training, it suffers from
considerable computational costs due to the need to process an extensive number
of tokens. To mitigate this issue, this paper introduces patch-level training
for LLMs, which reduces the sequence length by compressing multiple tokens into
a single patch. During patch-level training, we feed the language model shorter
sequences of patches and train it to predict the next patch, thereby processing
the majority of the training data at a significantly reduced computational
cost. Following this, the model continues token-level training on the remaining
training data to align with the inference mode. Experiments on a diverse range
of models (370M-2.7B parameters) demonstrate that patch-level training can
reduce overall computational costs to 0.5times, without compromising the
model performance compared to token-level training. Source code:
https://github.com/shaochenze/PatchTrain.
Patch-level training for LLMs reduces computational costs by compressing multiple tokens into patches, decreasing sequence length while maintaining model performance.
Large Language Models (LLMs) token-level training patch-level training computational costs sequence length patches token-level training inference mode parameters