Adaptive Layer-skipping in Pre-trained LLMs
Various layer-skipping methods have been proposed to accelerate token
generation in large language models (LLMs). However, they have overlooked a
fundamental question: How do computational demands vary across the generation
of different tokens? In this work, we introduce FlexiDepth, a method that
dynamically adjusts the number of Transformer layers used in text generation.
By incorporating a plug-in router and adapter, FlexiDepth enables adaptive
layer-skipping in LLMs without modifying their original parameters. Introducing
FlexiDepth to Llama-3-8B model achieves layer skipping of 8 layers out of 32,
and meanwhile maintains the full 100\% benchmark performance. Experimental
results with FlexiDepth demonstrate that computational demands in LLMs
significantly vary based on token type. Specifically, generating repetitive
tokens or fixed phrases requires fewer layers, whereas producing tokens
involving computation or high uncertainty requires more layers. Interestingly,
this adaptive allocation pattern aligns with human intuition. To advance
research in this area, we open sourced FlexiDepth and a dataset documenting
FlexiDepth's layer allocation patterns for future exploration.
67ec9ed83d267d266640c03d
Xuan Luo
claimed_verified
2025-04-02T08:22:44.959Z
659d98feec9f8f337f145ec4
/avatars/7de3bea0f203326e78916882655b5d55.svg
xuan luo
user
xuan-luo
67ec9ed83d267d266640c03e
Weizhi Wang
claimed_verified
2025-04-02T08:22:42.010Z
63d34004b734eaa4d4faeccf
https://cdn-avatars.huggingface.co/v1/production/uploads/63d34004b734eaa4d4faeccf/aH7F-xn4PMQjHHcvL-0vs.jpeg
Weizhi Wang
user
weizhiwang
67ec9ed83d267d266640c03f
Xifeng Yan
Adaptive Layer-skipping in Pre-trained LLMs
Various layer-skipping methods have been proposed to accelerate token
generation in large language models (LLMs). However, they have overlooked a
fundamental question: How do computational demands vary across the generation
of different tokens? In this work, we introduce FlexiDepth, a method that
dynamically adjusts the number of Transformer layers used in text generation.
By incorporating a plug-in router and adapter, FlexiDepth enables adaptive
layer-skipping in LLMs without modifying their original parameters. Introducing
FlexiDepth to Llama-3-8B model achieves layer skipping of 8 layers out of 32,
and meanwhile maintains the full 100\% benchmark performance. Experimental
results with FlexiDepth demonstrate that computational demands in LLMs
significantly vary based on token type. Specifically, generating repetitive
tokens or fixed phrases requires fewer layers, whereas producing tokens
involving computation or high uncertainty requires more layers. Interestingly,
this adaptive allocation pattern aligns with human intuition. To advance
research in this area, we open sourced FlexiDepth and a dataset documenting
FlexiDepth's layer allocation patterns for future exploration.
FlexiDepth dynamically adjusts Transformer layers in LLMs based on token complexity, enhancing computational efficiency without sacrificing performance.
layer-skipping token generation large language models LLMs Transformer layers FlexiDepth plug-in router adapter layer allocation benchmark performance