Fast-SLM: Towards Latency-Optimal Hybrid Small Language Models
Yonggan Fu, Xin Dong, Shizhe Diao, Matthijs Van keirsbilck, Hanrong Ye, Wonmin Byeon, Yashaswi Karnati, Lucas Liebenwein, Maksim Khadkevich, Alexander Keller, Jan Kautz, Yingyan (Celine) Lin, Pavlo Molchanov
Efficient deployment of small language models (SLMs) is essential for numerous real-world applications with stringent latency constraints. While previous work in SLM design mainly focuses on reducing the number of parameters to deliver parameter-optimal SLMs, parameter efficiency may not necessarily translate into proportional real-device speed-ups. This work aims to provide a systematic exploration and roadmap for latency-optimal SLMs. Our goal is to identify critical determinants of SLMs' real-device latency and provide generalizable principles and methodologies for SLM design and training when real-device latency becomes the primary consideration. Specifically, we first analyze two central architecture design factors: depth-width ratios and the involved operators. We find that although deep-thin models generally lead to better accuracy under the same parameter budget, they may not lie on the frontier of the accuracy-latency trade-off. To identify the latency-optimal depth-width ratio, we augment previous scaling laws by relating model loss to both model depth and width, thus enabling determination of the sweet spot depth-width ratio when combined with device-specific profiling. Additionally, we explore emerging efficient attention alternatives to understand their potential as candidate building operators. Using the identified promising operators, we build an evolutionary search framework to automatically pinpoint optimal latency combinations of these operators into hybrid SLMs to push the accuracy-latency frontier. In addition to architectural improvements, we further analyze and enhance SLM training by enabling more effective weight updates and improving cache initialization, which are generalizable add-on components for future SLMs. Combining these contributions, we introduce a new family of hybrid SLMs, called Fast-SLM, which significantly advances the accuracy–latency trade-off frontier of state-of-the-art SLMs, e.g., achieving 2.57\% higher accuracy, 1.46$\times$ speed-up, and over 10$\times$ cache reduction compared to Llama3.2-3B.