Let the LLM Stick to Its Strengths: Learning to Route Economical LLM
Yi-Kai Zhang, Shiyin Lu, Qingguo Chen, Weihua Luo, De-Chuan Zhan, Han-Jia Ye
Recently, test-time scaling of Large Language Models (LLMs) has emerged as a practical alternative to parameter and data scaling for reasoning tasks. Complex tasks often require large-scale parameters or advanced reasoning approaches like Chain-of-Thought (CoT) or RL-based LLMs, while more economical LLMs can handle simpler tasks. Routing an LLM tailored to domain and task complexity ensures flexibility and efficiency. We introduce EcoRouter, which routes the most suitable LLM based on throughput and reasoning process length. It constructs historical interactions between models and instructions as "preference scores" in a recommendation system and uses refined cold-start data for model representations. EcoRouter generalizes to unseen out-of-domain instructions and LLMs without fine-tuning. We also define the Pareto frontier-based metric under user-defined cost thresholds to balance cost and accuracy. We evaluate EcoRouter on a benchmark of over 50 LLMs, including DeepSeek-R1 and its distilled extensions, and three inference modes: self-consistency, CoT, and Tree-of-Thought (ToT). Our approach reduces unit costs by over 38% while maintaining performance, outperforming the average-optimal model for each dataset at the same cost level.