Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs
Efficient use of large language models (LLMs) is critical for deployment at
scale: without adaptive routing, systems either overpay for strong models or
risk poor performance from weaker ones. Selecting the right LLM for each query
is fundamentally an online decision problem: models differ in strengths, prices
fluctuate, and users value accuracy and cost differently. Yet most routers are
trained offline with labels for all candidate models, an assumption that breaks
in deployment, where only the outcome of the chosen model is observed. We
bridge this gap with BaRP, a Bandit-feedback Routing with Preferences approach
that trains under the same partial-feedback restriction as deployment, while
supporting preference-tunable inference: operators can dial the
performance/cost trade-off at test time without retraining. Framed as a
contextual bandit over prompt features and a user preference vector, our method
simulates an online feedback setting during training and adapts its routing
decisions to each new prompt, rather than depending on full-information offline
supervision. Comprehensive experiments show that our method consistently
outperforms strong offline routers by at least 12.46% and the largest LLM by at
least 2.45%, and generalizes robustly for unseen tasks.
68e86c0e95e8e6771df38902
Wang Wei
68e86c0e95e8e6771df38903
Tiankai Yang
68e86c0e95e8e6771df38904
Hongjie Chen
68e86c0e95e8e6771df38905
Yue Zhao
68e86c0e95e8e6771df38906
Franck Dernoncourt
claimed_verified
2025-10-11T13:48:15.426Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
68e86c0e95e8e6771df38907
Ryan A. Rossi
68e86c0e95e8e6771df38908
Hoda Eldardiry
Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs
Efficient use of large language models (LLMs) is critical for deployment at
scale: without adaptive routing, systems either overpay for strong models or
risk poor performance from weaker ones. Selecting the right LLM for each query
is fundamentally an online decision problem: models differ in strengths, prices
fluctuate, and users value accuracy and cost differently. Yet most routers are
trained offline with labels for all candidate models, an assumption that breaks
in deployment, where only the outcome of the chosen model is observed. We
bridge this gap with BaRP, a Bandit-feedback Routing with Preferences approach
that trains under the same partial-feedback restriction as deployment, while
supporting preference-tunable inference: operators can dial the
performance/cost trade-off at test time without retraining. Framed as a
contextual bandit over prompt features and a user preference vector, our method
simulates an online feedback setting during training and adapts its routing
decisions to each new prompt, rather than depending on full-information offline
supervision. Comprehensive experiments show that our method consistently
outperforms strong offline routers by at least 12.46% and the largest LLM by at
least 2.45%, and generalizes robustly for unseen tasks.
BaRP, a Bandit-feedback Routing with Preferences approach, optimizes large language model selection in an online setting with partial feedback, outperforming offline routers and large models.
large language models adaptive routing online decision problem partial-feedback restriction contextual bandit prompt features user preference vector performance/cost trade-off