When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM
  Ensembling
Ensembling Large Language Models (LLMs) has gained attention as a promising
approach to surpass the performance of individual models by leveraging their
complementary strengths. In particular, aggregating models' next-token
probability distributions to select the next token has been shown to be
effective in various tasks. However, while successful for short-form answers,
its application to long-form generation remains underexplored. In this paper,
we show that using existing ensemble methods in long-form generation requires a
careful choice of ensembling positions, since the standard practice of
ensembling at every token often degrades performance. We identify two key
factors for determining these positions: tokenization mismatch across models
and consensus in their next-token probability distributions. Based on this, we
propose SAFE, (Stable And Fast LLM Ensembling), a framework that selectively
ensembles by jointly considering these factors. To further improve stability,
we introduce a probability sharpening strategy that consolidates probabilities
spread across multiple sub-word tokens representing the same word into a single
representative token. Our experiments on diverse benchmarks, including MATH500
and BBH, demonstrate that SAFE outperforms existing methods in both accuracy
and efficiency, with gains achieved even when ensembling fewer than 1% of
tokens.
68f5cb858589920bf4d321c6
Heecheol Yun
claimed_verified
2025-10-20T12:09:53.779Z
67f778ddbb19958f5d96c2a8
/avatars/49a3f119b456ff94f28f09b2fe78bb18.svg
Heecheol Yun
user
yoon6503
68f5cb858589920bf4d321c7
Kwangmin Ki
claimed_verified
2025-10-21T14:04:48.478Z
66e5476b8edfc6dc4461af24
/avatars/7dbcc2465a6268ebeac118c396581fd1.svg
Ki Kwang Min
user
kiikiik
68f5cb858589920bf4d321c8
Junghyun Lee
68f5cb858589920bf4d321c9
Eunho Yang
When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM
  Ensembling
Ensembling Large Language Models (LLMs) has gained attention as a promising
approach to surpass the performance of individual models by leveraging their
complementary strengths. In particular, aggregating models' next-token
probability distributions to select the next token has been shown to be
effective in various tasks. However, while successful for short-form answers,
its application to long-form generation remains underexplored. In this paper,
we show that using existing ensemble methods in long-form generation requires a
careful choice of ensembling positions, since the standard practice of
ensembling at every token often degrades performance. We identify two key
factors for determining these positions: tokenization mismatch across models
and consensus in their next-token probability distributions. Based on this, we
propose SAFE, (Stable And Fast LLM Ensembling), a framework that selectively
ensembles by jointly considering these factors. To further improve stability,
we introduce a probability sharpening strategy that consolidates probabilities
spread across multiple sub-word tokens representing the same word into a single
representative token. Our experiments on diverse benchmarks, including MATH500
and BBH, demonstrate that SAFE outperforms existing methods in both accuracy
and efficiency, with gains achieved even when ensembling fewer than 1% of
tokens.
SAFE, a selective ensembling framework for large language models, improves long-form generation by considering tokenization mismatch and consensus in probability distributions, leading to better accuracy and efficiency.
Large Language Models (LLMs) ensembling next-token probability distributions long-form generation tokenization mismatch consensus probability sharpening MATH500 BBH