arxiv:2510.10977

Revisiting Model Interpolation for Efficient Reasoning

Published on Oct 13

· Submitted by

Taki WU on Oct 16

The University of Hong Kong

Upvote

Authors:

Taiqiang Wu ,

Abstract

Model merging, typically on Instruct and Thinking models, has shown remarkable performance for efficient reasoning. In this paper, we systematically revisit the simplest merging method that interpolates two weights directly. Particularly, we observe that model interpolation follows a three-stage evolutionary paradigm with distinct behaviors on the reasoning trajectory. These dynamics provide a principled guide for navigating the performance-cost trade-off. Empirical results demonstrate that a strategically interpolated model surprisingly surpasses sophisticated model merging baselines on both efficiency and effectiveness. We further validate our findings with extensive ablation studies on model layers, modules, and decoding strategies. Ultimately, this work demystifies model interpolation and offers a practical framework for crafting models with precisely targeted reasoning capabilities. Code is available at https://github.com/wutaiqiang/MI{Github}.

View arXiv page View PDF GitHub 6 Add to collection

Community

taki555

Paper author Paper submitter 14 days ago

Revisit the simple(est) method for efficient reasoning: Model Interpolation on Instruct and Thinking variants. This paper finds that model interpolation follows a three-stage evolutionary paradigm with distinct behaviors on the reasoning trajectory.

Code: https://github.com/wutaiqiang/MI

taki555

Paper author Paper submitter 14 days ago

Stage #1. Corresponding to λ ∈ [0, 0.4) for the Qwen3-4B model. In this initial phase, the merged model is dominated by the Instruct model but begins to incorporate traits from the Thinking model, thus generating longer outputs without adopting an explicit thinking process.

Stage #2. Corresponding to λ ∈ [0.4, 0.6] for Qwen3-4B. In this stage, the reasoning pattern following Thinking models rapidly emerges, leading to largely increased Mean@k and gently increased Pass@k and Token #N. This stage marks a
critical and dramatic phase transition.

Stage #3. Corresponding to λ ∈ (0.6, 1.0] for Qwen3-4B. In this final stage, the merged model converges to the pure Thinking model, with continuously increasing Token #N and a slight change in Pass@k and Mean@k.