SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models
Current large language models (LLMs) and spoken language models (SLMs) begin
thinking and taking actions only after the user has finished their turn. This
prevents the model from interacting during the user's turn and can lead to high
response latency while it waits to think. Consequently, thinking after
receiving the full input is not suitable for speech-to-speech interaction,
where real-time, low-latency exchange is important. We address this by noting
that humans naturally "think while listening." In this paper, we propose
SHANKS, a general inference framework that enables SLMs to generate unspoken
chain-of-thought reasoning while listening to the user input. SHANKS streams
the input speech in fixed-duration chunks and, as soon as a chunk is received,
generates unspoken reasoning based on all previous speech and reasoning, while
the user continues speaking. SHANKS uses this unspoken reasoning to decide
whether to interrupt the user and to make tool calls to complete the task. We
demonstrate that SHANKS enhances real-time user-SLM interaction in two
scenarios: (1) when the user is presenting a step-by-step solution to a math
problem, SHANKS can listen, reason, and interrupt when the user makes a
mistake, achieving 37.1% higher interruption accuracy than a baseline that
interrupts without thinking; and (2) in a tool-augmented dialogue, SHANKS can
complete 56.9% of the tool calls before the user finishes their turn. Overall,
SHANKS moves toward models that keep thinking throughout the conversation, not
only after a turn ends. Animated illustrations of Shanks can be found at
https://d223302.github.io/SHANKS/
68e70dc97ae125f9582e68c8
Cheng-Han Chiang
68e70dc97ae125f9582e68c9
Xiaofei Wang
68e70dc97ae125f9582e68ca
Linjie Li
68e70dc97ae125f9582e68cb
Chung-Ching Lin
68e70dc97ae125f9582e68cc
Kevin Lin
68e70dc97ae125f9582e68cd
Shujie Liu
68e70dc97ae125f9582e68ce
Zhendong Wang
68e70dc97ae125f9582e68cf
Zhengyuan Yang
68e70dc97ae125f9582e68d0
Hung-yi Lee
68e70dc97ae125f9582e68d1
Lijuan Wang
SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models
Current large language models (LLMs) and spoken language models (SLMs) begin
thinking and taking actions only after the user has finished their turn. This
prevents the model from interacting during the user's turn and can lead to high
response latency while it waits to think. Consequently, thinking after
receiving the full input is not suitable for speech-to-speech interaction,
where real-time, low-latency exchange is important. We address this by noting
that humans naturally "think while listening." In this paper, we propose
SHANKS, a general inference framework that enables SLMs to generate unspoken
chain-of-thought reasoning while listening to the user input. SHANKS streams
the input speech in fixed-duration chunks and, as soon as a chunk is received,
generates unspoken reasoning based on all previous speech and reasoning, while
the user continues speaking. SHANKS uses this unspoken reasoning to decide
whether to interrupt the user and to make tool calls to complete the task. We
demonstrate that SHANKS enhances real-time user-SLM interaction in two
scenarios: (1) when the user is presenting a step-by-step solution to a math
problem, SHANKS can listen, reason, and interrupt when the user makes a
mistake, achieving 37.1% higher interruption accuracy than a baseline that
interrupts without thinking; and (2) in a tool-augmented dialogue, SHANKS can
complete 56.9% of the tool calls before the user finishes their turn. Overall,
SHANKS moves toward models that keep thinking throughout the conversation, not
only after a turn ends. Animated illustrations of Shanks can be found at
https://d223302.github.io/SHANKS/
SHANKS, a general inference framework, enables spoken language models to generate unspoken reasoning while listening to user input, enhancing real-time interaction and task completion.
SHANKS spoken language models unspoken chain-of-thought reasoning real-time interaction tool calls interruption accuracy