Voice Evaluation of Reasoning Ability: Diagnosing the Modality-Induced
  Performance Gap
We present Voice Evaluation of Reasoning Ability (VERA), a benchmark for
evaluating reasoning ability in voice-interactive systems under real-time
conversational constraints. VERA comprises 2,931 voice-native episodes derived
from established text benchmarks and organized into five tracks (Math, Web,
Science, Long-Context, Factual). Each item is adapted for speech interaction
while preserving reasoning difficulty. VERA enables direct text-voice
comparison within model families and supports analysis of how architectural
choices affect reliability. We assess 12 contemporary voice systems alongside
strong text baselines and observe large, consistent modality gaps: on
competition mathematics a leading text model attains 74.8% accuracy while its
voice counterpart reaches 6.1%; macro-averaged across tracks the best text
models achieve 54.0% versus 11.3% for voice. Latency-accuracy analyses reveal a
low-latency plateau, where fast voice systems cluster around ~10% accuracy,
while approaching text performance requires sacrificing real-time interaction.
Diagnostic experiments indicate that common mitigations are insufficient.
Increasing "thinking time" yields negligible gains; a decoupled cascade that
separates reasoning from narration improves accuracy but still falls well short
of text and introduces characteristic grounding/consistency errors. Failure
analyses further show distinct error signatures across native streaming,
end-to-end, and cascade designs. VERA provides a reproducible testbed and
targeted diagnostics for architectures that decouple thinking from speaking,
offering a principled way to measure progress toward real-time voice assistants
that are both fluent and reliably reasoned.
68dc885c4159d1f2418f9a23
Yueqian Lin
claimed_verified
2025-10-01T10:22:20.736Z
64b5198c25882acb62fb77ef
https://cdn-avatars.huggingface.co/v1/production/uploads/64b5198c25882acb62fb77ef/HX9pfMEPQlfjvSAgSLplY.png
Yueqian Lin
user
linyueqian
68dc885c4159d1f2418f9a24
Zhengmian Hu
68dc885c4159d1f2418f9a25
Qinsi Wang
68dc885c4159d1f2418f9a26
Yudong Liu
68dc885c4159d1f2418f9a27
Hengfan Zhang
68dc885c4159d1f2418f9a28
Jayakumar Subramanian
68dc885c4159d1f2418f9a29
Nikos Vlassis
68dc885c4159d1f2418f9a2a
Hai Helen Li
68dc885c4159d1f2418f9a2b
Yiran Chen
Voice Evaluation of Reasoning Ability: Diagnosing the Modality-Induced
  Performance Gap
We present Voice Evaluation of Reasoning Ability (VERA), a benchmark for
evaluating reasoning ability in voice-interactive systems under real-time
conversational constraints. VERA comprises 2,931 voice-native episodes derived
from established text benchmarks and organized into five tracks (Math, Web,
Science, Long-Context, Factual). Each item is adapted for speech interaction
while preserving reasoning difficulty. VERA enables direct text-voice
comparison within model families and supports analysis of how architectural
choices affect reliability. We assess 12 contemporary voice systems alongside
strong text baselines and observe large, consistent modality gaps: on
competition mathematics a leading text model attains 74.8% accuracy while its
voice counterpart reaches 6.1%; macro-averaged across tracks the best text
models achieve 54.0% versus 11.3% for voice. Latency-accuracy analyses reveal a
low-latency plateau, where fast voice systems cluster around ~10% accuracy,
while approaching text performance requires sacrificing real-time interaction.
Diagnostic experiments indicate that common mitigations are insufficient.
Increasing "thinking time" yields negligible gains; a decoupled cascade that
separates reasoning from narration improves accuracy but still falls well short
of text and introduces characteristic grounding/consistency errors. Failure
analyses further show distinct error signatures across native streaming,
end-to-end, and cascade designs. VERA provides a reproducible testbed and
targeted diagnostics for architectures that decouple thinking from speaking,
offering a principled way to measure progress toward real-time voice assistants
that are both fluent and reliably reasoned.
VERA is a benchmark for evaluating reasoning ability in voice-interactive systems, revealing significant performance gaps compared to text models and highlighting challenges in real-time interaction.
voice-interactive systems reasoning ability real-time conversational constraints voice-native episodes text benchmarks speech interaction text-voice comparison architectural choices reliability latency-accuracy analyses low-latency plateau thinking time decoupled cascade error signatures native streaming end-to-end cascade designs real-time voice assistants fluent reasoning