Mind the Gap! Static and Interactive Evaluations of Large Audio Models
As AI chatbots become ubiquitous, voice interaction presents a compelling way
to enable rapid, high-bandwidth communication for both semantic and social
signals. This has driven research into Large Audio Models (LAMs) to power
voice-native experiences. However, aligning LAM development with user goals
requires a clear understanding of user needs and preferences to establish
reliable progress metrics. This study addresses these challenges by introducing
an interactive approach to evaluate LAMs and collecting 7,500 LAM interactions
from 484 participants. Through topic modeling of user queries, we identify
primary use cases for audio interfaces. We then analyze user preference
rankings and qualitative feedback to determine which models best align with
user needs. Finally, we evaluate how static benchmarks predict interactive
performance - our analysis reveals no individual benchmark strongly correlates
with interactive results (tau leq 0.33 for all benchmarks). While combining
multiple coarse-grained features yields modest predictive power (R^2=0.30),
only two out of twenty datasets on spoken question answering and age prediction
show significantly positive correlations. This suggests a clear need to develop
LAM evaluations that better correlate with user preferences.
67be33ffe30b2f126c599413
Minzhi Li
67be33ffe30b2f126c599414
William Barr Held
claimed_verified
2025-02-26T08:28:38.756Z
632116accafe12f481a473cb
https://cdn-avatars.huggingface.co/v1/production/uploads/1666207676653-632116accafe12f481a473cb.jpeg
Will Held
user
WillHeld
67be33ffe30b2f126c599415
Michael J Ryan
claimed_verified
2025-03-04T08:52:35.827Z
63878fa2e40346f68ede7fc4
https://cdn-avatars.huggingface.co/v1/production/uploads/63878fa2e40346f68ede7fc4/MOdQbApvhwpVzPo5FWLVV.jpeg
Michael Ryan
user
MichaelR207
67be33ffe30b2f126c599416
Kunat Pipatanakul
claimed_verified
2025-07-15T19:14:03.648Z
62d192c2d50433c35eb1b48e
https://cdn-avatars.huggingface.co/v1/production/uploads/62d192c2d50433c35eb1b48e/VjmDu8GOIuLuQNBQdQLLS.png
Kunat Pipatanakul
user
kunato
67be33ffe30b2f126c599417
Potsawee Manakul
67be33ffe30b2f126c599418
Hao Zhu
claimed_verified
2025-03-03T16:07:29.874Z
61aa376688c20eebf1e8deb3
/avatars/7c11dcb232c73547d7d87834be287822.svg
Hao Zhu
user
ProKil
67be33ffe30b2f126c599419
Diyi Yang
Mind the Gap! Static and Interactive Evaluations of Large Audio Models
As AI chatbots become ubiquitous, voice interaction presents a compelling way
to enable rapid, high-bandwidth communication for both semantic and social
signals. This has driven research into Large Audio Models (LAMs) to power
voice-native experiences. However, aligning LAM development with user goals
requires a clear understanding of user needs and preferences to establish
reliable progress metrics. This study addresses these challenges by introducing
an interactive approach to evaluate LAMs and collecting 7,500 LAM interactions
from 484 participants. Through topic modeling of user queries, we identify
primary use cases for audio interfaces. We then analyze user preference
rankings and qualitative feedback to determine which models best align with
user needs. Finally, we evaluate how static benchmarks predict interactive
performance - our analysis reveals no individual benchmark strongly correlates
with interactive results (tau leq 0.33 for all benchmarks). While combining
multiple coarse-grained features yields modest predictive power (R^2=0.30),
only two out of twenty datasets on spoken question answering and age prediction
show significantly positive correlations. This suggests a clear need to develop
LAM evaluations that better correlate with user preferences.
Research evaluates Large Audio Models through user interactions, topic modeling, and preference analysis, revealing the need for better correlation between static benchmarks and user preferences.
Large Audio Models LAMs topic modeling user preference rankings qualitative feedback interactive performance spoken question answering age prediction