Audio-FLAN: A Preliminary Release
Recent advancements in audio tokenization have significantly enhanced the
integration of audio capabilities into large language models (LLMs). However,
audio understanding and generation are often treated as distinct tasks,
hindering the development of truly unified audio-language models. While
instruction tuning has demonstrated remarkable success in improving
generalization and zero-shot learning across text and vision, its application
to audio remains largely unexplored. A major obstacle is the lack of
comprehensive datasets that unify audio understanding and generation. To
address this, we introduce Audio-FLAN, a large-scale instruction-tuning dataset
covering 80 diverse tasks across speech, music, and sound domains, with over
100 million instances. Audio-FLAN lays the foundation for unified
audio-language models that can seamlessly handle both understanding (e.g.,
transcription, comprehension) and generation (e.g., speech, music, sound) tasks
across a wide range of audio domains in a zero-shot manner. The Audio-FLAN
dataset is available on HuggingFace and GitHub and will be continuously
updated.
67bd42386959e61abd265a9b
Liumeng Xue
admin_assigned
2025-02-25T16:02:19.831Z
6290e961473e457463a53248
https://cdn-avatars.huggingface.co/v1/production/uploads/6290e961473e457463a53248/-58Dp5uHvdjs9yOupAMs0.jpeg
Liumeng Xue
user
lmxue
67bd42386959e61abd265a9c
Ziya Zhou
admin_assigned
2025-02-25T16:04:58.842Z
64191c925d6f3d15c65137b5
/avatars/0e6a5fabf11904b9c31073ad1e10f6c6.svg
Ziya Zhou
user
DangeZy
67bd42386959e61abd265a9d
Jiahao Pan
67bd42386959e61abd265a9e
Zixuan Li
67bd42386959e61abd265a9f
Shuai Fan
admin_assigned
2025-02-25T16:05:22.700Z
669a705f7c6ce7348dae9dcd
/avatars/b042a8ea45848e3c9c77ce532286692f.svg
Shuai Fan
user
Micro20
67bd42386959e61abd265aa0
Yinghao Ma
admin_assigned
2025-02-25T16:05:28.773Z
6410665d5364a661bee22524
/avatars/f1cb0e07f36933187ceccbd5dcbeff79.svg
Yinghao Ma
user
nicolaus625
67bd42386959e61abd265aa1
Sitong Cheng
67bd42386959e61abd265aa2
Dongchao Yang
admin_assigned
2025-02-25T16:05:44.137Z
63c7636b656e7822e23e6f6b
/avatars/41bfd5e1ce6daab6058eacfd33c7a268.svg
Dongchao Yang
user
Dongchao
67bd42386959e61abd265aa3
Haohan Guo
admin_assigned
2025-02-25T16:05:50.198Z
63783d9d84318944acd305c4
/avatars/cafd4fd0287e77cc6ebf37c8c8509174.svg
Haohan Guo
user
hhguo
67bd42386959e61abd265aa4
Yujia Xiao
claimed_verified
2025-03-04T13:00:34.602Z
674836767b7151c3ff30f865
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/jcwK5NW-efhCt8s2TE6vK.png
Yujia Xiao
user
Yogurt928
67bd42386959e61abd265aa5
Xinsheng Wang
admin_assigned
2025-02-25T16:06:03.447Z
64cc69be8174e45ae076393e
/avatars/6fd566535cadbed63e1f956587157d13.svg
Xinsheng Wang
user
wangxso
67bd42386959e61abd265aa6
Zixuan Shen
67bd42386959e61abd265aa7
Chuanbo Zhu
admin_assigned
2025-02-25T16:06:27.024Z
6721db0fab7602a59648aec6
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/bhWBJ26b_94mZICgi3jVZ.png
zhu chuanbo
user
zhuchb
67bd42386959e61abd265aa8
Xinshen Zhang
admin_assigned
2025-02-25T16:06:40.621Z
668b90710b04331a0bbacbb0
/avatars/1a457fc4b4d242a9a7104ba38d5c2467.svg
ZHANG Xinshen
user
Ashaire
67bd42386959e61abd265aa9
Tianchi Liu
admin_assigned
2025-02-25T16:06:47.569Z
6756ae32298969739a42d5f9
https://cdn-avatars.huggingface.co/v1/production/uploads/6756ae32298969739a42d5f9/XethqlkSvZUi3ZFnNLqaF.jpeg
Tianchi Liu
user
liu-tianchi
67bd42386959e61abd265aaa
Ruibin Yuan
admin_assigned
2025-02-25T16:06:54.742Z
5fd6f670053c8345eddc1b68
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
Ruibin Yuan
user
a43992899
67bd42386959e61abd265aab
Zeyue Tian
claimed_verified
2025-03-16T21:18:50.821Z
6504409b52ca06fef957266d
/avatars/ac96eab7318cd8d472601cf7e7c50c97.svg
Zeyue
user
Zeyue7
67bd42386959e61abd265aac
Haohe Liu
admin_assigned
2025-02-25T16:07:13.402Z
6155245d1c762d4d61b51d5d
https://cdn-avatars.huggingface.co/v1/production/uploads/1677233708065-6155245d1c762d4d61b51d5d.png
haoheliu
user
haoheliu
67bd42386959e61abd265aad
Emmanouil Benetos
admin_assigned
2025-02-25T16:07:19.685Z
66b0a0839797b94f001ed874
https://cdn-avatars.huggingface.co/v1/production/uploads/66b0a0839797b94f001ed874/BBtlJdQNyzQX10chnU81s.jpeg
Emmanouil Benetos
user
emmanouilb
67bd42386959e61abd265aae
Ge Zhang
admin_assigned
2025-02-25T16:07:26.226Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
67bd42386959e61abd265aaf
Yike Guo
admin_assigned
2025-02-25T16:07:36.243Z
64762b325f70f9b2d0ade28e
/avatars/1f37382724b475ece805f943a8858acd.svg
Yike Guo
user
SuaLily
67bd42386959e61abd265ab0
Wei Xue
Audio-FLAN: A Preliminary Release
Recent advancements in audio tokenization have significantly enhanced the
integration of audio capabilities into large language models (LLMs). However,
audio understanding and generation are often treated as distinct tasks,
hindering the development of truly unified audio-language models. While
instruction tuning has demonstrated remarkable success in improving
generalization and zero-shot learning across text and vision, its application
to audio remains largely unexplored. A major obstacle is the lack of
comprehensive datasets that unify audio understanding and generation. To
address this, we introduce Audio-FLAN, a large-scale instruction-tuning dataset
covering 80 diverse tasks across speech, music, and sound domains, with over
100 million instances. Audio-FLAN lays the foundation for unified
audio-language models that can seamlessly handle both understanding (e.g.,
transcription, comprehension) and generation (e.g., speech, music, sound) tasks
across a wide range of audio domains in a zero-shot manner. The Audio-FLAN
dataset is available on HuggingFace and GitHub and will be continuously
updated.
A new large-scale dataset, Audio-FLAN, supports unified audio-language models by covering diverse tasks across speech, music, and sound for both understanding and generation.
audio tokenization large language models (LLMs) audio understanding audio generation instruction tuning zero-shot learning Audio-FLAN transcription comprehension speech music sound