MuPT: A Generative Symbolic Music Pretrained Transformer
In this paper, we explore the application of Large Language Models (LLMs) to
the pre-training of music. While the prevalent use of MIDI in music modeling is
well-established, our findings suggest that LLMs are inherently more compatible
with ABC Notation, which aligns more closely with their design and strengths,
thereby enhancing the model's performance in musical composition. To address
the challenges associated with misaligned measures from different tracks during
generation, we propose the development of a Synchronized
Multi-Track ABC Notation (SMT-ABC Notation),
which aims to preserve coherence across multiple musical tracks. Our
contributions include a series of models capable of handling up to 8192 tokens,
covering 90\% of the symbolic music data in our training set. Furthermore, we
explore the implications of the Symbolic Music
Scaling Law (SMS Law) on model performance. The results
indicate a promising direction for future research in music generation,
offering extensive resources for community-led research through our open-source
contributions.
6615ef8c45336ca7744f3583
Xingwei Qu
6615ef8c45336ca7744f3584
Yuelin Bai
6615ef8c45336ca7744f3585
Yinghao Ma
6615ef8c45336ca7744f3586
Ziya Zhou
6615ef8c45336ca7744f3587
Ka Man Lo
claimed_verified
2024-06-27T09:47:41.055Z
649ad817c5d076065a514a41
https://cdn-avatars.huggingface.co/v1/production/uploads/649ad817c5d076065a514a41/WsGymqaiZFsx3jDtMCycl.jpeg
Ka Man Lo
user
kamanphoebe
6615ef8c45336ca7744f3588
Jiaheng Liu
claimed_verified
2025-01-05T23:03:54.527Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
6615ef8c45336ca7744f3589
Ruibin Yuan
claimed_verified
2025-03-12T08:57:32.770Z
5fd6f670053c8345eddc1b68
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
Ruibin Yuan
user
a43992899
6615ef8c45336ca7744f358a
Lejun Min
6615ef8c45336ca7744f358b
Xueling Liu
6615ef8c45336ca7744f358c
Tianyu Zhang
6615ef8c45336ca7744f358d
Xinrun Du
claimed_verified
2024-05-31T06:53:34.584Z
654907a4a1faff97850c4eff
/avatars/458c90151614bc7f116943b6e67d6b8a.svg
du
user
dododododo
6615ef8c45336ca7744f358e
Shuyue Guo
6615ef8c45336ca7744f358f
Yiming Liang
6615ef8c45336ca7744f3590
Yizhi Li
claimed_verified
2024-05-31T06:53:36.464Z
6382252f54421460665ec501
https://cdn-avatars.huggingface.co/v1/production/uploads/6382252f54421460665ec501/gW9fev3T5QPcNq4f9hqB1.jpeg
Yizhi Li
user
yizhilll
6615ef8c45336ca7744f3591
Shangda Wu
6615ef8c45336ca7744f3592
Junting Zhou
6615ef8c45336ca7744f3593
Tianyu Zheng
claimed_verified
2024-04-10T10:57:49.141Z
64ab99dcb76bfd863eba64c1
https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
TY.Zheng
user
aaabiao
6615ef8c45336ca7744f3594
Ziyang Ma
6615ef8c45336ca7744f3595
Fengze Han
claimed_verified
2024-04-10T10:58:00.703Z
65d22ca28842349c64e3e7f1
/avatars/38720c9fa9dbf9cf1a73cc4d418836e1.svg
Fengze Han
user
Fengze
6615ef8c45336ca7744f3596
Wei Xue
claimed_verified
2024-04-24T11:12:13.207Z
6628adb14277eae0da5eee28
/avatars/6cb41b80cc5e014e455dfc2a22682e64.svg
HKUST Audio
user
HKUST-Audio
6615ef8c45336ca7744f3597
Gus Xia
6615ef8c45336ca7744f3598
Emmanouil Benetos
claimed_verified
2024-08-05T10:24:11.931Z
66b0a0839797b94f001ed874
https://cdn-avatars.huggingface.co/v1/production/uploads/66b0a0839797b94f001ed874/BBtlJdQNyzQX10chnU81s.jpeg
Emmanouil Benetos
user
emmanouilb
6615ef8c45336ca7744f3599
Xiang Yue
claimed_verified
2024-05-27T07:21:02.656Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
6615ef8c45336ca7744f359a
Chenghua Lin
6615ef8c45336ca7744f359b
Xu Tan
claimed_verified
2024-08-28T13:47:24.072Z
5f1040b6e9d71719e3be71d2
/avatars/a2f28940236ae625ed3810ad62e343ff.svg
Xu Tan
user
xutan
6615ef8c45336ca7744f359c
Stephen W. Huang
claimed_verified
2024-10-13T20:26:00.069Z
65bb11cb00a03997849e9e85
/avatars/17022b0254192a837f4fe00d84389cda.svg
Wenhao Huang
user
StephenHuang
6615ef8c45336ca7744f359d
Wenhu Chen
6615ef8c45336ca7744f359e
Jie Fu
6615ef8c45336ca7744f359f
Ge Zhang
claimed_verified
2024-05-31T06:53:32.871Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
MuPT: A Generative Symbolic Music Pretrained Transformer
In this paper, we explore the application of Large Language Models (LLMs) to
the pre-training of music. While the prevalent use of MIDI in music modeling is
well-established, our findings suggest that LLMs are inherently more compatible
with ABC Notation, which aligns more closely with their design and strengths,
thereby enhancing the model's performance in musical composition. To address
the challenges associated with misaligned measures from different tracks during
generation, we propose the development of a Synchronized
Multi-Track ABC Notation (SMT-ABC Notation),
which aims to preserve coherence across multiple musical tracks. Our
contributions include a series of models capable of handling up to 8192 tokens,
covering 90\% of the symbolic music data in our training set. Furthermore, we
explore the implications of the Symbolic Music
Scaling Law (SMS Law) on model performance. The results
indicate a promising direction for future research in music generation,
offering extensive resources for community-led research through our open-source
contributions.
Large Language Models are adapted for music pre-training using ABC Notation instead of MIDI, leading to enhanced musical composition through synchronized multi-track handling and exploration of the Symbolic Music Scaling Law.
Large Language Models LLMs MIDI ABC Notation SMT-ABC Notation symbolic music Symbolic Music Scaling Law SMS Law