Large Language Models for Compiler Optimization
We explore the novel application of Large Language Models to code
optimization. We present a 7B-parameter transformer model trained from scratch
to optimize LLVM assembly for code size. The model takes as input unoptimized
assembly and outputs a list of compiler options to best optimize the program.
Crucially, during training, we ask the model to predict the instruction counts
before and after optimization, and the optimized code itself. These auxiliary
learning tasks significantly improve the optimization performance of the model
and improve the model's depth of understanding.
  We evaluate on a large suite of test programs. Our approach achieves a 3.0%
improvement in reducing instruction counts over the compiler, outperforming two
state-of-the-art baselines that require thousands of compilations. Furthermore,
the model shows surprisingly strong code reasoning abilities, generating
compilable code 91% of the time and perfectly emulating the output of the
compiler 70% of the time.
65026e90cccc7b28a3660451
Chris Cummins
admin_assigned
2023-09-14T07:33:07.487Z
63d32301c20657d2434d2fcb
https://cdn-avatars.huggingface.co/v1/production/uploads/1674781395584-noauth.jpeg
Chris Cummins
user
ChrisCummins
65026e90cccc7b28a3660452
Volker Seeker
65026e90cccc7b28a3660453
Dejan Grubisic
65026e90cccc7b28a3660454
Mostafa Elhoushi
admin_assigned
2023-09-14T07:33:31.367Z
63c9725ebedad7e2bf160bdc
https://cdn-avatars.huggingface.co/v1/production/uploads/63c9725ebedad7e2bf160bdc/wzPuyhOXCYBNGwZDshbnL.jpeg
Mostafa Elhoushi
user
melhoushi
65026e90cccc7b28a3660455
Youwei Liang
admin_assigned
2023-09-14T07:33:39.520Z
64a475aa33ded98516c6a37b
/avatars/3abf5f8c7f56cc8612ab590f1b75c1d5.svg
Youwei Liang
user
youweiliang
65026e90cccc7b28a3660456
Baptiste Roziere
admin_assigned
2023-09-14T07:33:47.267Z
62d063dac375d0c84255b9a1
/avatars/de0fc34bad8c761210c0895ebfa4feba.svg
Baptiste Roziere
user
broz
65026e90cccc7b28a3660457
Jonas Gehring
admin_assigned
2023-09-14T07:33:54.909Z
6327890128396be9bfec7995
/avatars/5216ebcee5c41990a57e7a29fe114d6c.svg
Jonas Gehring
user
jgehring
65026e90cccc7b28a3660458
Fabian Gloeckle
admin_assigned
2023-09-14T07:34:02.396Z
64eca847971abfd26fe965be
/avatars/c53e119c6319768ffedc10442d4830ab.svg
Fabian Gloeckle
user
faaabian
65026e90cccc7b28a3660459
Kim Hazelwood
65026e90cccc7b28a366045a
Gabriel Synnaeve
admin_assigned
2023-09-14T07:34:19.086Z
630eac7931970d1cd4fbacf2
/avatars/b7ccbddfa745db854dc342be1327cd53.svg
Gabriel Synnaeve
user
gsynnaeve
65026e90cccc7b28a366045b
Hugh Leather
Large Language Models for Compiler Optimization
We explore the novel application of Large Language Models to code
optimization. We present a 7B-parameter transformer model trained from scratch
to optimize LLVM assembly for code size. The model takes as input unoptimized
assembly and outputs a list of compiler options to best optimize the program.
Crucially, during training, we ask the model to predict the instruction counts
before and after optimization, and the optimized code itself. These auxiliary
learning tasks significantly improve the optimization performance of the model
and improve the model's depth of understanding.
  We evaluate on a large suite of test programs. Our approach achieves a 3.0%
improvement in reducing instruction counts over the compiler, outperforming two
state-of-the-art baselines that require thousands of compilations. Furthermore,
the model shows surprisingly strong code reasoning abilities, generating
compilable code 91% of the time and perfectly emulating the output of the
compiler 70% of the time.
A transformer model trained to optimize LLVM assembly code size significantly improves compiler performance with auxiliary learning tasks, achieving better results than baselines and demonstrating strong code reasoning abilities.
Large Language Models transformer model LLVM assembly instruction counts code reasoning compilable code