MPIrigen: MPI Code Generation through Domain-Specific Language Models
The imperative need to scale computation across numerous nodes highlights the
significance of efficient parallel computing, particularly in the realm of
Message Passing Interface (MPI) integration. The challenging parallel
programming task of generating MPI-based parallel programs has remained
unexplored. This study first investigates the performance of state-of-the-art
language models in generating MPI-based parallel programs. Findings reveal that
widely used models such as GPT-3.5 and PolyCoder (specialized multi-lingual
code models) exhibit notable performance degradation, when generating MPI-based
programs compared to general-purpose programs. In contrast, domain-specific
models such as MonoCoder, which are pretrained on MPI-related programming
languages of C and C++, outperform larger models. Subsequently, we introduce a
dedicated downstream task of MPI-based program generation by fine-tuning
MonoCoder on HPCorpusMPI. We call the resulting model as MPIrigen. We propose
an innovative preprocessing for completion only after observing the whole code,
thus enabling better completion with a wider context. Comparative analysis
against GPT-3.5 zero-shot performance, using a novel HPC-oriented evaluation
method, demonstrates that MPIrigen excels in generating accurate MPI functions
up to 0.8 accuracy in location and function predictions, and with more than 0.9
accuracy for argument predictions. The success of this tailored solution
underscores the importance of domain-specific fine-tuning in optimizing
language models for parallel computing code generation, paving the way for a
new generation of automatic parallelization tools. The sources of this work are
available at our GitHub MPIrigen repository:
https://github.com/Scientific-Computing-Lab-NRCN/MPI-rigen
65cd99f61b2738e61ebbce18
Nadav Schneider
claimed_verified
2024-02-15T17:28:28.001Z
65ce0b4a03a8179f5da5d4ef
/avatars/c0b68efb885486c5180d1d69c4e317ac.svg
Nadav Schneider
user
nadavsc
65cd99f61b2738e61ebbce19
Niranjan Hasabnis
extracted_pending
2024-02-15T04:58:30.887Z
650dd3a6c945dfc9385e52d7
/avatars/c9a41b4cb4733f136d4a65e3d33693c4.svg
Niranjan Hasabnis
user
nhasabni
65cd99f61b2738e61ebbce1a
Vy A. Vo
extracted_pending
2024-02-15T04:58:30.887Z
62fabd8920be129c47683fe4
/avatars/3dd320882633a648facaccb80c5b583d.svg
Vy Vo
user
vyaivo
65cd99f61b2738e61ebbce1b
Tal Kadosh
65cd99f61b2738e61ebbce1c
Neva Krien
extracted_pending
2024-02-15T04:58:30.887Z
63a325778c0c89dcae33484b
/avatars/3eca79d6a0a00652c906e30a83d4f8af.svg
נבו קרין
user
nevakrien
65cd99f61b2738e61ebbce1d
Mihai Capotă
65cd99f61b2738e61ebbce1e
Abdul Wasay
65cd99f61b2738e61ebbce1f
Guy Tamir
65cd99f61b2738e61ebbce20
Ted Willke
extracted_confirmed
2024-02-16T01:44:55.610Z
65cebdbce0723b01a3774e8c
/avatars/5ac01078ab89b3525fbb5d0069e16607.svg
Ted Willke
user
tlwillke
65cd99f61b2738e61ebbce21
Nesreen Ahmed
extracted_confirmed
2025-01-07T07:13:11.063Z
663e588ccca86c8371a913d9
https://cdn-avatars.huggingface.co/v1/production/uploads/663e588ccca86c8371a913d9/wgVGw5U77cqwtjovhsXyR.jpeg
Nesreen Ahmed
user
nkahmed
65cd99f61b2738e61ebbce22
Yuval Pinter
claimed_verified
2024-02-15T17:28:23.651Z
65562ab891ec2c213fba90d9
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/pywmO-m8WhvkQFTqZmCum.jpeg
Yuval Pinter
user
yuvalpinter
65cd99f61b2738e61ebbce23
Timothy Mattson
65cd99f61b2738e61ebbce24
Gal Oren
claimed_verified
2024-02-15T17:28:25.926Z
65ce14f4167be8a5f3e7646e
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/nzeXCf5QQKLlFaich-d1u.jpeg
Gal Oren
user
GalOren8
MPIrigen: MPI Code Generation through Domain-Specific Language Models
The imperative need to scale computation across numerous nodes highlights the
significance of efficient parallel computing, particularly in the realm of
Message Passing Interface (MPI) integration. The challenging parallel
programming task of generating MPI-based parallel programs has remained
unexplored. This study first investigates the performance of state-of-the-art
language models in generating MPI-based parallel programs. Findings reveal that
widely used models such as GPT-3.5 and PolyCoder (specialized multi-lingual
code models) exhibit notable performance degradation, when generating MPI-based
programs compared to general-purpose programs. In contrast, domain-specific
models such as MonoCoder, which are pretrained on MPI-related programming
languages of C and C++, outperform larger models. Subsequently, we introduce a
dedicated downstream task of MPI-based program generation by fine-tuning
MonoCoder on HPCorpusMPI. We call the resulting model as MPIrigen. We propose
an innovative preprocessing for completion only after observing the whole code,
thus enabling better completion with a wider context. Comparative analysis
against GPT-3.5 zero-shot performance, using a novel HPC-oriented evaluation
method, demonstrates that MPIrigen excels in generating accurate MPI functions
up to 0.8 accuracy in location and function predictions, and with more than 0.9
accuracy for argument predictions. The success of this tailored solution
underscores the importance of domain-specific fine-tuning in optimizing
language models for parallel computing code generation, paving the way for a
new generation of automatic parallelization tools. The sources of this work are
available at our GitHub MPIrigen repository:
https://github.com/Scientific-Computing-Lab-NRCN/MPI-rigen
Domain-specific fine-tuning of MonoCoder on MPI-related code significantly improves the generation of MPI-based parallel programs compared to general-purpose models.
GPT-3.5 PolyCoder MonoCoder HPCorpusMPI MPIrigen MPI-based programs parallel computing code generation domain-specific fine-tuning HPC-oriented evaluation MPI functions argument predictions