BioCoder: A Benchmark for Bioinformatics Code Generation with Contextual
  Pragmatic Knowledge
Pre-trained language models like ChatGPT have significantly improved code
generation. As these models scale up, there is an increasing need for the
output to handle more intricate tasks. Moreover, in bioinformatics, generating
functional programs poses additional notable challenges due to the amount of
domain knowledge, the need for complicated data operations, and intricate
functional dependencies between the operations. Here, we present BioCoder, a
benchmark developed to evaluate existing pre-trained models in generating
bioinformatics code. In relation to function-code generation, BioCoder covers
potential package dependencies, class declarations, and global variables. It
incorporates 1026 functions and 1243 methods in Python and Java from GitHub and
253 examples from the Rosalind Project. BioCoder incorporates a fuzz-testing
framework for evaluation, and we have applied it to evaluate many models
including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+,
InstructCodeT5+, and ChatGPT. Our detailed analysis of these models emphasizes
the importance of domain knowledge, pragmatic code generation, and contextual
understanding. Our dataset, benchmark, Docker images, and scripts required for
testing are all available at https://github.com/gersteinlab/biocoder.
64f1581a3bdfd057d4567a17
Xiangru Tang
admin_assigned
2023-09-01T12:35:58.595Z
64709e80d9360cd9d8e7073d
/avatars/ec37977b9a10e823e0d5bb73d152f253.svg
tang xiangrun
user
stxrun
64f1581a3bdfd057d4567a18
Bill Qian
admin_assigned
2023-09-01T12:36:33.046Z
641fa2e0863b87326f4a7fd5
/avatars/7b29c20f942d83c08f63f04a85c15c13.svg
Bill Qian
user
lilbillbiscuit
64f1581a3bdfd057d4567a19
Rick Gao
admin_assigned
2023-09-01T12:36:41.741Z
649e6ead35032bf979eded1c
/avatars/c2b5bc24abd5d28bcf29b01f185fbe67.svg
Rick Gao
user
rikkii
64f1581a3bdfd057d4567a1a
Jiakang Chen
64f1581a3bdfd057d4567a1b
Xinyun Chen
64f1581a3bdfd057d4567a1c
Mark Gerstein
BioCoder: A Benchmark for Bioinformatics Code Generation with Contextual
  Pragmatic Knowledge
Pre-trained language models like ChatGPT have significantly improved code
generation. As these models scale up, there is an increasing need for the
output to handle more intricate tasks. Moreover, in bioinformatics, generating
functional programs poses additional notable challenges due to the amount of
domain knowledge, the need for complicated data operations, and intricate
functional dependencies between the operations. Here, we present BioCoder, a
benchmark developed to evaluate existing pre-trained models in generating
bioinformatics code. In relation to function-code generation, BioCoder covers
potential package dependencies, class declarations, and global variables. It
incorporates 1026 functions and 1243 methods in Python and Java from GitHub and
253 examples from the Rosalind Project. BioCoder incorporates a fuzz-testing
framework for evaluation, and we have applied it to evaluate many models
including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+,
InstructCodeT5+, and ChatGPT. Our detailed analysis of these models emphasizes
the importance of domain knowledge, pragmatic code generation, and contextual
understanding. Our dataset, benchmark, Docker images, and scripts required for
testing are all available at https://github.com/gersteinlab/biocoder.
BioCoder evaluates pre-trained models for bioinformatics code generation, focusing on domain knowledge and complex functional dependencies.
pre-trained language models bioinformatics code generation domain knowledge data operations functional dependencies function-code generation package dependencies class declarations global variables fuzz-testing framework InCoder CodeGen CodeGen2 SantaCoder StarCoder StarCoder+ InstructCodeT5+ ChatGPT