ML-Bench: Large Language Models Leverage Open-source Libraries for
  Machine Learning Tasks
Large language models have shown promising performance in code generation
benchmarks. However, a considerable divide exists between these benchmark
achievements and their practical applicability, primarily attributed to
real-world programming's reliance on pre-existing libraries. Instead of
evaluating LLMs to code from scratch, this work aims to propose a new
evaluation setup where LLMs use open-source libraries to finish machine
learning tasks. Therefore, we propose ML-Bench, an expansive benchmark
developed to assess the effectiveness of LLMs in leveraging existing functions
in open-source libraries. Consisting of 10044 samples spanning 130 tasks over
14 notable machine learning GitHub repositories. In this setting, given a
specific machine learning task instruction and the accompanying README in a
codebase, an LLM is tasked to generate code to accomplish the task. This
necessitates the comprehension of long and language-code interleaved documents,
as well as the understanding of complex cross-file code structures, introducing
new challenges. Notably, while GPT-4 exhibits remarkable improvement over other
LLMs, it manages to accomplish only 39.73\% of the tasks, leaving a huge space
for improvement. We address these challenges by proposing ML-Agent, designed to
effectively navigate the codebase, locate documentation, retrieve code, and
generate executable code. Empirical results demonstrate that ML-Agent, built
upon GPT-4, results in further improvements. Code, data, and models are
available at https://ml-bench.github.io/.
6556dcf347c93d37a885c33e
Yuliang Liu
admin_assigned
2023-11-17T16:28:44.735Z
64b74e149f5987572c9bf8d8
/avatars/5429714d53a1bfdf6a6c7f077b3b5cf6.svg
Yuliang Liu
user
YuliangLiu
6556dcf347c93d37a885c33f
Xiangru Tang
extracted_confirmed
2023-11-20T03:25:55.732Z
63357c608adfa81faf2ac180
/avatars/ae0314c644f882251baf59b9134fd36f.svg
Xiangru Tang
user
RTT1
6556dcf347c93d37a885c340
Zefan Cai
admin_assigned
2023-11-17T16:28:51.397Z
64b15284372d4340772a3dca
/avatars/417d5f1bc1bcb5e4d5de6169673c2cf7.svg
Zefan Cai
user
ZefanCai
6556dcf347c93d37a885c341
Junjie Lu
claimed_verified
2025-02-24T13:05:43.940Z
660e1bb6e089e44d2906a7e8
https://cdn-avatars.huggingface.co/v1/production/uploads/660e1bb6e089e44d2906a7e8/MJU8nHQ0xgiPzO8J6L8EC.jpeg
Junjie Lu
user
Lux0926
6556dcf347c93d37a885c342
Yichi Zhang
admin_assigned
2023-11-17T16:30:15.504Z
6369b1d456d1f93498130a8a
/avatars/8ec228aa6f171715652511f948765db9.svg
Yichi Zhang
user
594zyc
6556dcf347c93d37a885c343
Yanjun Shao
admin_assigned
2023-11-17T16:30:30.870Z
62970df979f193515da13dc0
https://cdn-avatars.huggingface.co/v1/production/uploads/62970df979f193515da13dc0/A-mgKIcgTXRJ54GCHswTq.jpeg
Yanjun Shao
user
super-dainiu
6556dcf347c93d37a885c344
Zexuan Deng
6556dcf347c93d37a885c345
Helan Hu
admin_assigned
2023-11-17T16:32:05.953Z
6491aec41e161baf7894b54b
/avatars/0af10e24669125e4329c003fc27b603e.svg
huhelan
user
hhhllan
6556dcf347c93d37a885c346
Zengxian Yang
6556dcf347c93d37a885c347
Kaikai An
claimed_verified
2024-10-23T07:37:40.004Z
66c89152d33e34fbc29497d7
/avatars/bbddabf6532393951c4759e5915a065b.svg
KaikaiAn
user
kkk-an
6556dcf347c93d37a885c348
Ruijun Huang
6556dcf347c93d37a885c349
Shuzheng Si
claimed_verified
2024-10-11T07:47:09.953Z
637c99bbfe115289cfedfb44
https://cdn-avatars.huggingface.co/v1/production/uploads/637c99bbfe115289cfedfb44/344NN9KKF_XXTlVYaGaMW.png
ssz
user
ssz1111
6556dcf347c93d37a885c34a
Sheng Chen
6556dcf347c93d37a885c34b
Haozhe Zhao
claimed_verified
2024-07-22T08:43:47.916Z
63ff09a02098b9ad105a09f6
/avatars/4409ca5d320050cf4c3df05962c7ff58.svg
Hans Zhao
user
BleachNick
6556dcf347c93d37a885c34c
Zhengliang Li
6556dcf347c93d37a885c34d
Liang Chen
claimed_verified
2024-03-12T12:06:38.496Z
61b0a4ce1b3d95b3d1ed9251
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wwjr26vdudX5KYVTb8Q0a.png
Liang Chen
user
leonardPKU
6556dcf347c93d37a885c34e
Yiming Zong
6556dcf347c93d37a885c34f
Yan Wang
6556dcf347c93d37a885c350
Tianyu Liu
admin_assigned
2023-11-17T16:34:35.651Z
62fc97c2ee999004b5a8fec2
/avatars/0a7805c014c26b36e607155c30620142.svg
Tianyu Liu
user
tliu
6556dcf347c93d37a885c351
Zhiwei Jiang
6556dcf347c93d37a885c352
Baobao Chang
6556dcf347c93d37a885c353
Yujia Qin
admin_assigned
2023-11-17T16:35:03.572Z
643f37cce9d063936912048b
/avatars/25822ea5676a79b2e1ddf08d5fc2226c.svg
Yujia Qin
user
YujiaHi
6556dcf347c93d37a885c354
Wangchunshu Zhou
admin_assigned
2023-11-17T16:35:13.290Z
628c8598ef14f971b698107f
/avatars/3a4ad87e6b5f9e836a1160d869df1447.svg
Zhou
user
Wangchunshu
6556dcf347c93d37a885c355
Yilun Zhao
admin_assigned
2023-11-17T16:35:20.914Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
6556dcf347c93d37a885c356
Arman Cohan
admin_assigned
2023-11-17T16:35:31.956Z
5f5ba21188f57f65f951f255
https://cdn-avatars.huggingface.co/v1/production/uploads/1599840760465-noauth.png
Arman Cohan
user
armanc
6556dcf347c93d37a885c357
Mark Gerstein
ML-Bench: Large Language Models Leverage Open-source Libraries for
  Machine Learning Tasks
Large language models have shown promising performance in code generation
benchmarks. However, a considerable divide exists between these benchmark
achievements and their practical applicability, primarily attributed to
real-world programming's reliance on pre-existing libraries. Instead of
evaluating LLMs to code from scratch, this work aims to propose a new
evaluation setup where LLMs use open-source libraries to finish machine
learning tasks. Therefore, we propose ML-Bench, an expansive benchmark
developed to assess the effectiveness of LLMs in leveraging existing functions
in open-source libraries. Consisting of 10044 samples spanning 130 tasks over
14 notable machine learning GitHub repositories. In this setting, given a
specific machine learning task instruction and the accompanying README in a
codebase, an LLM is tasked to generate code to accomplish the task. This
necessitates the comprehension of long and language-code interleaved documents,
as well as the understanding of complex cross-file code structures, introducing
new challenges. Notably, while GPT-4 exhibits remarkable improvement over other
LLMs, it manages to accomplish only 39.73\% of the tasks, leaving a huge space
for improvement. We address these challenges by proposing ML-Agent, designed to
effectively navigate the codebase, locate documentation, retrieve code, and
generate executable code. Empirical results demonstrate that ML-Agent, built
upon GPT-4, results in further improvements. Code, data, and models are
available at https://ml-bench.github.io/.
ML-Bench and ML-Agent evaluate and enhance the ability of large language models to use existing open-source libraries for machine learning tasks.
large language models code generation ML-Bench open-source libraries machine learning tasks README code comprehension cross-file code structures ML-Agent GPT-4 documentation retrieval executable code