TOMG-Bench: Evaluating LLMs on Text-based Open Molecule Generation
In this paper, we propose Text-based Open Molecule Generation Benchmark
(TOMG-Bench), the first benchmark to evaluate the open-domain molecule
generation capability of LLMs. TOMG-Bench encompasses a dataset of three major
tasks: molecule editing (MolEdit), molecule optimization (MolOpt), and
customized molecule generation (MolCustom). Each task further contains three
subtasks, with each subtask comprising 5,000 test samples. Given the inherent
complexity of open molecule generation, we have also developed an automated
evaluation system that helps measure both the quality and the accuracy of the
generated molecules. Our comprehensive benchmarking of 25 LLMs reveals the
current limitations and potential areas for improvement in text-guided molecule
discovery. Furthermore, with the assistance of OpenMolIns, a specialized
instruction tuning dataset proposed for solving challenges raised by
TOMG-Bench, Llama3.1-8B could outperform all the open-source general LLMs, even
surpassing GPT-3.5-turbo by 46.5\% on TOMG-Bench. Our codes and datasets are
available through https://github.com/phenixace/TOMG-Bench.
6764de0ca246952fabef4309
Jiatong Li
claimed_verified
2024-12-30T19:36:31.168Z
6438e55cb2ea24b52ebc45ec
/avatars/e13c7398f77e7e0bd5eed03102aa5c36.svg
Jiatong LI
user
phenixace
6764de0ca246952fabef430a
Junxian Li
claimed_verified
2024-12-20T08:32:48.973Z
656ae4088fb1ddf0d5ec9ac5
/avatars/e38468d2c0274f3c0f5732f30a2e3436.svg
Junxian Li
user
Duke-de-Artois
6764de0ca246952fabef430b
Yunqing Liu
6764de0ca246952fabef430c
Dongzhan Zhou
admin_assigned
2024-12-20T09:30:10.820Z
6538b861613fe158bd581e35
/avatars/6817dbfe903675721fd227058b0a91ac.svg
Dongzhan Zhou
user
schrodingers-tiger
6764de0ca246952fabef430d
Qing Li
TOMG-Bench: Evaluating LLMs on Text-based Open Molecule Generation
In this paper, we propose Text-based Open Molecule Generation Benchmark
(TOMG-Bench), the first benchmark to evaluate the open-domain molecule
generation capability of LLMs. TOMG-Bench encompasses a dataset of three major
tasks: molecule editing (MolEdit), molecule optimization (MolOpt), and
customized molecule generation (MolCustom). Each task further contains three
subtasks, with each subtask comprising 5,000 test samples. Given the inherent
complexity of open molecule generation, we have also developed an automated
evaluation system that helps measure both the quality and the accuracy of the
generated molecules. Our comprehensive benchmarking of 25 LLMs reveals the
current limitations and potential areas for improvement in text-guided molecule
discovery. Furthermore, with the assistance of OpenMolIns, a specialized
instruction tuning dataset proposed for solving challenges raised by
TOMG-Bench, Llama3.1-8B could outperform all the open-source general LLMs, even
surpassing GPT-3.5-turbo by 46.5\% on TOMG-Bench. Our codes and datasets are
available through https://github.com/phenixace/TOMG-Bench.
A benchmark evaluates text-guided molecule generation across editing, optimization, and customization tasks, revealing limitations and improvements for large language models.
Text-based Open Molecule Generation Benchmark TOMG-Bench molecule editing molecule optimization customized molecule generation automated evaluation system large language models LLMs OpenMolIns instruction tuning Llama3.1-8B GPT-3.5-turbo