Towards Data-Efficient Pretraining for Atomic Property Prediction
This paper challenges the recent paradigm in atomic property prediction that
links progress to growing dataset sizes and computational resources. We show
that pretraining on a carefully selected, task-relevant dataset can match or
even surpass large-scale pretraining, while using as little as 1/24th of the
computational cost. We introduce the Chemical Similarity Index (CSI), a novel
metric inspired by computer vision's Fr\'echet Inception Distance, for
molecular graphs which quantifies the alignment between upstream pretraining
datasets and downstream tasks. By selecting the most relevant dataset with
minimal CSI distance, we show that models pretrained on a smaller, focused
dataset consistently outperform those pretrained on massive, mixed datasets
such as JMP, even when those larger datasets include the relevant dataset.
Counterintuitively, we also find that indiscriminately adding more data can
degrade model performance when the additional data poorly aligns with the task
at hand. Our findings highlight that quality often outperforms quantity in
pretraining for atomic property prediction.
67b44f44620ae0bad17d6699
Yasir Ghunaim
67b44f44620ae0bad17d669a
Hasan Abed Al Kader Hammoud
claimed_verified
2025-02-18T09:30:43.057Z
642b51385bf2355d02a23d15
/avatars/87985347643b2647555f2453fa4d94fb.svg
Hasan Abed Al Kader Hammoud
user
hammh0a
67b44f44620ae0bad17d669b
Bernard Ghanem
Towards Data-Efficient Pretraining for Atomic Property Prediction
This paper challenges the recent paradigm in atomic property prediction that
links progress to growing dataset sizes and computational resources. We show
that pretraining on a carefully selected, task-relevant dataset can match or
even surpass large-scale pretraining, while using as little as 1/24th of the
computational cost. We introduce the Chemical Similarity Index (CSI), a novel
metric inspired by computer vision's Fr\'echet Inception Distance, for
molecular graphs which quantifies the alignment between upstream pretraining
datasets and downstream tasks. By selecting the most relevant dataset with
minimal CSI distance, we show that models pretrained on a smaller, focused
dataset consistently outperform those pretrained on massive, mixed datasets
such as JMP, even when those larger datasets include the relevant dataset.
Counterintuitively, we also find that indiscriminately adding more data can
degrade model performance when the additional data poorly aligns with the task
at hand. Our findings highlight that quality often outperforms quantity in
pretraining for atomic property prediction.