Towards overcoming data scarcity in materials science: unifying models and datasets with a mixture of experts framework
While machine learning has emerged in recent years as a useful tool for rapid
prediction of materials properties, generating sufficient data to reliably
train models without overfitting is still impractical for many applications.
Towards overcoming this limitation, we present a general framework for
leveraging complementary information across different models and datasets for
accurate prediction of data scarce materials properties. Our approach, based on
a machine learning paradigm called mixture of experts, outperforms pairwise
transfer learning on 16 of 19 materials property regression tasks, performing
comparably on the remaining three. Unlike pairwise transfer learning, our
framework automatically learns to combine information from multiple source
tasks in a single training run, alleviating the need for brute-force
experiments to determine which source task to transfer from. The approach also
provides an interpretable, model-agnostic, and scalable mechanism to transfer
information from an arbitrary number of models and datasets to any downstream
property prediction task. We anticipate the performance of our framework will
further improve as better model architectures, new pre-training tasks, and
larger materials datasets are developed by the community.