Calibration and generalizability of probabilistic models on low-data chemical datasets with DIONYSUS
Deep learning models that leverage large datasets are often the state of the
art for modelling molecular properties. When the datasets are smaller (< 2000
molecules), it is not clear that deep learning approaches are the right
modelling tool. In this work we perform an extensive study of the calibration
and generalizability of probabilistic machine learning models on small chemical
datasets. Using different molecular representations and models, we analyse the
quality of their predictions and uncertainties in a variety of tasks (binary,
regression) and datasets. We also introduce two simulated experiments that
evaluate their performance: (1) Bayesian optimization guided molecular design,
(2) inference on out-of-distribution data via ablated cluster splits. We offer
practical insights into model and feature choice for modelling small chemical
datasets, a common scenario in new chemical experiments. We have packaged our
analysis into the DIONYSUS repository, which is open sourced to aid in
reproducibility and extension to new datasets.