All that structure matches does not glitter
Maya Martirossyan, Thomas Egg, Philipp Höllmer, George Karypis, Mark Transtrum, Adrian Roitberg, Mingjie Liu, Richard Hennig, Ellad Tadmor, Stefano Martiniani
Generative models for materials, especially inorganic crystals, have attracted significant interest for their potential to transform the theoretical prediction of novel compounds and structures. Progress in this area depends critically on robust benchmarks and minimal, information-rich datasets that enable efficient and meaningful model evaluation. This paper critically examines commonly used datasets, methodologies, and evaluation metrics for the crystal structure prediction task—predicting the most likely structures given a chemical composition—and offers concrete solutions. We focus on three key issues: First, materials datasets should contain diverse and unique crystal structures; for example, we show that the widely-utilized carbon-24 dataset only contains $\approx 40$% unique structures, with duplicates differing only by the choice of unit cell representation. Second, materials datasets need to be split with care, rather than randomly, if polymorphs of many different compositions are numerous—which we discover to be the case for the perov-5 dataset. Third, benchmarks for evaluation of generative models can be misleading if used uncritically—for example, the reporting of a 'match rate' metric without consideration of the structural complexity that can be exhibited by identical building blocks (atoms). To address these oft-overlooked issues, we introduce several fixes. We provide revised versions of the carbon-24 dataset: one with duplicates removed, one deduplicated and split by number of atoms $N$, and two containing only duplicates. We also propose a new split for the perov-5 dataset that ensures polymorphs are grouped within the same training, validation, or test set in order to set a more sensible standard for benchmarking model performance. Finally, we present METRe and cRMSE, new model evaluation metrics that can correctly handle materials datasets with polymorphs.