Distance-based Analysis of Machine Learning Prediction Reliability for Datasets in Materials Science and Other Fields
Despite successful use in a wide variety of disciplines for data analysis and
prediction, machine learning (ML) methods suffer from a lack of understanding
of the reliability of predictions due to the lack of transparency and black-box
nature of ML models. In materials science and other fields, typical ML model
results include a significant number of low-quality predictions. This problem
is known to be particularly acute for target systems which differ significantly
from the data used for ML model training. However, to date, a general method
for characterization of the difference between the predicted and training
system has not been available. Here, we show that a simple metric based on
Euclidean feature space distance and sampling density allows effective
separation of the accurately predicted data points from data points with poor
prediction accuracy. We show that the metric effectiveness is enhanced by the
decorrelation of the features using Gram-Schmidt orthogonalization. To
demonstrate the generality of the method, we apply it to support vector
regression models for various small data sets in materials science and other
fields. Our method is computationally simple, can be used with any ML learning
method and enables analysis of the sources of the ML prediction errors.
Therefore, it is suitable for use as a standard technique for the estimation of
ML prediction reliability for small data sets and as a tool for data set
design.