A Dataset and Analysis of Open-Source Machine Learning Products
Machine learning (ML) components are increasingly incorporated into software
products, yet developers face challenges in transitioning from ML prototypes to
products. Academic researchers struggle to propose solutions to these
challenges and evaluate interventions because they often do not have access to
close-sourced ML products from industry. In this study, we define and identify
open-source ML products, curating a dataset of 262 repositories from GitHub, to
facilitate further research and education. As a start, we explore six broad
research questions related to different development activities and report 21
findings from a sample of 30 ML products from the dataset. Our findings reveal
a variety of development practices and architectural decisions surrounding
different types and uses of ML models that offer ample opportunities for future
research innovations. We also find very little evidence of industry best
practices such as model testing and pipeline automation within the open-source
ML products, which leaves room for further investigation to understand its
potential impact on the development and eventual end-user experience for the
products.