Using Imperfect Synthetic Data in Downstream Inference Tasks
Yewon Byun, Shantanu Gupta, Zachary Lipton, Rachel Childers, Bryan Wilder
Predictions and generations from large language models are increasingly being explored as an aid to computational social science and human subject research. One common hope is to augment data in cases where labeling data is expensive, leading to imprecise estimates of statistical parameters. While previous technical work has explored the potential to use model-predicted labels for unlabeled data in a principled manner, there is increasing interest in using large language models to generate entirely new synthetic samples, such as in responses to surveys or in human behavior simulation. However, it is not immediately clear by what means practitioners can incorporate such data and yet produce statistically valid conclusions upon them. In this work, we introduce a new estimator, providing a hyperparameter-free solution with strong theoretical guarantees to address the challenge at hand. We empirically validate the finite-sample performance of our estimator across different regression tasks in computational social science applications. To the best of our knowledge, our framework provides the first theoretically-sound approach for incorporating fully synthetic samples in downstream statistical analyses.