Leveraging semantic similarity for experimentation with AI-generated treatments
Lei Shi, David Arbour, Raghavendra Addanki, Ritwik Sinha, Avi Feller
The growing popularity of Large Language Models (LLMs) introduces exciting opportunities for digital experimentation, as marketers increasingly combine human-generated and model-generated content to design treatments. However, the high volume and dimensionality of these treatments pose significant challenges, particularly for integration into traditional evaluation paradigms. In this paper, we propose to explicitly incorporate semantic similarity from treatment embeddings to improve statistical efficiency in the design and analysis of LLM-generated treatments at scale. Our proposed method, called double kernel representation learning, factorizes the causal effect through the inner product of the low-dimensional kernel-based representations of the treatments and user covariates. We design an alternating-minimization algorithm to efficiently implement this to learn the representation from data. Additionally, we propose an adaptive design strategy for online experimentation. We prove convergence under a low-rank factor structure and establish a sublinear regret bound for the adaptive algorithm. Finally, we demonstrate the practical performance of our approaches through experiments on semi-synthetic data.