YEAST: Yet Another Sequential Test
Alexey Kurennoy, Majed Dodin, Tural Gurbanov, Ana Peleteiro Ramallo
Online evaluation of machine learning models is typically conducted through randomised experiments. Sequential statistical tests are valuable tools for analysing these experiments, as they enable researchers to stop data collection early without increasing the risk of false discoveries. However, existing sequential tests either limit the number of interim analyses or suffer from low statistical power. In this paper, we introduce a novel sequential test designed for continuous monitoring of online experiments. We validate our method using semi-synthetic simulations and demonstrate that it outperforms current state-of-the-art sequential testing approaches. Our method is derived using a new technique that ``inverts'' a bound on the probability of threshold crossing, based on a classical maximal inequality.