Zero-Shot Performance Prediction for Probabilistic Scaling Laws
Viktoria Schram, Markus Hiller, Daniel Beck, Trevor Cohn
The prediction of training and learning curves of Natural Language Processing (NLP) models enables targeted decision-making to achieve performance objectives, reduces computational overhead and minimizes costs associated with dataset acquisition and curation. In this work, we formulate both prediction tasks as multitask learning problems, where each task’s data is modelled as being organized within a two-layer hierarchy. We employ latent variable multi-output Gaussian processes to capture correlations across tasks and hierarchies in order to support zero-shot prediction. We demonstrate that this approach facilitates the development of probabilistic scaling laws at lower costs. Applying an active learning strategy, training curves can be queried to reduce predictive uncertainty and provide predictions close to ground truth scaling laws. We validate the generality of our modelling approach and the strength of the proposed framework for NLP tasks using three small-scale datasets consisting of at maximum $30$ curves: Training curves from nanoGPT models of varying sizes, learning curves from mBART and a Transformer model for bilingual translation, and learning curves for multilingual translation using M2M100 architectures of three different sizes.