Performance Prediction for Large Systems via Text-to-Text Regression
In many industries, predicting metric outcomes of large systems is a
fundamental problem, driven largely by traditional tabular regression. However,
such methods struggle on complex systems data in the wild such as configuration
files or system logs, where feature engineering is often infeasible. We propose
text-to-text regression as a general, scalable alternative. For predicting
resource efficiency on Borg, Google's massive compute cluster scheduling
system, a 60M parameter encoder-decoder, trained from random initialization,
achieves up to a near perfect 0.99 (0.9 average) rank correlation across the
entire fleet, and 100x lower MSE than tabular approaches. The model also easily
adapts to new tasks in only 500 few-shot examples and captures the densities of
complex outcome distributions. Ablation studies highlight the importance of
using encoders, increasing sequence length, and the model's inherent
uncertainty quantification. These findings pave the way for universal
simulators of real-world outcomes.
6862a52d3b7fc785c7ca5dfb
Yash Akhauri
6862a52d3b7fc785c7ca5dfc
Bryan Lewandowski
6862a52d3b7fc785c7ca5dfd
Cheng-Hsi Lin
6862a52d3b7fc785c7ca5dfe
Adrian N. Reyes
6862a52d3b7fc785c7ca5dff
Grant C. Forbes
6862a52d3b7fc785c7ca5e00
Arissa Wongpanich
6862a52d3b7fc785c7ca5e01
Bangding Yang
6862a52d3b7fc785c7ca5e02
Mohamed S. Abdelfattah
6862a52d3b7fc785c7ca5e03
Sagi Perel
6862a52d3b7fc785c7ca5e04
Xingyou Song
Performance Prediction for Large Systems via Text-to-Text Regression
In many industries, predicting metric outcomes of large systems is a
fundamental problem, driven largely by traditional tabular regression. However,
such methods struggle on complex systems data in the wild such as configuration
files or system logs, where feature engineering is often infeasible. We propose
text-to-text regression as a general, scalable alternative. For predicting
resource efficiency on Borg, Google's massive compute cluster scheduling
system, a 60M parameter encoder-decoder, trained from random initialization,
achieves up to a near perfect 0.99 (0.9 average) rank correlation across the
entire fleet, and 100x lower MSE than tabular approaches. The model also easily
adapts to new tasks in only 500 few-shot examples and captures the densities of
complex outcome distributions. Ablation studies highlight the importance of
using encoders, increasing sequence length, and the model's inherent
uncertainty quantification. These findings pave the way for universal
simulators of real-world outcomes.
A text-to-text regression model achieves high accuracy in predicting resource efficiency for Google's Borg system, surpassing tabular methods, and demonstrates adaptability and uncertainty quantification.
text-to-text regression encoder-decoder feature engineering few-shot learning rank correlation mean squared error uncertainty quantification universal simulators