Farzi Data: Autoregressive Data Distillation
We study data distillation for auto-regressive machine learning tasks, where
the input and output have a strict left-to-right causal structure. More
specifically, we propose Farzi, which summarizes an event sequence dataset into
a small number of synthetic sequences -- Farzi Data -- which are optimized to
maintain (if not improve) model performance compared to training on the full
dataset. Under the hood, Farzi conducts memory-efficient data distillation by
(i) deriving efficient reverse-mode differentiation of the Adam optimizer by
leveraging Hessian-Vector Products; and (ii) factorizing the high-dimensional
discrete event-space into a latent-space which provably promotes implicit
regularization. Empirically, for sequential recommendation and language
modeling tasks, we are able to achieve 98-120% of downstream full-data
performance when training state-of-the-art models on Farzi Data of size as
little as 0.1% of the original dataset. Notably, being able to train better
models with significantly less data sheds light on the design of future large
auto-regressive models, and opens up new opportunities to further scale up
model and data sizes.
652deefa2d617ec278bfee1d
Noveen Sachdeva
claimed_verified
2023-10-17T15:11:52.497Z
652e529acb01191873067b02
/avatars/a340d5faca6ad1d120e8a904380276e9.svg
Sachdeva
user
Noveen
652deefa2d617ec278bfee1e
Zexue He
admin_assigned
2023-10-17T08:05:14.295Z
64d3f0462ad1d0608d95550f
/avatars/75055e03ceade47a50f5111e27079a3a.svg
Zexue He
user
JJesssie
652deefa2d617ec278bfee1f
Wang-Cheng Kang
652deefa2d617ec278bfee20
Jianmo Ni
652deefa2d617ec278bfee21
Derek Zhiyuan Cheng
admin_assigned
2023-10-17T08:06:11.115Z
624cedaa43ce7877c5af2c4e
/avatars/86f13f8e082e5c0b8ea671c374e4d675.svg
Zhiyuan Cheng
user
willcee
652deefa2d617ec278bfee22
Julian McAuley
Farzi Data: Autoregressive Data Distillation
We study data distillation for auto-regressive machine learning tasks, where
the input and output have a strict left-to-right causal structure. More
specifically, we propose Farzi, which summarizes an event sequence dataset into
a small number of synthetic sequences -- Farzi Data -- which are optimized to
maintain (if not improve) model performance compared to training on the full
dataset. Under the hood, Farzi conducts memory-efficient data distillation by
(i) deriving efficient reverse-mode differentiation of the Adam optimizer by
leveraging Hessian-Vector Products; and (ii) factorizing the high-dimensional
discrete event-space into a latent-space which provably promotes implicit
regularization. Empirically, for sequential recommendation and language
modeling tasks, we are able to achieve 98-120% of downstream full-data
performance when training state-of-the-art models on Farzi Data of size as
little as 0.1% of the original dataset. Notably, being able to train better
models with significantly less data sheds light on the design of future large
auto-regressive models, and opens up new opportunities to further scale up
model and data sizes.
Farzi distills large event sequence datasets into smaller synthetic datasets, significantly reducing data size while maintaining high model performance for auto-regressive tasks.
Farzi data distillation auto-regressive tasks event sequence synthetic sequences Adam optimizer reverse-mode differentiation Hessian-Vector Products latent-space implicit regularization sequential recommendation language modeling