Farzi Data: Autoregressive Data Distillation
We study data distillation for auto-regressive machine learning tasks, where
the input and output have a strict left-to-right causal structure. More
specifically, we propose Farzi, which summarizes an event sequence dataset into
a small number of synthetic sequences -- Farzi Data -- which are optimized to
maintain (if not improve) model performance compared to training on the full
dataset. Under the hood, Farzi conducts memory-efficient data distillation by
(i) deriving efficient reverse-mode differentiation of the Adam optimizer by
leveraging Hessian-Vector Products; and (ii) factorizing the high-dimensional
discrete event-space into a latent-space which provably promotes implicit
regularization. Empirically, for sequential recommendation and language
modeling tasks, we are able to achieve 98-120% of downstream full-data
performance when training state-of-the-art models on Farzi Data of size as
little as 0.1% of the original dataset. Notably, being able to train better
models with significantly less data sheds light on the design of future large
auto-regressive models, and opens up new opportunities to further scale up
model and data sizes.
652deefa2d617ec278bfee1d
Noveen Sachdeva
claimed_verified
2023-10-17T15:11:52.497Z
652e529acb01191873067b02
/avatars/a340d5faca6ad1d120e8a904380276e9.svg
Sachdeva
user
Noveen
652deefa2d617ec278bfee1e
Zexue He
admin_assigned
2023-10-17T08:05:14.295Z
64d3f0462ad1d0608d95550f
/avatars/75055e03ceade47a50f5111e27079a3a.svg
Zexue He
user
JJesssie
652deefa2d617ec278bfee1f
Wang-Cheng Kang
652deefa2d617ec278bfee20
Jianmo Ni
652deefa2d617ec278bfee21
Derek Zhiyuan Cheng
admin_assigned
2023-10-17T08:06:11.115Z
624cedaa43ce7877c5af2c4e
/avatars/86f13f8e082e5c0b8ea671c374e4d675.svg
Zhiyuan Cheng
user
willcee
652deefa2d617ec278bfee22
Julian McAuley
Farzi Data: Autoregressive Data Distillation
We study data distillation for auto-regressive machine learning tasks, where
the input and output have a strict left-to-right causal structure. More
specifically, we propose Farzi, which summarizes an event sequence dataset into
a small number of synthetic sequences -- Farzi Data -- which are optimized to
maintain (if not improve) model performance compared to training on the full
dataset. Under the hood, Farzi conducts memory-efficient data distillation by
(i) deriving efficient reverse-mode differentiation of the Adam optimizer by
leveraging Hessian-Vector Products; and (ii) factorizing the high-dimensional
discrete event-space into a latent-space which provably promotes implicit
regularization. Empirically, for sequential recommendation and language
modeling tasks, we are able to achieve 98-120% of downstream full-data
performance when training state-of-the-art models on Farzi Data of size as
little as 0.1% of the original dataset. Notably, being able to train better
models with significantly less data sheds light on the design of future large
auto-regressive models, and opens up new opportunities to further scale up
model and data sizes.
data distillation auto-regressive Farzi synthetic sequences Farzi Data reverse-mode differentiation Adam optimizer Hessian-Vector Products factorizing latent-space implicit regularization sequential recommendation language modeling