ZIP-FIT: Embedding-Free Data Selection via Compression-Based Alignment
Data selection is crucial for optimizing language model (LM) performance on
specific tasks, yet most existing methods fail to effectively consider the
target task distribution.
  Current approaches either ignore task-specific requirements entirely or rely
on approximations that fail to capture the nuanced patterns needed for tasks
like Autoformalization or code generation.
  Methods that do consider the target distribution often rely on simplistic,
sometimes noisy, representations, like hashed n-gram features, which can lead
to collisions and introduce noise.
  We introduce ZIP-FIT, a data selection framework that uses gzip compression
to directly measure alignment between potential training data and the target
task distribution.
  In extensive evaluations on Autoformalization and Python code generation,
ZIP-FIT significantly outperforms leading baselines like DSIR and D4.
  Models trained on ZIP-FIT-selected data achieve their lowest cross-entropy
loss up to 85.1\% faster than baselines, demonstrating that better task
alignment leads to more efficient learning.
  In addition, ZIP-FIT performs selection up to 65.8\% faster than DSIR and two
orders of magnitude faster than D4.
  Notably, ZIP-FIT shows that smaller, well-aligned datasets often outperform
larger but less targeted ones, demonstrating that a small amount of higher
quality data is superior to a large amount of lower quality data.
  Our results imply that task-aware data selection is crucial for efficient
domain adaptation, and that compression offers a principled way to measure task
alignment.
  By showing that targeted data selection can dramatically improve
task-specific performance, our work provides new insights into the relationship
between data quality, task alignment, and model learning efficiency.
671c2017b91f5a4a2468bd4d
Elyas Obbad
671c2017b91f5a4a2468bd4e
Iddah Mlauzi
671c2017b91f5a4a2468bd4f
Brando Miranda
claimed_verified
2024-10-28T10:37:15.108Z
60ec6453046c81764c7e06f0
https://cdn-avatars.huggingface.co/v1/production/uploads/60ec6453046c81764c7e06f0/gnKlFE5bkpvbDQNb2jrnV.png
Brando Miranda
user
brando
671c2017b91f5a4a2468bd50
Rylan Schaeffer
671c2017b91f5a4a2468bd51
Kamal Obbad
671c2017b91f5a4a2468bd52
Suhana Bedi
671c2017b91f5a4a2468bd53
Sanmi Koyejo
ZIP-FIT: Embedding-Free Data Selection via Compression-Based Alignment
Data selection is crucial for optimizing language model (LM) performance on
specific tasks, yet most existing methods fail to effectively consider the
target task distribution.
  Current approaches either ignore task-specific requirements entirely or rely
on approximations that fail to capture the nuanced patterns needed for tasks
like Autoformalization or code generation.
  Methods that do consider the target distribution often rely on simplistic,
sometimes noisy, representations, like hashed n-gram features, which can lead
to collisions and introduce noise.
  We introduce ZIP-FIT, a data selection framework that uses gzip compression
to directly measure alignment between potential training data and the target
task distribution.
  In extensive evaluations on Autoformalization and Python code generation,
ZIP-FIT significantly outperforms leading baselines like DSIR and D4.
  Models trained on ZIP-FIT-selected data achieve their lowest cross-entropy
loss up to 85.1\% faster than baselines, demonstrating that better task
alignment leads to more efficient learning.
  In addition, ZIP-FIT performs selection up to 65.8\% faster than DSIR and two
orders of magnitude faster than D4.
  Notably, ZIP-FIT shows that smaller, well-aligned datasets often outperform
larger but less targeted ones, demonstrating that a small amount of higher
quality data is superior to a large amount of lower quality data.
  Our results imply that task-aware data selection is crucial for efficient
domain adaptation, and that compression offers a principled way to measure task
alignment.
  By showing that targeted data selection can dramatically improve
task-specific performance, our work provides new insights into the relationship
between data quality, task alignment, and model learning efficiency.
ZIP-FIT, a data selection framework using gzip compression, significantly improves task-specific performance and efficiency in language models by better aligning training data with the target task distribution.
ZIP-FIT gzip compression task-specific requirements target task distribution Autoformalization code generation DSIR D4 cross-entropy loss domain adaptation