Specialized Language Models with Cheap Inference from Limited Domain
  Data
Large language models have emerged as a versatile tool but are challenging to
apply to tasks lacking large inference budgets and large in-domain training
sets. This work formalizes these constraints and distinguishes four important
variables: the pretraining budget (for training before the target domain is
known), the specialization budget (for training after the target domain is
known), the inference budget, and the in-domain training set size. Across these
settings, we compare different approaches from the machine learning literature.
Limited by inference cost, we find better alternatives to the standard practice
of training very large vanilla transformer models. In particular, we show that
hyper-networks and mixture of experts have better perplexity for large
pretraining budgets, while small models trained on importance sampled datasets
are attractive for large specialization budgets.
65c04b5fef796a0c7f6b3337
David Grangier
65c04b5fef796a0c7f6b3338
Angelos Katharopoulos
65c04b5fef796a0c7f6b3339
Pierre Ablin
65c04b5fef796a0c7f6b333a
Awni Hannun
admin_assigned
2024-02-05T11:56:06.024Z
623c830997ddced06d78699b
https://cdn-avatars.huggingface.co/v1/production/uploads/623c830997ddced06d78699b/ucB8joTPONCnc_Gj0mssR.jpeg
Awni Hannun
user
awni
Specialized Language Models with Cheap Inference from Limited Domain
  Data
Large language models have emerged as a versatile tool but are challenging to
apply to tasks lacking large inference budgets and large in-domain training
sets. This work formalizes these constraints and distinguishes four important
variables: the pretraining budget (for training before the target domain is
known), the specialization budget (for training after the target domain is
known), the inference budget, and the in-domain training set size. Across these
settings, we compare different approaches from the machine learning literature.
Limited by inference cost, we find better alternatives to the standard practice
of training very large vanilla transformer models. In particular, we show that
hyper-networks and mixture of experts have better perplexity for large
pretraining budgets, while small models trained on importance sampled datasets
are attractive for large specialization budgets.
The study evaluates various machine learning approaches under constrained inference and training budgets, demonstrating that hyper-networks and mixture of experts perform well with large pretraining budgets, while small models trained on importance sampled datasets are better with large specialization budgets.
large language models pretraining budget specialization budget inference budget in-domain training set vanilla transformer models hyper-networks mixture of experts importance sampled datasets