Benchmarking Large Language Model Capabilities for Conditional
  Generation
Pre-trained large language models (PLMs) underlie most new developments in
natural language processing. They have shifted the field from
application-specific model pipelines to a single model that is adapted to a
wide range of tasks. Autoregressive PLMs like GPT-3 or PaLM, alongside
techniques like few-shot learning, have additionally shifted the output
modality to generation instead of classification or regression. Despite their
ubiquitous use, the generation quality of language models is rarely evaluated
when these models are introduced. Additionally, it is unclear how existing
generation tasks--while they can be used to compare systems at a high
level--relate to the real world use cases for which people have been adopting
them. In this work, we discuss how to adapt existing application-specific
generation benchmarks to PLMs and provide an in-depth, empirical study of the
limitations and capabilities of PLMs in natural language generation tasks along
dimensions such as scale, architecture, input and output language. Our results
show that PLMs differ in their applicability to different data regimes and
their generalization to multiple languages and inform which PLMs to use for a
given generation task setup. We share best practices to be taken into
consideration when benchmarking generation capabilities during the development
of upcoming PLMs.
649e3d76a8314342768b87c2
Joshua Maynez
claimed_verified
2023-06-30T20:55:58.479Z
63f5fd44b607296857b3d059
/avatars/7ea62ff0d82e24edfbc7a562484d0947.svg
Joshua Maynez
user
joshuahm
649e3d76a8314342768b87c3
Priyanka Agrawal
649e3d76a8314342768b87c4
Sebastian Gehrmann
Benchmarking Large Language Model Capabilities for Conditional
  Generation
Pre-trained large language models (PLMs) underlie most new developments in
natural language processing. They have shifted the field from
application-specific model pipelines to a single model that is adapted to a
wide range of tasks. Autoregressive PLMs like GPT-3 or PaLM, alongside
techniques like few-shot learning, have additionally shifted the output
modality to generation instead of classification or regression. Despite their
ubiquitous use, the generation quality of language models is rarely evaluated
when these models are introduced. Additionally, it is unclear how existing
generation tasks--while they can be used to compare systems at a high
level--relate to the real world use cases for which people have been adopting
them. In this work, we discuss how to adapt existing application-specific
generation benchmarks to PLMs and provide an in-depth, empirical study of the
limitations and capabilities of PLMs in natural language generation tasks along
dimensions such as scale, architecture, input and output language. Our results
show that PLMs differ in their applicability to different data regimes and
their generalization to multiple languages and inform which PLMs to use for a
given generation task setup. We share best practices to be taken into
consideration when benchmarking generation capabilities during the development
of upcoming PLMs.
Researchers adapt application-specific generation benchmarks to pre-trained large language models (PLMs) to evaluate their limitations and capabilities across different dimensions like scale, architecture, and language.
pre-trained large language models PLMs natural language processing autoregressive PLMs few-shot learning generation classification regression benchmarking natural language generation