Copy Is All You Need
The dominant text generation models compose the output by sequentially
selecting words from a fixed vocabulary. In this paper, we formulate text
generation as progressively copying text segments (e.g., words or phrases) from
an existing text collection. We compute the contextualized representations of
meaningful text segments and index them using efficient vector search toolkits.
The task of text generation is then decomposed into a series of copy-and-paste
operations: at each time step, we seek suitable text spans from the text
collection rather than selecting from a standalone vocabulary. Experiments on
the standard language modeling benchmark (WikiText-103) show that our approach
achieves better generation quality according to both automatic and human
evaluations. Besides, its inference efficiency is comparable to token-level
autoregressive models thanks to the reduction of decoding steps. We also show
that our approach allows for effective domain adaptation by simply switching to
domain-specific text collection without extra training. Finally, we observe
that our approach attains additional performance gains by simply scaling up to
larger text collections, again without further training.Our source
codes are publicly available at
\url{https://github.com/gmftbyGMFTBY/Copyisallyouneed.}
64b48f5993679cd0095d8b5a
Tian Lan
claimed_verified
2023-07-25T08:08:18.837Z
627ca439a09edfe62239c671
https://cdn-avatars.huggingface.co/v1/production/uploads/1652335660508-noauth.jpeg
Tian Lan
user
GMFTBY
64b48f5993679cd0095d8b5b
Deng Cai
admin_assigned
2023-07-17T10:35:39.765Z
646735a1946476c5d214e5b2
/avatars/23392fda23fa6288fcb309f7eaa73472.svg
Deng Cai
user
jcyk
64b48f5993679cd0095d8b5c
Yan Wang
claimed_verified
2023-07-24T07:44:19.066Z
64674375946476c5d215382d
/avatars/0f0b98080c064a17d003da8a22a047cc.svg
Yan Wang
user
libertywing
64b48f5993679cd0095d8b5d
Heyan Huang
64b48f5993679cd0095d8b5e
Xian-Ling Mao
Copy Is All You Need
The dominant text generation models compose the output by sequentially
selecting words from a fixed vocabulary. In this paper, we formulate text
generation as progressively copying text segments (e.g., words or phrases) from
an existing text collection. We compute the contextualized representations of
meaningful text segments and index them using efficient vector search toolkits.
The task of text generation is then decomposed into a series of copy-and-paste
operations: at each time step, we seek suitable text spans from the text
collection rather than selecting from a standalone vocabulary. Experiments on
the standard language modeling benchmark (WikiText-103) show that our approach
achieves better generation quality according to both automatic and human
evaluations. Besides, its inference efficiency is comparable to token-level
autoregressive models thanks to the reduction of decoding steps. We also show
that our approach allows for effective domain adaptation by simply switching to
domain-specific text collection without extra training. Finally, we observe
that our approach attains additional performance gains by simply scaling up to
larger text collections, again without further training.Our source
codes are publicly available at
\url{https://github.com/gmftbyGMFTBY/Copyisallyouneed.}
Text generation is improved by copying text segments from a pre-existing collection, resulting in better quality and efficiency compared to traditional methods.
contextualized representations vector search toolkits copy-and-paste operations standard language modeling benchmark WikiText-103 autoregressive models domain adaptation