SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced
  Token Detection
Pre-training large language models is known to be extremely resource
intensive and often times inefficient, under-utilizing the information
encapsulated in the training text sequences. In this paper, we present SpacTor,
a new training procedure consisting of (1) a hybrid objective combining span
corruption (SC) and token replacement detection (RTD), and (2) a two-stage
curriculum that optimizes the hybrid objective over the initial tau
iterations, then transitions to standard SC loss. We show empirically that the
effectiveness of the hybrid objective is tied to the two-stage pre-training
schedule, and provide extensive analysis on why this is the case. In our
experiments with encoder-decoder architectures (T5) on a variety of NLP tasks,
SpacTor-T5 yields the same downstream performance as standard SC pre-training,
while enabling a 50% reduction in pre-training iterations and 40% reduction in
total FLOPs. Alternatively, given the same amount of computing budget, we find
that SpacTor results in significantly improved downstream benchmark
performance.
65b1e70254fa188a2fbf4852
Ke Ye
65b1e70254fa188a2fbf4853
Heinrich Jiang
admin_assigned
2024-01-25T09:23:19.387Z
64628caf77f63463fc5f8d6d
/avatars/1f2db6194e24602a1e254e31ba609ee3.svg
Heinrich Jiang
user
heinrichjiang
65b1e70254fa188a2fbf4854
Afshin Rostamizadeh
65b1e70254fa188a2fbf4855
Ayan Chakrabarti
65b1e70254fa188a2fbf4856
Giulia DeSalvo
65b1e70254fa188a2fbf4857
Jean-François Kagy
extracted_pending
2024-01-25T04:43:46.730Z
646282962538819c729edca1
/avatars/89888fef14a957b89333c6f1cc6fab23.svg
JF Kagy
user
jfkagy
65b1e70254fa188a2fbf4858
Lazaros Karydas
65b1e70254fa188a2fbf4859
Gui Citovsky
65b1e70254fa188a2fbf485a
Sanjiv Kumar
SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced
  Token Detection
Pre-training large language models is known to be extremely resource
intensive and often times inefficient, under-utilizing the information
encapsulated in the training text sequences. In this paper, we present SpacTor,
a new training procedure consisting of (1) a hybrid objective combining span
corruption (SC) and token replacement detection (RTD), and (2) a two-stage
curriculum that optimizes the hybrid objective over the initial tau
iterations, then transitions to standard SC loss. We show empirically that the
effectiveness of the hybrid objective is tied to the two-stage pre-training
schedule, and provide extensive analysis on why this is the case. In our
experiments with encoder-decoder architectures (T5) on a variety of NLP tasks,
SpacTor-T5 yields the same downstream performance as standard SC pre-training,
while enabling a 50% reduction in pre-training iterations and 40% reduction in
total FLOPs. Alternatively, given the same amount of computing budget, we find
that SpacTor results in significantly improved downstream benchmark
performance.
SpacTor, a hybrid training procedure combining span corruption and token replacement detection with a two-stage curriculum, enhances pre-training efficiency and performance for large language models.
span corruption token replacement detection two-stage curriculum encoder-decoder architectures T5 downstream performance pre-training FLOPs