Training Noise Token Pruning
In the present work we present Training Noise Token (TNT) Pruning for vision
transformers. Our method relaxes the discrete token dropping condition to
continuous additive noise, providing smooth optimization in training, while
retaining discrete dropping computational gains in deployment settings. We
provide theoretical connections to Rate-Distortion literature, and empirical
evaluations on the ImageNet dataset using ViT and DeiT architectures
demonstrating TNT's advantages over previous pruning methods.
674e25943cf5e90cbc7f0e9e
Mingxing Rao
claimed_verified
2024-12-03T11:16:04.218Z
6648ce9df16e33cc991f2b94
/avatars/a3e4c39146b34ec82ab0eb31444b678c.svg
Mingxing Rao
user
ethanrao1
674e25943cf5e90cbc7f0e9f
Bohan Jiang
674e25943cf5e90cbc7f0ea0
Daniel Moyer
Training Noise Token Pruning
In the present work we present Training Noise Token (TNT) Pruning for vision
transformers. Our method relaxes the discrete token dropping condition to
continuous additive noise, providing smooth optimization in training, while
retaining discrete dropping computational gains in deployment settings. We
provide theoretical connections to Rate-Distortion literature, and empirical
evaluations on the ImageNet dataset using ViT and DeiT architectures
demonstrating TNT's advantages over previous pruning methods.
TNT Pruning introduces continuous additive noise to vision transformers, enabling smooth training while maintaining computational benefits of token dropping during deployment.
Training Noise Token (TNT) token dropping continuous additive noise smooth optimization deployment Rate-Distortion ImageNet ViT DeiT