Battle of the Backbones: A Large-Scale Comparison of Pretrained Models
  across Computer Vision Tasks
Neural network based computer vision systems are typically built on a
backbone, a pretrained or randomly initialized feature extractor. Several years
ago, the default option was an ImageNet-trained convolutional neural network.
However, the recent past has seen the emergence of countless backbones
pretrained using various algorithms and datasets. While this abundance of
choice has led to performance increases for a range of systems, it is difficult
for practitioners to make informed decisions about which backbone to choose.
Battle of the Backbones (BoB) makes this choice easier by benchmarking a
diverse suite of pretrained models, including vision-language models, those
trained via self-supervised learning, and the Stable Diffusion backbone, across
a diverse set of computer vision tasks ranging from classification to object
detection to OOD generalization and more. Furthermore, BoB sheds light on
promising directions for the research community to advance computer vision by
illuminating strengths and weakness of existing approaches through a
comprehensive analysis conducted on more than 1500 training runs. While vision
transformers (ViTs) and self-supervised learning (SSL) are increasingly
popular, we find that convolutional neural networks pretrained in a supervised
fashion on large training sets still perform best on most tasks among the
models we consider. Moreover, in apples-to-apples comparisons on the same
architectures and similarly sized pretraining datasets, we find that SSL
backbones are highly competitive, indicating that future works should perform
SSL pretraining with advanced architectures and larger pretraining datasets. We
release the raw results of our experiments along with code that allows
researchers to put their own backbones through the gauntlet here:
https://github.com/hsouri/Battle-of-the-Backbones
6541b949b887e805a441ebc2
Micah Goldblum
6541b949b887e805a441ebc3
Hossein Souri
extracted_confirmed
2024-03-28T16:16:09.450Z
6317bf031d6018cb85185870
https://cdn-avatars.huggingface.co/v1/production/uploads/6317bf031d6018cb85185870/cZpMcYaMVZkUEuuG1DXkI.png
Hossein Souri
user
hsouri
6541b949b887e805a441ebc4
Renkun Ni
admin_assigned
2023-11-02T15:32:13.771Z
637a79ea3f8f451d6bd50580
/avatars/80618daa0fcb3ac9e4387f385e40824f.svg
Renkun Ni
user
rn9zm
6541b949b887e805a441ebc5
Manli Shu
admin_assigned
2023-11-02T15:32:21.639Z
63bdcb129a15a3e94198e7d3
https://cdn-avatars.huggingface.co/v1/production/uploads/63bdcb129a15a3e94198e7d3/wFVtkekqyQLbS0-78fqcs.jpeg
Manli Shu
user
Manli
6541b949b887e805a441ebc6
Viraj Prabhu
admin_assigned
2023-11-02T15:32:32.150Z
63fe4f92c0ec83fda436b6bf
/avatars/1021060e042ce3257444577724aa9d34.svg
Viraj Prabhu
user
virajp
6541b949b887e805a441ebc7
Gowthami Somepalli
admin_assigned
2023-11-02T15:32:39.789Z
62ae853a4b5d6165083a12f8
https://cdn-avatars.huggingface.co/v1/production/uploads/62ae853a4b5d6165083a12f8/3_Q1yB3Qfpd9PngzzBlgZ.jpeg
Gowthami Somepalli
user
somepago
6541b949b887e805a441ebc8
Prithvijit Chattopadhyay
claimed_verified
2023-11-02T11:34:59.837Z
628d451386d23ad1560882c4
https://cdn-avatars.huggingface.co/v1/production/uploads/628d451386d23ad1560882c4/UMxez0DEvX5qdP5ddqi-8.png
Prithvijit Chattopadhyay
user
prithv1
6541b949b887e805a441ebc9
Mark Ibrahim
claimed_verified
2024-11-18T15:20:25.645Z
64c17345e82e55936cf971bc
/avatars/b39c5637e00291410620afb7c770d587.svg
Mark Ibrahim
user
marksibrahim
6541b949b887e805a441ebca
Adrien Bardes
6541b949b887e805a441ebcb
Judy Hoffman
admin_assigned
2023-11-02T15:32:54.177Z
6477790c4ae93470ffc297bb
/avatars/f8d5e08719297a335731a117416d6336.svg
Judy Hoffman
user
judyhoffman
6541b949b887e805a441ebcc
Rama Chellappa
6541b949b887e805a441ebcd
Andrew Gordon Wilson
6541b949b887e805a441ebce
Tom Goldstein
admin_assigned
2023-11-02T15:33:09.360Z
6381ca7d65dc156aba0b933d
/avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
Tom Goldstein
user
tomgoldstein
Battle of the Backbones: A Large-Scale Comparison of Pretrained Models
  across Computer Vision Tasks
Neural network based computer vision systems are typically built on a
backbone, a pretrained or randomly initialized feature extractor. Several years
ago, the default option was an ImageNet-trained convolutional neural network.
However, the recent past has seen the emergence of countless backbones
pretrained using various algorithms and datasets. While this abundance of
choice has led to performance increases for a range of systems, it is difficult
for practitioners to make informed decisions about which backbone to choose.
Battle of the Backbones (BoB) makes this choice easier by benchmarking a
diverse suite of pretrained models, including vision-language models, those
trained via self-supervised learning, and the Stable Diffusion backbone, across
a diverse set of computer vision tasks ranging from classification to object
detection to OOD generalization and more. Furthermore, BoB sheds light on
promising directions for the research community to advance computer vision by
illuminating strengths and weakness of existing approaches through a
comprehensive analysis conducted on more than 1500 training runs. While vision
transformers (ViTs) and self-supervised learning (SSL) are increasingly
popular, we find that convolutional neural networks pretrained in a supervised
fashion on large training sets still perform best on most tasks among the
models we consider. Moreover, in apples-to-apples comparisons on the same
architectures and similarly sized pretraining datasets, we find that SSL
backbones are highly competitive, indicating that future works should perform
SSL pretraining with advanced architectures and larger pretraining datasets. We
release the raw results of our experiments along with code that allows
researchers to put their own backbones through the gauntlet here:
https://github.com/hsouri/Battle-of-the-Backbones
Battle of the Backbones benchmarks diverse pretrained models across various computer vision tasks, highlighting the current strengths and weaknesses of convolutional neural networks, vision transformers, and self-supervised learning approaches.
convolutional neural networks pretrained models vision-language models self-supervised learning Stable Diffusion backbone computer vision tasks classification object detection OOD generalization vision transformers self-supervised learning SSL pretraining pretraining datasets