Does Visual Pretraining Help End-to-End Reasoning?
We aim to investigate whether end-to-end learning of visual reasoning can be
achieved with general-purpose neural networks, with the help of visual
pretraining. A positive result would refute the common belief that explicit
visual abstraction (e.g. object detection) is essential for compositional
generalization on visual reasoning, and confirm the feasibility of a neural
network "generalist" to solve visual recognition and reasoning tasks. We
propose a simple and general self-supervised framework which "compresses" each
video frame into a small set of tokens with a transformer network, and
reconstructs the remaining frames based on the compressed temporal context. To
minimize the reconstruction loss, the network must learn a compact
representation for each image, as well as capture temporal dynamics and object
permanence from temporal context. We perform evaluation on two visual reasoning
benchmarks, CATER and ACRE. We observe that pretraining is essential to achieve
compositional generalization for end-to-end visual reasoning. Our proposed
framework outperforms traditional supervised pretraining, including image
classification and explicit object detection, by large margins.
64b60d23d2a61d0b6adf4eaa
Chen Sun
claimed_verified
2023-07-19T06:27:56.248Z
64b6bac29bdb2adec64b1f86
https://cdn-avatars.huggingface.co/v1/production/uploads/64b6bac29bdb2adec64b1f86/516JDGISCKjtYvaYvprp4.jpeg
Chen Sun
user
jesu9
64b60d23d2a61d0b6adf4eab
Calvin Luo
claimed_verified
2025-06-11T08:41:29.899Z
676b5ea796427da73e73c301
/avatars/347251b1005830c4fe2dd6a28664f40d.svg
Calvin Luo
user
calvinyluo
64b60d23d2a61d0b6adf4eac
Xingyi Zhou
admin_assigned
2023-07-18T08:19:53.052Z
629ee11846b4826be2d94676
/avatars/95ab04122ad626d74bf8a65d7b2bf48a.svg
Xingyi Zhou
user
zhouxy
64b60d23d2a61d0b6adf4ead
Anurag Arnab
admin_assigned
2023-07-18T08:20:14.585Z
63d7f0466ac3104e50c5311e
/avatars/2421bfb7747664d1acefbac1c00c3475.svg
Anurag Arnab
user
anurag-arnab
64b60d23d2a61d0b6adf4eae
Cordelia Schmid
Does Visual Pretraining Help End-to-End Reasoning?
We aim to investigate whether end-to-end learning of visual reasoning can be
achieved with general-purpose neural networks, with the help of visual
pretraining. A positive result would refute the common belief that explicit
visual abstraction (e.g. object detection) is essential for compositional
generalization on visual reasoning, and confirm the feasibility of a neural
network "generalist" to solve visual recognition and reasoning tasks. We
propose a simple and general self-supervised framework which "compresses" each
video frame into a small set of tokens with a transformer network, and
reconstructs the remaining frames based on the compressed temporal context. To
minimize the reconstruction loss, the network must learn a compact
representation for each image, as well as capture temporal dynamics and object
permanence from temporal context. We perform evaluation on two visual reasoning
benchmarks, CATER and ACRE. We observe that pretraining is essential to achieve
compositional generalization for end-to-end visual reasoning. Our proposed
framework outperforms traditional supervised pretraining, including image
classification and explicit object detection, by large margins.
End-to-end visual reasoning is achieved using a self-supervised transformer-based framework that compresses video frames into tokens and reconstructs them, outperforming supervised pretraining methods.
end-to-end learning visual reasoning visual pretraining compositional generalization self-supervised framework transformer network video frames temporal context reconstruction loss compact representation temporal dynamics object permanence CATER ACRE supervised pretraining image classification explicit object detection