Convolutional Set Transformer
We introduce the Convolutional Set Transformer (CST), a novel neural
architecture designed to process image sets of arbitrary cardinality that are
visually heterogeneous yet share high-level semantics - such as a common
category, scene, or concept. Existing set-input networks, e.g., Deep Sets and
Set Transformer, are limited to vector inputs and cannot directly handle 3D
image tensors. As a result, they must be cascaded with a feature extractor,
typically a CNN, which encodes images into embeddings before the set-input
network can model inter-image relationships. In contrast, CST operates directly
on 3D image tensors, performing feature extraction and contextual modeling
simultaneously, thereby enabling synergies between the two processes. This
design yields superior performance in tasks such as Set Classification and Set
Anomaly Detection and further provides native compatibility with CNN
explainability methods such as Grad-CAM, unlike competing approaches that
remain opaque. Finally, we show that CSTs can be pre-trained on large-scale
datasets and subsequently adapted to new domains and tasks through standard
Transfer Learning schemes. To support further research, we release CST-15, a
CST backbone pre-trained on ImageNet
(https://github.com/chinefed/convolutional-set-transformer).
68dc101c4159d1f2418f98b6
Federico Chinello
claimed_verified
2025-10-01T10:23:11.191Z
68dc0fdce927b1b11ce08bea
/avatars/5bba8ded95a8bb10f92426a20fccff92.svg
Federico Chinello
user
chinefed
68dc101c4159d1f2418f98b7
Giacomo Boracchi
Convolutional Set Transformer
We introduce the Convolutional Set Transformer (CST), a novel neural
architecture designed to process image sets of arbitrary cardinality that are
visually heterogeneous yet share high-level semantics - such as a common
category, scene, or concept. Existing set-input networks, e.g., Deep Sets and
Set Transformer, are limited to vector inputs and cannot directly handle 3D
image tensors. As a result, they must be cascaded with a feature extractor,
typically a CNN, which encodes images into embeddings before the set-input
network can model inter-image relationships. In contrast, CST operates directly
on 3D image tensors, performing feature extraction and contextual modeling
simultaneously, thereby enabling synergies between the two processes. This
design yields superior performance in tasks such as Set Classification and Set
Anomaly Detection and further provides native compatibility with CNN
explainability methods such as Grad-CAM, unlike competing approaches that
remain opaque. Finally, we show that CSTs can be pre-trained on large-scale
datasets and subsequently adapted to new domains and tasks through standard
Transfer Learning schemes. To support further research, we release CST-15, a
CST backbone pre-trained on ImageNet
(https://github.com/chinefed/convolutional-set-transformer).
The Convolutional Set Transformer (CST) processes image sets directly, combining feature extraction and contextual modeling for improved performance in set classification and anomaly detection, with compatibility for CNN explainability methods.
Convolutional Set Transformer CST set-input networks Deep Sets Set Transformer 3D image tensors feature extraction contextual modeling Set Classification Set Anomaly Detection Grad-CAM Transfer Learning CST-15 ImageNet