CosmoCLIP: Generalizing Large Vision-Language Models for Astronomical
  Imaging
Existing vision-text contrastive learning models enhance representation
transferability and support zero-shot prediction by matching paired image and
caption embeddings while pushing unrelated pairs apart. However, astronomical
image-label datasets are significantly smaller compared to general image and
label datasets available from the internet. We introduce CosmoCLIP, an
astronomical image-text contrastive learning framework precisely fine-tuned on
the pre-trained CLIP model using SpaceNet and BLIP-based captions. SpaceNet,
attained via FLARE, constitutes ~13k optimally distributed images, while BLIP
acts as a rich knowledge extractor. The rich semantics derived from this
SpaceNet and BLIP descriptions, when learned contrastively, enable CosmoCLIP to
achieve superior generalization across various in-domain and out-of-domain
tasks. Our results demonstrate that CosmoCLIP is a straightforward yet powerful
framework, significantly outperforming CLIP in zero-shot classification and
image-text retrieval tasks.
668fb090c9ca04b1c10d53a9
Raza Imam
admin_assigned
2024-07-11T10:58:12.875Z
664396275778efc851b14408
/avatars/8ea99f8c6c59a7d49295aebfea026963.svg
Raza Imam
user
razaimam45
668fb090c9ca04b1c10d53aa
Mohammed Talha Alam
admin_assigned
2024-07-11T10:58:19.295Z
65a942038b3bd440586ae467
/avatars/5c109e67f66bd47a2ea85db837e49e6e.svg
Mohammed Talha Alam
user
talha-alam
668fb090c9ca04b1c10d53ab
Umaima Rahman
668fb090c9ca04b1c10d53ac
Mohsen Guizani
668fb090c9ca04b1c10d53ad
Fakhri Karray
CosmoCLIP: Generalizing Large Vision-Language Models for Astronomical
  Imaging
Existing vision-text contrastive learning models enhance representation
transferability and support zero-shot prediction by matching paired image and
caption embeddings while pushing unrelated pairs apart. However, astronomical
image-label datasets are significantly smaller compared to general image and
label datasets available from the internet. We introduce CosmoCLIP, an
astronomical image-text contrastive learning framework precisely fine-tuned on
the pre-trained CLIP model using SpaceNet and BLIP-based captions. SpaceNet,
attained via FLARE, constitutes ~13k optimally distributed images, while BLIP
acts as a rich knowledge extractor. The rich semantics derived from this
SpaceNet and BLIP descriptions, when learned contrastively, enable CosmoCLIP to
achieve superior generalization across various in-domain and out-of-domain
tasks. Our results demonstrate that CosmoCLIP is a straightforward yet powerful
framework, significantly outperforming CLIP in zero-shot classification and
image-text retrieval tasks.
CosmoCLIP enhances astronomical image-text contrastive learning by fine-tuning the pre-trained CLIP model using SpaceNet and BLIP captions, achieving superior generalization and performance in zero-shot classification and image-text retrieval.
contrastive learning CLIP model SpaceNet BLIP FLARE zero-shot classification image-text retrieval