PaliGemma 2: A Family of Versatile VLMs for Transfer
PaliGemma 2 is an upgrade of the PaliGemma open Vision-Language Model (VLM)
based on the Gemma 2 family of language models. We combine the SigLIP-So400m
vision encoder that was also used by PaliGemma with the whole range of Gemma 2
models, from the 2B one all the way up to the 27B model. We train these models
at three resolutions (224px, 448px, and 896px) in multiple stages to equip them
with broad knowledge for transfer via fine-tuning. The resulting family of base
models covering different model sizes and resolutions allows us to investigate
factors impacting transfer performance (such as learning rate) and to analyze
the interplay between the type of task, model size, and resolution. We further
increase the number and breadth of transfer tasks beyond the scope of PaliGemma
including different OCR-related tasks such as table structure recognition,
molecular structure recognition, music score recognition, as well as long
fine-grained captioning and radiography report generation, on which PaliGemma 2
obtains state-of-the-art results.
675159fd74b66f2e014ede09
Andreas Steiner
675159fd74b66f2e014ede0a
André Susano Pinto
675159fd74b66f2e014ede0b
Michael Tschannen
admin_assigned
2024-12-05T10:40:57.297Z
6489893e1ec8356ba5bb9777
/avatars/54354c1e5774cadd1d83d42054e9d96b.svg
Michael Tschannen
user
mitsch
675159fd74b66f2e014ede0c
Daniel Keysers
claimed_verified
2024-12-05T09:43:54.472Z
662fc73ccee11621a1067db1
/avatars/4fbfbc0621f631d8e95a2a642fa0cd27.svg
Daniel Keysers
user
dkeysers
675159fd74b66f2e014ede0d
Xiao Wang
675159fd74b66f2e014ede0e
Yonatan Bitton
admin_assigned
2024-12-05T10:40:46.933Z
632e0771ae0a7b1fc95630bf
https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
Yonatan
user
yonatanbitton
675159fd74b66f2e014ede0f
Alexey Gritsenko
admin_assigned
2024-12-05T10:40:40.352Z
62d8f9887b8dc0ba17271415
/avatars/12ec78d34fd849bad44217b212f31e98.svg
Alexey Gritsenko
user
AlexeyG
675159fd74b66f2e014ede10
Matthias Minderer
admin_assigned
2024-12-05T10:40:35.142Z
649ac6e57c36fc2dc6e6b0f4
https://cdn-avatars.huggingface.co/v1/production/uploads/649ac6e57c36fc2dc6e6b0f4/Scd3LiYW5Y2qunDDHnPWw.jpeg
Matthias Minderer
user
mjlm
675159fd74b66f2e014ede11
Anthony Sherbondy
admin_assigned
2024-12-05T10:40:29.132Z
65986c7fbf533e3c0dd00d83
/avatars/d038ad9e50002ab4ef99e34d4ba369e7.svg
Anthony Sherbondy
user
asherbondy
675159fd74b66f2e014ede12
Shangbang Long
admin_assigned
2024-12-05T10:40:23.703Z
6525e3d136f98a576aa3d2a8
/avatars/b5a12dd73ff05ae38c4d60d685db7c08.svg
ShangbangLong
user
Shangbang
675159fd74b66f2e014ede13
Siyang Qin
675159fd74b66f2e014ede14
Reeve Ingle
675159fd74b66f2e014ede15
Emanuele Bugliarello
admin_assigned
2024-12-05T10:40:04.842Z
626a9284783d5891f45beb53
/avatars/8737662babc5d42cafba6087ab33e716.svg
Emanuele Bugliarello
user
e-bug
675159fd74b66f2e014ede16
Sahar Kazemzadeh
675159fd74b66f2e014ede17
Thomas Mesnard
admin_assigned
2024-12-05T10:39:55.694Z
6639adab8eb8c9c62ac3fb15
/avatars/83133b04922143be1d8a1fb71fb2938f.svg
Thomas Mesnard
user
thomasmesnard
675159fd74b66f2e014ede18
Ibrahim Alabdulmohsin
admin_assigned
2024-12-05T10:39:49.864Z
630545da20668afe24860235
/avatars/5d82be2e7412bff1af15cc5eafa60b7d.svg
Ibrahim Alabdulmohsin
user
ibomohsin
675159fd74b66f2e014ede19
Lucas Beyer
admin_assigned
2024-12-05T10:39:44.061Z
642d334ff65714b4585f2de4
https://cdn-avatars.huggingface.co/v1/production/uploads/642d334ff65714b4585f2de4/gxBynq5KyoUP0VlAQD3-w.jpeg
Lucas Beyer
user
giffmana
675159fd74b66f2e014ede1a
Xiaohua Zhai
admin_assigned
2024-12-05T10:39:18.732Z
65dcd90082bddd501f68174b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/M2bc9PyKeFs1cCXjTfGGq.jpeg
Xiaohua Zhai
user
xiaohuazhai
PaliGemma 2: A Family of Versatile VLMs for Transfer
PaliGemma 2 is an upgrade of the PaliGemma open Vision-Language Model (VLM)
based on the Gemma 2 family of language models. We combine the SigLIP-So400m
vision encoder that was also used by PaliGemma with the whole range of Gemma 2
models, from the 2B one all the way up to the 27B model. We train these models
at three resolutions (224px, 448px, and 896px) in multiple stages to equip them
with broad knowledge for transfer via fine-tuning. The resulting family of base
models covering different model sizes and resolutions allows us to investigate
factors impacting transfer performance (such as learning rate) and to analyze
the interplay between the type of task, model size, and resolution. We further
increase the number and breadth of transfer tasks beyond the scope of PaliGemma
including different OCR-related tasks such as table structure recognition,
molecular structure recognition, music score recognition, as well as long
fine-grained captioning and radiography report generation, on which PaliGemma 2
obtains state-of-the-art results.
PaliGemma 2 integrates a SigLIP-So400m vision encoder with Gemma 2 models of varying sizes and resolutions, advancing transfer performance across diverse vision-language tasks, including OCR and captioning.
SigLIP-So400m vision encoder Gemma 2 models transfer performance model sizes resolutions OCR table structure recognition molecular structure recognition music score recognition long fine-grained captioning radiography report generation