Inject Semantic Concepts into Image Tagging for Open-Set Recognition
In this paper, we introduce the Recognize Anything Plus Model~(RAM++), a
fundamental image recognition model with strong open-set recognition
capabilities, by injecting semantic concepts into image tagging training
framework. Previous approaches are either image tagging models constrained by
limited semantics, or vision-language models with shallow interaction for
suboptimal performance in multi-tag recognition. In contrast, RAM++ integrates
image-text alignment and image-tagging within a unified fine-grained
interaction framework based on image-tags-text triplets. This design enables
RAM++ not only excel in identifying predefined categories, but also
significantly augment the recognition ability in open-set categories. Moreover,
RAM++ employs large language models~(LLMs) to generate diverse visual tag
descriptions, pioneering the integration of LLM's knowledge into image tagging
training. This approach empowers RAM++ to integrate visual description concepts
for open-set recognition during inference. Evaluations on comprehensive image
recognition benchmarks demonstrate RAM++ exceeds existing state-of-the-art
(SOTA) fundamental image recognition models on most aspects. Specifically, for
predefined common-used tag categories, RAM++ showcases 10.2 mAP and 15.4 mAP
enhancements over CLIP on OpenImages and ImageNet. For open-set categories
beyond predefined, RAM++ records improvements of 5 mAP and 6.4 mAP over CLIP
and RAM respectively on OpenImages. For diverse human-object interaction
phrases, RAM++ achieves 7.8 mAP and 4.7 mAP improvements on the HICO benchmark.
Code, datasets and pre-trained models are available at
https://github.com/xinyu1205/recognize-anything.
6538858bbf790526ef3ba93f
Xinyu Huang
admin_assigned
2023-10-26T11:50:59.370Z
63bf7ba8da08ed0544ff20e9
https://cdn-avatars.huggingface.co/v1/production/uploads/1673493776367-63bf7ba8da08ed0544ff20e9.jpeg
Xinyu Huang
user
xinyu1205
6538858bbf790526ef3ba940
Yi-Jie Huang
admin_assigned
2023-10-26T11:51:18.420Z
648136af7d65d9ac17346bac
/avatars/7fe65fe712b69f16d81123f0d67719ab.svg
Yijie Huang
user
asdfagafgs
6538858bbf790526ef3ba941
Youcai Zhang
admin_assigned
2023-10-26T11:51:26.531Z
6480327e136d429d0671ecd6
/avatars/306decb0cee314df2c4bf8ad61938147.svg
Youcai Zhang
user
Coler
6538858bbf790526ef3ba942
Weiwei Tian
6538858bbf790526ef3ba943
Rui Feng
6538858bbf790526ef3ba944
Yuejie Zhang
6538858bbf790526ef3ba945
Yanchun Xie
admin_assigned
2023-10-26T11:52:12.964Z
62a69c16db0b844f8b15e6e1
/avatars/971811e630bd5354a008f6abe3bffcce.svg
yanchun xie
user
xyc
6538858bbf790526ef3ba946
Yaqian Li
6538858bbf790526ef3ba947
Lei Zhang
Inject Semantic Concepts into Image Tagging for Open-Set Recognition
In this paper, we introduce the Recognize Anything Plus Model~(RAM++), a
fundamental image recognition model with strong open-set recognition
capabilities, by injecting semantic concepts into image tagging training
framework. Previous approaches are either image tagging models constrained by
limited semantics, or vision-language models with shallow interaction for
suboptimal performance in multi-tag recognition. In contrast, RAM++ integrates
image-text alignment and image-tagging within a unified fine-grained
interaction framework based on image-tags-text triplets. This design enables
RAM++ not only excel in identifying predefined categories, but also
significantly augment the recognition ability in open-set categories. Moreover,
RAM++ employs large language models~(LLMs) to generate diverse visual tag
descriptions, pioneering the integration of LLM's knowledge into image tagging
training. This approach empowers RAM++ to integrate visual description concepts
for open-set recognition during inference. Evaluations on comprehensive image
recognition benchmarks demonstrate RAM++ exceeds existing state-of-the-art
(SOTA) fundamental image recognition models on most aspects. Specifically, for
predefined common-used tag categories, RAM++ showcases 10.2 mAP and 15.4 mAP
enhancements over CLIP on OpenImages and ImageNet. For open-set categories
beyond predefined, RAM++ records improvements of 5 mAP and 6.4 mAP over CLIP
and RAM respectively on OpenImages. For diverse human-object interaction
phrases, RAM++ achieves 7.8 mAP and 4.7 mAP improvements on the HICO benchmark.
Code, datasets and pre-trained models are available at
https://github.com/xinyu1205/recognize-anything.
RAM++, a unified image-text interaction framework, enhances open-set recognition and predefined category identification by integrating large language models and semantic concepts into image tagging.
RAM++ open-set recognition image tagging image-text alignment image-tags-text triplets large language models visual description concepts OpenImages ImageNet HICO benchmark mAP