AnyCap Project: A Unified Framework, Dataset, and Benchmark for
  Controllable Omni-modal Captioning
Controllable captioning is essential for precise multimodal alignment and
instruction following, yet existing models often lack fine-grained control and
reliable evaluation protocols. To address this gap, we present the AnyCap
Project, an integrated solution spanning model, dataset, and evaluation. We
introduce AnyCapModel (ACM), a lightweight plug-and-play framework that
enhances the controllability of existing foundation models for omni-modal
captioning without retraining the base model. ACM reuses the original captions
from base models while incorporating user instructions and modality features to
generate improved captions. To remedy the data scarcity in controllable
multimodal captioning, we build AnyCapDataset (ACD), covering three modalities,
28 user-instruction types, and 300\,k high-quality data entries. We further
propose AnyCapEval, a new benchmark that provides more reliable evaluation
metrics for controllable captioning by decoupling content accuracy and
stylistic fidelity. ACM markedly improves caption quality across a diverse set
of base models on AnyCapEval. Notably, ACM-8B raises GPT-4o\'s content scores
by 45\% and style scores by 12\%, and it also achieves substantial gains on
widely used benchmarks such as MIA-Bench and VidCapBench.
6879bba621b37e676c8e4195
Yiming Ren
claimed_verified
2025-07-21T08:30:07.252Z
643375324521083b9d2c4f85
/avatars/a15b0aad0e2e61cf76dcb2559350a0d4.svg
qishisuren
user
qishisuren
6879bba621b37e676c8e4196
Zhiqiang Lin
6879bba621b37e676c8e4197
Yu Li
6879bba621b37e676c8e4198
Gao Meng
6879bba621b37e676c8e4199
Weiyun Wang
claimed_verified
2025-08-25T08:29:06.715Z
619507e7b74b6c591f794340
https://cdn-avatars.huggingface.co/v1/production/uploads/619507e7b74b6c591f794340/JbPDoy6Ko1V1-6oJJwFV8.jpeg
Weiyun Wang
user
Weiyun1025
6879bba621b37e676c8e419a
Junjie Wang
claimed_verified
2025-08-06T19:33:43.548Z
62579c55b98dcaa7e0de285d
https://cdn-avatars.huggingface.co/v1/production/uploads/62579c55b98dcaa7e0de285d/0YUd5nloul_bW9yolDGGo.jpeg
wangjunjie
user
wanng
6879bba621b37e676c8e419b
Zicheng Lin
6879bba621b37e676c8e419c
Jifeng Dai
6879bba621b37e676c8e419d
Yujiu Yang
6879bba621b37e676c8e419e
Wenhai Wang
6879bba621b37e676c8e419f
Ruihang Chu
claimed_verified
2025-07-18T07:49:30.538Z
642e3bcb958faf258a40e89c
/avatars/213501def37dc53032cee17e37fcc4c1.svg
Ruihang Chu
user
Ruihang
AnyCap Project: A Unified Framework, Dataset, and Benchmark for
  Controllable Omni-modal Captioning
Controllable captioning is essential for precise multimodal alignment and
instruction following, yet existing models often lack fine-grained control and
reliable evaluation protocols. To address this gap, we present the AnyCap
Project, an integrated solution spanning model, dataset, and evaluation. We
introduce AnyCapModel (ACM), a lightweight plug-and-play framework that
enhances the controllability of existing foundation models for omni-modal
captioning without retraining the base model. ACM reuses the original captions
from base models while incorporating user instructions and modality features to
generate improved captions. To remedy the data scarcity in controllable
multimodal captioning, we build AnyCapDataset (ACD), covering three modalities,
28 user-instruction types, and 300\,k high-quality data entries. We further
propose AnyCapEval, a new benchmark that provides more reliable evaluation
metrics for controllable captioning by decoupling content accuracy and
stylistic fidelity. ACM markedly improves caption quality across a diverse set
of base models on AnyCapEval. Notably, ACM-8B raises GPT-4o\'s content scores
by 45\% and style scores by 12\%, and it also achieves substantial gains on
widely used benchmarks such as MIA-Bench and VidCapBench.
The AnyCap Project introduces a framework, dataset, and evaluation protocol to enhance controllability and reliability in multimodal captioning.
AnyCapModel ACM omni-modal captioning AnyCapDataset ACD AnyCapEval content accuracy stylistic fidelity MIA-Bench VidCapBench