Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical
  Report
Multimodal Foundation Models (MMFMs) have shown remarkable performance on
various computer vision and natural language processing tasks. However, their
performance on particular tasks such as document understanding is still
limited. They also require more compute, time, and engineering resources to
finetune and deploy compared to traditional, unimodal models. In this report,
we present Multimodal Structured Generation, a general framework which
constrains the output logits of frozen MMFMs to force them to reason before
responding with structured outputs that downstream APIs can parse and use. We
provide a detailed account of our approach, including the technical details,
theoretical discussions, and final evaluation results in the 2nd Multimodal
Foundation Models Challenge hosted by the Computer Vision and Pattern
Recognition (CVPR) conference. Our approach achieved the second highest score
in the hidden test set for Phase 2 and third highest overall. This shows the
method's ability to generalize to unseen tasks. And that simple engineering can
beat expensive & complicated modelling steps as we first discussed in our
paper, Retrieval Augmented Structured Generation: Business Document Information
Extraction as Tool Use. All of our scripts, deployment steps, and evaluation
results can be accessed in https://github.com/leloykun/MMFM-Challenge
6670f843c4148dbf42778bc6
Franz Louis Cesista
extracted_confirmed
2024-06-18T03:07:31.274Z
60d36cdb07da9c17c727090f
https://cdn-avatars.huggingface.co/v1/production/uploads/1647859018354-60d36cdb07da9c17c727090f.png
Franz Louis Cesista
user
leloy
Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical
  Report
Multimodal Foundation Models (MMFMs) have shown remarkable performance on
various computer vision and natural language processing tasks. However, their
performance on particular tasks such as document understanding is still
limited. They also require more compute, time, and engineering resources to
finetune and deploy compared to traditional, unimodal models. In this report,
we present Multimodal Structured Generation, a general framework which
constrains the output logits of frozen MMFMs to force them to reason before
responding with structured outputs that downstream APIs can parse and use. We
provide a detailed account of our approach, including the technical details,
theoretical discussions, and final evaluation results in the 2nd Multimodal
Foundation Models Challenge hosted by the Computer Vision and Pattern
Recognition (CVPR) conference. Our approach achieved the second highest score
in the hidden test set for Phase 2 and third highest overall. This shows the
method's ability to generalize to unseen tasks. And that simple engineering can
beat expensive & complicated modelling steps as we first discussed in our
paper, Retrieval Augmented Structured Generation: Business Document Information
Extraction as Tool Use. All of our scripts, deployment steps, and evaluation
results can be accessed in https://github.com/leloykun/MMFM-Challenge
A framework constraining multimodal foundation models to produce structured outputs enhances their performance in document understanding tasks.
Multimodal Foundation Models document understanding unimodal models multimodal structured generation output logits structured outputs second Multimodal Foundation Models Challenge CVPR conference Retrieval Augmented Structured Generation business document information extraction