MMFactory: A Universal Solution Search Engine for Vision-Language Tasks
With advances in foundational and vision-language models, and effective
fine-tuning techniques, a large number of both general and special-purpose
models have been developed for a variety of visual tasks. Despite the
flexibility and accessibility of these models, no single model is able to
handle all tasks and/or applications that may be envisioned by potential users.
Recent approaches, such as visual programming and multimodal LLMs with
integrated tools aim to tackle complex visual tasks, by way of program
synthesis. However, such approaches overlook user constraints (e.g.,
performance / computational needs), produce test-time sample-specific solutions
that are difficult to deploy, and, sometimes, require low-level instructions
that maybe beyond the abilities of a naive user. To address these limitations,
we introduce MMFactory, a universal framework that includes model and metrics
routing components, acting like a solution search engine across various
available models. Based on a task description and few sample input-output pairs
and (optionally) resource and/or performance constraints, MMFactory can suggest
a diverse pool of programmatic solutions by instantiating and combining
visio-lingual tools from its model repository. In addition to synthesizing
these solutions, MMFactory also proposes metrics and benchmarks performance /
resource characteristics, allowing users to pick a solution that meets their
unique design constraints. From the technical perspective, we also introduced a
committee-based solution proposer that leverages multi-agent LLM conversation
to generate executable, diverse, universal, and robust solutions for the user.
Experimental results show that MMFactory outperforms existing methods by
delivering state-of-the-art solutions tailored to user problem specifications.
Project page is available at https://davidhalladay.github.io/mmfactory_demo.
676e9cfc11998b72ab00be60
Wan-Cyuan Fan
admin_assigned
2024-12-30T21:13:13.288Z
63357efea01bd734f7243c61
/avatars/459d3e35bb6ae60f69a0c7bd77a39d58.svg
WanCyuan Fan
user
ChrisFan
676e9cfc11998b72ab00be61
Tanzila Rahman
admin_assigned
2024-12-30T21:12:47.649Z
64062c3ca577649430c1006b
/avatars/870ac9e8dc15051e97a7a6efd46e8a36.svg
Tanzila Rahman
user
trahman
676e9cfc11998b72ab00be62
Leonid Sigal
MMFactory: A Universal Solution Search Engine for Vision-Language Tasks
With advances in foundational and vision-language models, and effective
fine-tuning techniques, a large number of both general and special-purpose
models have been developed for a variety of visual tasks. Despite the
flexibility and accessibility of these models, no single model is able to
handle all tasks and/or applications that may be envisioned by potential users.
Recent approaches, such as visual programming and multimodal LLMs with
integrated tools aim to tackle complex visual tasks, by way of program
synthesis. However, such approaches overlook user constraints (e.g.,
performance / computational needs), produce test-time sample-specific solutions
that are difficult to deploy, and, sometimes, require low-level instructions
that maybe beyond the abilities of a naive user. To address these limitations,
we introduce MMFactory, a universal framework that includes model and metrics
routing components, acting like a solution search engine across various
available models. Based on a task description and few sample input-output pairs
and (optionally) resource and/or performance constraints, MMFactory can suggest
a diverse pool of programmatic solutions by instantiating and combining
visio-lingual tools from its model repository. In addition to synthesizing
these solutions, MMFactory also proposes metrics and benchmarks performance /
resource characteristics, allowing users to pick a solution that meets their
unique design constraints. From the technical perspective, we also introduced a
committee-based solution proposer that leverages multi-agent LLM conversation
to generate executable, diverse, universal, and robust solutions for the user.
Experimental results show that MMFactory outperforms existing methods by
delivering state-of-the-art solutions tailored to user problem specifications.
Project page is available at https://davidhalladay.github.io/mmfactory_demo.
MMFactory is a universal framework that combines various models and metrics to propose tailored solutions for visual tasks based on user specifications and constraints.
visual programming multimodal LLMs program synthesis model repository solution search engine committee-based solution proposer multi-agent LLM conversation