DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile
  Phone Agents
Mobile Phone Agents (MPAs) have emerged as a promising research direction due
to their broad applicability across diverse scenarios. While Multimodal Large
Language Models (MLLMs) serve as the foundation for MPAs, their effectiveness
in handling multiple mobile phone tasks simultaneously remains limited.
Although multitask supervised fine-tuning (SFT) is widely adopted for multitask
learning, existing approaches struggle to determine optimal training data
compositions for peak performance. To address this challenge, we propose DaMo
(Data Mixture Optimizer) - a novel solution employing a trainable network that
predicts optimal data mixtures by forecasting downstream task performance for
any given dataset ratio. To support comprehensive evaluation, we introduce
PhoneAgentBench, the first specialized benchmark to evaluate MLLMs on
multimodal mobile phone tasks, comprising 1235 QA pairs spanning diverse
real-world industrial mobile application scenarios. Demonstrating strong
predictive capability (R^2=0.81) in small-scale pilot experiments, DaMo
efficiently extrapolates optimal data mixing configurations. Our results show
DaMo achieves a 3.38% performance improvement on PhoneAgentBench compared to
alternative methods. Furthermore, extensive experiments across established
benchmarks including BFCL-v3, MME-Reasoning, MME-Perception, and OCRBench
reveal DaMo's superior generalization, outperforming other approaches by 2.57%
in terms of average score. When used solely for MLLM optimization on the
BFCL-v3 task, DaMo improves the metrics by 12.47% than other methods. Notably,
DaMo maintains robust scalability, preserving its effectiveness when applied to
other model architectures. The code and dataset are available at
https://github.com/OPPO-Mente-Lab/DaMo.git
68f98a97b9b2e4ae0467377e
Kai Shi
68f98a97b9b2e4ae0467377f
Jun Yang
68f98a97b9b2e4ae04673780
Ni Yang
68f98a97b9b2e4ae04673781
Binqiang Pan
68f98a97b9b2e4ae04673782
Qingsong Xie
68f98a97b9b2e4ae04673783
Chao Zhang
68f98a97b9b2e4ae04673784
Zhenyu Yang
68f98a97b9b2e4ae04673785
Tianhuang Su
68f98a97b9b2e4ae04673786
Haonan Lu
DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile
  Phone Agents
Mobile Phone Agents (MPAs) have emerged as a promising research direction due
to their broad applicability across diverse scenarios. While Multimodal Large
Language Models (MLLMs) serve as the foundation for MPAs, their effectiveness
in handling multiple mobile phone tasks simultaneously remains limited.
Although multitask supervised fine-tuning (SFT) is widely adopted for multitask
learning, existing approaches struggle to determine optimal training data
compositions for peak performance. To address this challenge, we propose DaMo
(Data Mixture Optimizer) - a novel solution employing a trainable network that
predicts optimal data mixtures by forecasting downstream task performance for
any given dataset ratio. To support comprehensive evaluation, we introduce
PhoneAgentBench, the first specialized benchmark to evaluate MLLMs on
multimodal mobile phone tasks, comprising 1235 QA pairs spanning diverse
real-world industrial mobile application scenarios. Demonstrating strong
predictive capability (R^2=0.81) in small-scale pilot experiments, DaMo
efficiently extrapolates optimal data mixing configurations. Our results show
DaMo achieves a 3.38% performance improvement on PhoneAgentBench compared to
alternative methods. Furthermore, extensive experiments across established
benchmarks including BFCL-v3, MME-Reasoning, MME-Perception, and OCRBench
reveal DaMo's superior generalization, outperforming other approaches by 2.57%
in terms of average score. When used solely for MLLM optimization on the
BFCL-v3 task, DaMo improves the metrics by 12.47% than other methods. Notably,
DaMo maintains robust scalability, preserving its effectiveness when applied to
other model architectures. The code and dataset are available at
https://github.com/OPPO-Mente-Lab/DaMo.git
DaMo, a trainable network optimizing data mixtures for Multimodal Large Language Models, enhances performance across various mobile phone tasks and benchmarks.
Multimodal Large Language Models multitask supervised fine-tuning DaMo Data Mixture Optimizer PhoneAgentBench BFCL-v3 MME-Reasoning MME-Perception OCRBench