Lightweight Neural App Control
This paper introduces a novel mobile phone control architecture, termed ``app
agents", for efficient interactions and controls across various Android apps.
The proposed Lightweight Multi-modal App Control (LiMAC) takes as input a
textual goal and a sequence of past mobile observations, such as screenshots
and corresponding UI trees, to generate precise actions. To address the
computational constraints inherent to smartphones, within LiMAC, we introduce a
small Action Transformer (AcT) integrated with a fine-tuned vision-language
model (VLM) for real-time decision-making and task execution. We evaluate LiMAC
on two open-source mobile control datasets, demonstrating the superior
performance of our small-form-factor approach against fine-tuned versions of
open-source VLMs, such as Florence2 and Qwen2-VL. It also significantly
outperforms prompt engineering baselines utilising closed-source foundation
models like GPT-4o. More specifically, LiMAC increases the overall action
accuracy by up to 19% compared to fine-tuned VLMs, and up to 42% compared to
prompt-engineering baselines.
6719e4bd2c519a08bd42b9d5
Filippos Christianos
admin_assigned
2024-10-24T08:34:29.693Z
64f46b681d337935d0495d4d
/avatars/cce5a4910617931fb13062b832e14ef8.svg
Filippos Christianos
user
semitable
6719e4bd2c519a08bd42b9d6
Georgios Papoudakis
admin_assigned
2024-10-24T08:34:34.954Z
66e451f4c3b3a128bd078bdf
/avatars/624e2520129f408874f227df04025ec7.svg
Georgios Papoudakis
user
gpap
6719e4bd2c519a08bd42b9d7
Thomas Coste
admin_assigned
2024-10-24T08:34:41.324Z
64ca5357ead94891d1d53cd1
/avatars/187073352d7767f18d5fd50a6ca69fb5.svg
Thomas COSTE
user
Fahren24
6719e4bd2c519a08bd42b9d8
Jianye Hao
6719e4bd2c519a08bd42b9d9
Jun Wang
6719e4bd2c519a08bd42b9da
Kun Shao
admin_assigned
2024-10-24T08:34:58.709Z
66840c818f6c78ebd41f86ca
/avatars/626ffa3d262c02f964f15fa420af414c.svg
Kun Shao
user
ShaoKun-HW
Lightweight Neural App Control
This paper introduces a novel mobile phone control architecture, termed ``app
agents", for efficient interactions and controls across various Android apps.
The proposed Lightweight Multi-modal App Control (LiMAC) takes as input a
textual goal and a sequence of past mobile observations, such as screenshots
and corresponding UI trees, to generate precise actions. To address the
computational constraints inherent to smartphones, within LiMAC, we introduce a
small Action Transformer (AcT) integrated with a fine-tuned vision-language
model (VLM) for real-time decision-making and task execution. We evaluate LiMAC
on two open-source mobile control datasets, demonstrating the superior
performance of our small-form-factor approach against fine-tuned versions of
open-source VLMs, such as Florence2 and Qwen2-VL. It also significantly
outperforms prompt engineering baselines utilising closed-source foundation
models like GPT-4o. More specifically, LiMAC increases the overall action
accuracy by up to 19% compared to fine-tuned VLMs, and up to 42% compared to
prompt-engineering baselines.
LiMAC, a lightweight mobile control system using an Action Transformer and fine-tuned vision-language model, achieves superior performance in real-time decision-making and task execution on Android apps.
Lightweight Multi-modal App Control (LiMAC) Action Transformer (AcT) vision-language model (VLM)