LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent
  Training
Digital agents require diverse, large-scale UI trajectories to generalize
across real-world tasks, yet collecting such data is prohibitively expensive in
both human annotation, infra and engineering perspectives. To this end, we
introduce UI-Simulator, a scalable paradigm that generates
structured UI states and transitions to synthesize training trajectories at
scale. Our paradigm integrates a digital world simulator for diverse UI states,
a guided rollout process for coherent exploration, and a trajectory wrapper
that produces high-quality and diverse trajectories for agent training. We
further propose UI-Simulator-Grow, a targeted scaling strategy that
enables more rapid and data-efficient scaling by prioritizing high-impact tasks
and synthesizes informative trajectory variants. Experiments on WebArena and
AndroidWorld show that UI-Simulator rivals or surpasses open-source agents
trained on real UIs with significantly better robustness, despite using weaker
teacher models. Moreover, UI-Simulator-Grow matches the performance of
Llama-3-70B-Instruct using only Llama-3-8B-Instruct as the base model,
highlighting the potential of targeted synthesis scaling paradigm to
continuously and efficiently enhance the digital agents.
68f19da26e0bef323a68fc68
Yiming Wang
68f19da26e0bef323a68fc69
Da Yin
68f19da26e0bef323a68fc6a
Yuedong Cui
68f19da26e0bef323a68fc6b
Ruichen Zheng
68f19da26e0bef323a68fc6c
Zhiqian Li
68f19da26e0bef323a68fc6d
Zongyu Lin
68f19da26e0bef323a68fc6e
Di Wu
68f19da26e0bef323a68fc6f
Xueqing Wu
68f19da26e0bef323a68fc70
Chenchen Ye
68f19da26e0bef323a68fc71
Yu Zhou
68f19da26e0bef323a68fc72
Kai-Wei Chang
LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent
  Training
Digital agents require diverse, large-scale UI trajectories to generalize
across real-world tasks, yet collecting such data is prohibitively expensive in
both human annotation, infra and engineering perspectives. To this end, we
introduce UI-Simulator, a scalable paradigm that generates
structured UI states and transitions to synthesize training trajectories at
scale. Our paradigm integrates a digital world simulator for diverse UI states,
a guided rollout process for coherent exploration, and a trajectory wrapper
that produces high-quality and diverse trajectories for agent training. We
further propose UI-Simulator-Grow, a targeted scaling strategy that
enables more rapid and data-efficient scaling by prioritizing high-impact tasks
and synthesizes informative trajectory variants. Experiments on WebArena and
AndroidWorld show that UI-Simulator rivals or surpasses open-source agents
trained on real UIs with significantly better robustness, despite using weaker
teacher models. Moreover, UI-Simulator-Grow matches the performance of
Llama-3-70B-Instruct using only Llama-3-8B-Instruct as the base model,
highlighting the potential of targeted synthesis scaling paradigm to
continuously and efficiently enhance the digital agents.
UI-Simulator generates diverse UI trajectories for digital agents using a scalable paradigm, improving robustness and performance with targeted scaling strategies.
UI-Simulator digital world simulator guided rollout process trajectory wrapper UI-Simulator-Grow WebArena AndroidWorld Llama-3-70B-Instruct Llama-3-8B-Instruct