Skill Discovery for Software Scripting Automation via Offline
  Simulations with LLMs
Scripting interfaces enable users to automate tasks and customize software
workflows, but creating scripts traditionally requires programming expertise
and familiarity with specific APIs, posing barriers for many users. While Large
Language Models (LLMs) can generate code from natural language queries, runtime
code generation is severely limited due to unverified code, security risks,
longer response times, and higher computational costs. To bridge the gap, we
propose an offline simulation framework to curate a software-specific skillset,
a collection of verified scripts, by exploiting LLMs and publicly available
scripting guides. Our framework comprises two components: (1) task creation,
using top-down functionality guidance and bottom-up API synergy exploration to
generate helpful tasks; and (2) skill generation with trials, refining and
validating scripts based on execution feedback. To efficiently navigate the
extensive API landscape, we introduce a Graph Neural Network (GNN)-based link
prediction model to capture API synergy, enabling the generation of skills
involving underutilized APIs and expanding the skillset's diversity.
Experiments with Adobe Illustrator demonstrate that our framework significantly
improves automation success rates, reduces response time, and saves runtime
token costs compared to traditional runtime code generation. This is the first
attempt to use software scripting interfaces as a testbed for LLM-based
systems, highlighting the advantages of leveraging execution feedback in a
controlled environment and offering valuable insights into aligning AI
capabilities with user needs in specialized software domains.
681561d98a46db172ad08f35
Paiheng Xu
admin_assigned
2025-05-05T07:49:56.029Z
62505101a0f6b0ed18114323
/avatars/fdd0cd6abba33740b037b71876e8af41.svg
Paiheng Xu
user
paiheng
681561d98a46db172ad08f36
Gang Wu
681561d98a46db172ad08f37
Xiang Chen
admin_assigned
2025-05-05T07:50:05.179Z
64c073417255ff868721fb54
/avatars/c2a00114598aa89f071b937f0af4077f.svg
Xiang CHEN
user
xiangchen-dvi
681561d98a46db172ad08f38
Tong Yu
681561d98a46db172ad08f39
Chang Xiao
681561d98a46db172ad08f3a
Franck Dernoncourt
claimed_verified
2025-05-04T10:04:28.382Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
681561d98a46db172ad08f3b
Tianyi Zhou
claimed_verified
2025-05-15T10:33:17.253Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
681561d98a46db172ad08f3c
Wei Ai
681561d98a46db172ad08f3d
Viswanathan Swaminathan
admin_assigned
2025-05-05T07:50:32.106Z
66a443ed55a50dd32d4034e8
/avatars/5a75cfa6137459f01309ecc38ea386cf.svg
Viswanathan Swaminathan
user
viswathegreat
Skill Discovery for Software Scripting Automation via Offline
  Simulations with LLMs
Scripting interfaces enable users to automate tasks and customize software
workflows, but creating scripts traditionally requires programming expertise
and familiarity with specific APIs, posing barriers for many users. While Large
Language Models (LLMs) can generate code from natural language queries, runtime
code generation is severely limited due to unverified code, security risks,
longer response times, and higher computational costs. To bridge the gap, we
propose an offline simulation framework to curate a software-specific skillset,
a collection of verified scripts, by exploiting LLMs and publicly available
scripting guides. Our framework comprises two components: (1) task creation,
using top-down functionality guidance and bottom-up API synergy exploration to
generate helpful tasks; and (2) skill generation with trials, refining and
validating scripts based on execution feedback. To efficiently navigate the
extensive API landscape, we introduce a Graph Neural Network (GNN)-based link
prediction model to capture API synergy, enabling the generation of skills
involving underutilized APIs and expanding the skillset's diversity.
Experiments with Adobe Illustrator demonstrate that our framework significantly
improves automation success rates, reduces response time, and saves runtime
token costs compared to traditional runtime code generation. This is the first
attempt to use software scripting interfaces as a testbed for LLM-based
systems, highlighting the advantages of leveraging execution feedback in a
controlled environment and offering valuable insights into aligning AI
capabilities with user needs in specialized software domains.
An offline framework using LLMs and GNNs curates verified scripts for specific software APIs, improving automation success rates and reducing runtime costs.
Large Language Models runtime code generation Graph Neural Network link prediction model API synergy verified scripts task creation skill generation software scripting interfaces execution feedback Adobe Illustrator automation success rates runtime token costs