Can Large Language Models Understand Symbolic Graphics Programs?
Assessing the capabilities of large language models (LLMs) is often
challenging, in part, because it is hard to find tasks to which they have not
been exposed during training. We take one step to address this challenge by
turning to a new task: focusing on symbolic graphics programs, which are a
popular representation for graphics content that procedurally generates visual
data. LLMs have shown exciting promise towards program synthesis, but do they
understand symbolic graphics programs? Unlike conventional programs, symbolic
graphics programs can be translated to graphics content. Here, we characterize
an LLM's understanding of symbolic programs in terms of their ability to answer
questions related to the graphics content. This task is challenging as the
questions are difficult to answer from the symbolic programs alone -- yet, they
would be easy to answer from the corresponding graphics content as we verify
through a human experiment. To understand symbolic programs, LLMs may need to
possess the ability to imagine how the corresponding graphics content would
look without directly accessing the rendered visual content. We use this task
to evaluate LLMs by creating a large benchmark for the semantic understanding
of symbolic graphics programs. This benchmark is built via program-graphics
correspondence, hence requiring minimal human efforts. We evaluate current LLMs
on our benchmark to elucidate a preliminary assessment of their ability to
reason about visual scenes from programs. We find that this task distinguishes
existing LLMs and models considered good at reasoning perform better. Lastly,
we introduce Symbolic Instruction Tuning (SIT) to improve this ability.
Specifically, we query GPT4-o with questions and images generated by symbolic
programs. Such data are then used to finetune an LLM. We also find that SIT
data can improve the general instruction following ability of LLMs.
66beb9de6e746f4f3ca27cb7
Zeju Qiu
66beb9de6e746f4f3ca27cb8
Weiyang Liu
66beb9de6e746f4f3ca27cb9
Haiwen Feng
66beb9de6e746f4f3ca27cba
Zhen Liu
66beb9de6e746f4f3ca27cbb
Tim Z. Xiao
66beb9de6e746f4f3ca27cbc
Katherine M. Collins
claimed_verified
2025-07-23T08:38:59.007Z
64514cd06db3be7ee57b6f34
/avatars/3276eeb344f57601e2a2199dd090935b.svg
Katie Collins
user
kmcollins
66beb9de6e746f4f3ca27cbd
Joshua B. Tenenbaum
66beb9de6e746f4f3ca27cbe
Adrian Weller
66beb9de6e746f4f3ca27cbf
Michael J. Black
66beb9de6e746f4f3ca27cc0
Bernhard Schölkopf
Can Large Language Models Understand Symbolic Graphics Programs?
Assessing the capabilities of large language models (LLMs) is often
challenging, in part, because it is hard to find tasks to which they have not
been exposed during training. We take one step to address this challenge by
turning to a new task: focusing on symbolic graphics programs, which are a
popular representation for graphics content that procedurally generates visual
data. LLMs have shown exciting promise towards program synthesis, but do they
understand symbolic graphics programs? Unlike conventional programs, symbolic
graphics programs can be translated to graphics content. Here, we characterize
an LLM's understanding of symbolic programs in terms of their ability to answer
questions related to the graphics content. This task is challenging as the
questions are difficult to answer from the symbolic programs alone -- yet, they
would be easy to answer from the corresponding graphics content as we verify
through a human experiment. To understand symbolic programs, LLMs may need to
possess the ability to imagine how the corresponding graphics content would
look without directly accessing the rendered visual content. We use this task
to evaluate LLMs by creating a large benchmark for the semantic understanding
of symbolic graphics programs. This benchmark is built via program-graphics
correspondence, hence requiring minimal human efforts. We evaluate current LLMs
on our benchmark to elucidate a preliminary assessment of their ability to
reason about visual scenes from programs. We find that this task distinguishes
existing LLMs and models considered good at reasoning perform better. Lastly,
we introduce Symbolic Instruction Tuning (SIT) to improve this ability.
Specifically, we query GPT4-o with questions and images generated by symbolic
programs. Such data are then used to finetune an LLM. We also find that SIT
data can improve the general instruction following ability of LLMs.
A benchmark evaluates large language models' understanding of symbolic graphics programs by assessing their ability to answer questions about the generated graphics content.
large language models (LLMs) symbolic graphics programs program synthesis semantic understanding benchmark program-graphics correspondence Symbolic Instruction Tuning (SIT) finetune instruction following