StructEval: Benchmarking LLMs' Capabilities to Generate Structural
  Outputs
As Large Language Models (LLMs) become integral to software development
workflows, their ability to generate structured outputs has become critically
important. We introduce StructEval, a comprehensive benchmark for evaluating
LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and
renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks,
StructEval systematically evaluates structural fidelity across diverse formats
through two paradigms: 1) generation tasks, producing structured output from
natural language prompts, and 2) conversion tasks, translating between
structured formats. Our benchmark encompasses 18 formats and 44 types of task,
with novel metrics for format adherence and structural correctness. Results
reveal significant performance gaps, even state-of-the-art models like o1-mini
achieve only 75.58 average score, with open-source alternatives lagging
approximately 10 points behind. We find generation tasks more challenging than
conversion tasks, and producing correct visual content more difficult than
generating text-only structures.
6835744884d4600675a4449c
Jialin Yang
claimed_verified
2025-05-28T09:00:22.494Z
65d78a07ab0790921b087b68
/avatars/869bbdb12f12b0dd46ee9f503416288e.svg
Jialin Yang
user
Sherloc512
6835744884d4600675a4449d
Dongfu Jiang
admin_assigned
2025-05-27T12:58:01.881Z
62567c86d444a9b5a0ec51c1
https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
Dongfu Jiang
user
DongfuJiang
6835744884d4600675a4449e
Lipeng He
claimed_verified
2025-05-28T09:00:20.417Z
6433707307bad11484af1d2a
https://cdn-avatars.huggingface.co/v1/production/uploads/6433707307bad11484af1d2a/w5zB-zstJzY561n6q7m4D.jpeg
Lipeng (Tony) He
user
ttttonyhe
6835744884d4600675a4449f
Sherman Siu
admin_assigned
2025-05-27T12:57:49.231Z
630d73387dacb93b335b00e9
/avatars/2ded3550aedf77a58175a66319d01820.svg
Sherman Siu
user
shermansiu
6835744884d4600675a444a0
Yuxuan Zhang
claimed_verified
2025-05-28T08:59:36.788Z
63b908d0e3c78740d8e950d0
/avatars/3e80075e92aebdfea712f70b00d5ec7d.svg
Yuxuan Zhang
user
Reacherx
6835744884d4600675a444a1
Disen Liao
admin_assigned
2025-05-27T12:57:33.699Z
66c6853c13962d19a8395499
/avatars/3066df4244a064e49880c19fd83c9426.svg
disen-liao
user
Dliao
6835744884d4600675a444a2
Zhuofeng Li
admin_assigned
2025-05-27T12:57:27.171Z
66349404f2c753240d02952a
https://cdn-avatars.huggingface.co/v1/production/uploads/66349404f2c753240d02952a/xKBKicwyk7BoOITQPwBJn.png
ZhuofengLi
user
ZhuofengLi
6835744884d4600675a444a3
Huaye Zeng
6835744884d4600675a444a4
Yiming Jia
admin_assigned
2025-05-27T12:57:12.783Z
6721451d41cc176331607843
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/HQ6_0b2Y-X1ORS6tAz2cq.png
Yiming Jia
user
jymmmmm
6835744884d4600675a444a5
Haozhe Wang
claimed_verified
2025-06-09T10:13:07.420Z
65bf52f0259bc6caeb74f8bf
/avatars/b38392e954466df784a5760ded5df804.svg
Haozhe Wang
user
JasperHaozhe
6835744884d4600675a444a6
Benjamin Schneider
admin_assigned
2025-05-27T12:56:55.022Z
6669ffeecf27eed0f2f2581e
/avatars/6a1012cce15da12fa3bd6f9e0f87d100.svg
Benjamin Schneider
user
BenSchneider
6835744884d4600675a444a7
Chi Ruan
admin_assigned
2025-05-27T12:56:34.920Z
673b578551d8d86ed0abcb7c
/avatars/41dbd6103456569efb707601930fec7e.svg
chiruan
user
chiruan
6835744884d4600675a444a8
Wentao Ma
claimed_verified
2025-05-27T15:49:46.788Z
65c387c807a1445dfe1e9452
https://cdn-avatars.huggingface.co/v1/production/uploads/65c387c807a1445dfe1e9452/NCsYIYrCQxQcPBiMe_4ZI.jpeg
Wentao Ma
user
tonymwt
6835744884d4600675a444a9
Zhiheng Lyu
admin_assigned
2025-05-27T12:56:27.752Z
6114ea319ae90b69cb29fc92
https://cdn-avatars.huggingface.co/v1/production/uploads/6114ea319ae90b69cb29fc92/uw2n4iiXGD5v5iUocmbxE.jpeg
Zhiheng Lyu
user
cogito233
6835744884d4600675a444aa
Yifei Wang
claimed_verified
2025-05-28T08:59:38.669Z
683623338f559a703165971d
/avatars/73bae160a706f831110f235df41a2ced.svg
Yifei Wang
user
yifei-wang
6835744884d4600675a444ab
Yi Lu
claimed_verified
2025-09-03T19:26:32.202Z
65095591cc02352e1e106d0a
/avatars/55bb1dcc31fca1659bcea57d15da50e5.svg
Tom Lu
user
eigentom
6835744884d4600675a444ac
Quy Duc Do
admin_assigned
2025-05-27T12:55:41.859Z
65fa5fa1180d06d37c845d51
/avatars/a04eb59f0a3a3b5800ae443a8eaab785.svg
Quy Duc Do
user
qd2do
6835744884d4600675a444ad
Ziyan Jiang
admin_assigned
2025-05-27T12:55:33.868Z
64778fb8168cb428e00f69b0
https://cdn-avatars.huggingface.co/v1/production/uploads/64778fb8168cb428e00f69b0/D_XYg74zHG9K3HUJj_gD4.jpeg
Ziyan Jiang
user
ziyjiang
6835744884d4600675a444ae
Ping Nie
admin_assigned
2025-05-27T12:55:11.300Z
65358802a920f38780b3248a
/avatars/9415510b598079973c2b0436ad12db9c.svg
Ping Nie
user
pingnieuk
6835744884d4600675a444af
Wenhu Chen
admin_assigned
2025-05-27T12:54:56.382Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
StructEval: Benchmarking LLMs' Capabilities to Generate Structural
  Outputs
As Large Language Models (LLMs) become integral to software development
workflows, their ability to generate structured outputs has become critically
important. We introduce StructEval, a comprehensive benchmark for evaluating
LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and
renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks,
StructEval systematically evaluates structural fidelity across diverse formats
through two paradigms: 1) generation tasks, producing structured output from
natural language prompts, and 2) conversion tasks, translating between
structured formats. Our benchmark encompasses 18 formats and 44 types of task,
with novel metrics for format adherence and structural correctness. Results
reveal significant performance gaps, even state-of-the-art models like o1-mini
achieve only 75.58 average score, with open-source alternatives lagging
approximately 10 points behind. We find generation tasks more challenging than
conversion tasks, and producing correct visual content more difficult than
generating text-only structures.
StructEval benchmarks Large Language Models for generating and converting structured outputs, highlighting performance gaps and challenges in producing visual content.
Large Language Models (LLMs) StructEval structured outputs JSON YAML CSV HTML React SVG generation tasks conversion tasks format adherence structural correctness