Struct-Bench: A Benchmark for Differentially Private Structured Text
  Generation
Differentially private (DP) synthetic data generation is a promising
technique for utilizing private datasets that otherwise cannot be exposed for
model training or other analytics. While much research literature has focused
on generating private unstructured text and image data, in enterprise settings,
structured data (e.g., tabular) is more common, often including natural
language fields or components. Existing synthetic data evaluation techniques
(e.g., FID) struggle to capture the structural properties and correlations of
such datasets. In this work, we propose Struct-Bench, a framework and benchmark
for evaluating synthetic datasets derived from structured datasets that contain
natural language data. The Struct-Bench framework requires users to provide a
representation of their dataset structure as a Context-Free Grammar (CFG). Our
benchmark comprises 5 real-world and 2 synthetically generated datasets, each
annotated with CFGs. We show that these datasets demonstrably present a great
challenge even for state-of-the-art DP synthetic data generation methods.
Struct-Bench also includes reference implementations of different metrics and a
leaderboard, thereby providing researchers a standardized evaluation platform
to benchmark and investigate privacy-preserving synthetic data generation
methods. Further, we also present a case study showing how to use Struct-Bench
to improve the synthetic data quality of Private Evolution (PE) on structured
data. The benchmark and the leaderboard have been publicly made available at
https://struct-bench.github.io.
68c984f5fbbc67c7b9472b30
Shuaiqi Wang
claimed_verified
2025-09-17T13:05:31.782Z
66ef2897e4b4a1c1cf34af7e
/avatars/ddd78c7ce336b20be878835ffc8536b9.svg
Shuaiqi Wang
user
Shuaiqiw
68c984f5fbbc67c7b9472b31
Vikas Raunak
68c984f5fbbc67c7b9472b32
Arturs Backurs
68c984f5fbbc67c7b9472b33
Victor Reis
68c984f5fbbc67c7b9472b34
Pei Zhou
68c984f5fbbc67c7b9472b35
Sihao Chen
68c984f5fbbc67c7b9472b36
Longqi Yang
68c984f5fbbc67c7b9472b37
Zinan Lin
claimed_verified
2025-09-22T10:31:59.406Z
64c832a8c547ed5243d29630
https://cdn-avatars.huggingface.co/v1/production/uploads/64c832a8c547ed5243d29630/a46V0xOVRVknM9mUjKkTf.jpeg
Zinan Lin
user
fjxmlzn
68c984f5fbbc67c7b9472b38
Sergey Yekhanin
68c984f5fbbc67c7b9472b39
Giulia Fanti
Struct-Bench: A Benchmark for Differentially Private Structured Text
  Generation
Differentially private (DP) synthetic data generation is a promising
technique for utilizing private datasets that otherwise cannot be exposed for
model training or other analytics. While much research literature has focused
on generating private unstructured text and image data, in enterprise settings,
structured data (e.g., tabular) is more common, often including natural
language fields or components. Existing synthetic data evaluation techniques
(e.g., FID) struggle to capture the structural properties and correlations of
such datasets. In this work, we propose Struct-Bench, a framework and benchmark
for evaluating synthetic datasets derived from structured datasets that contain
natural language data. The Struct-Bench framework requires users to provide a
representation of their dataset structure as a Context-Free Grammar (CFG). Our
benchmark comprises 5 real-world and 2 synthetically generated datasets, each
annotated with CFGs. We show that these datasets demonstrably present a great
challenge even for state-of-the-art DP synthetic data generation methods.
Struct-Bench also includes reference implementations of different metrics and a
leaderboard, thereby providing researchers a standardized evaluation platform
to benchmark and investigate privacy-preserving synthetic data generation
methods. Further, we also present a case study showing how to use Struct-Bench
to improve the synthetic data quality of Private Evolution (PE) on structured
data. The benchmark and the leaderboard have been publicly made available at
https://struct-bench.github.io.
Struct-Bench is a framework and benchmark for evaluating synthetic structured datasets with natural language components, addressing challenges in differentially private synthetic data generation.
differentially private synthetic data generation structured data tabular data natural language data Context-Free Grammar CFG Struct-Bench benchmark privacy-preserving Private Evolution PE