Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI
  Models for Scatterplot-Related Tasks
AI models are increasingly used for data analysis and visualization, yet
benchmarks rarely address scatterplot-specific tasks, limiting insight into
performance. To address this gap for one of the most common chart types, we
introduce a synthetic, annotated dataset of over 18,000 scatterplots from six
data generators and 17 chart designs, and a benchmark based on it. We evaluate
proprietary models from OpenAI and Google using N-shot prompting on five
distinct tasks derived from annotations of cluster bounding boxes, their center
coordinates, and outlier coordinates. OpenAI models and Gemini 2.5 Flash,
especially when prompted with examples, are viable options for counting
clusters and, in Flash's case, outliers (90%+ Accuracy). However, the results
for localization-related tasks are unsatisfactory: Precision and Recall are
near or below 50%, except for Flash in outlier identification (65.01%).
Furthermore, the impact of chart design on performance appears to be a
secondary factor, but it is advisable to avoid scatterplots with wide aspect
ratios (16:9 and 21:9) or those colored randomly. Supplementary materials are
available at https://github.com/feedzai/biy-paper.
68e64331975ac4c405ef2247
João Palmeiro
claimed_verified
2025-10-08T11:48:46.449Z
67ec4013d674ac4ba71dd264
https://cdn-avatars.huggingface.co/v1/production/uploads/67ec4013d674ac4ba71dd264/eY5KFSwHfiRY7MwS0P2PY.jpeg
João Palmeiro
user
joaompalmeiro
68e64331975ac4c405ef2248
Diogo Duarte
68e64331975ac4c405ef2249
Rita Costa
68e64331975ac4c405ef224a
Pedro Bizarro
Benchmark It Yourself (BIY): Preparing a Dataset and Benchmarking AI
  Models for Scatterplot-Related Tasks
AI models are increasingly used for data analysis and visualization, yet
benchmarks rarely address scatterplot-specific tasks, limiting insight into
performance. To address this gap for one of the most common chart types, we
introduce a synthetic, annotated dataset of over 18,000 scatterplots from six
data generators and 17 chart designs, and a benchmark based on it. We evaluate
proprietary models from OpenAI and Google using N-shot prompting on five
distinct tasks derived from annotations of cluster bounding boxes, their center
coordinates, and outlier coordinates. OpenAI models and Gemini 2.5 Flash,
especially when prompted with examples, are viable options for counting
clusters and, in Flash's case, outliers (90%+ Accuracy). However, the results
for localization-related tasks are unsatisfactory: Precision and Recall are
near or below 50%, except for Flash in outlier identification (65.01%).
Furthermore, the impact of chart design on performance appears to be a
secondary factor, but it is advisable to avoid scatterplots with wide aspect
ratios (16:9 and 21:9) or those colored randomly. Supplementary materials are
available at https://github.com/feedzai/biy-paper.
A benchmark for scatterplot-specific tasks using synthetic datasets evaluates AI models' performance in counting clusters and identifying outliers, with mixed results for localization tasks.
N-shot prompting cluster bounding boxes center coordinates outlier coordinates localization-related tasks Precision Recall aspect ratios