Can Models Help Us Create Better Models? Evaluating LLMs as Data
  Scientists
We present a benchmark for large language models designed to tackle one of
the most knowledge-intensive tasks in data science: writing feature engineering
code, which requires domain knowledge in addition to a deep understanding of
the underlying problem and data structure. The model is provided with a dataset
description in a prompt and asked to generate code transforming it. The
evaluation score is derived from the improvement achieved by an XGBoost model
fit on the modified dataset compared to the original data. By an extensive
evaluation of state-of-the-art models and comparison to well-established
benchmarks, we demonstrate that the FeatEng of our proposal can cheaply and
efficiently assess the broad capabilities of LLMs, in contrast to the existing
methods.
67242c0223ae9645e6021fbe
Michał Pietruszka
67242c0223ae9645e6021fbf
Łukasz Borchmann
claimed_verified
2024-11-03T14:22:38.885Z
600b381d3cc3b87db94bc0ce
https://cdn-avatars.huggingface.co/v1/production/uploads/600b381d3cc3b87db94bc0ce/I3xpr4gzcG1uXawXBpWpD.jpeg
Łukasz Borchmann
user
Borchmann
67242c0223ae9645e6021fc0
Aleksander Jędrosz
67242c0223ae9645e6021fc1
Paweł Morawiecki
Can Models Help Us Create Better Models? Evaluating LLMs as Data
  Scientists
We present a benchmark for large language models designed to tackle one of
the most knowledge-intensive tasks in data science: writing feature engineering
code, which requires domain knowledge in addition to a deep understanding of
the underlying problem and data structure. The model is provided with a dataset
description in a prompt and asked to generate code transforming it. The
evaluation score is derived from the improvement achieved by an XGBoost model
fit on the modified dataset compared to the original data. By an extensive
evaluation of state-of-the-art models and comparison to well-established
benchmarks, we demonstrate that the FeatEng of our proposal can cheaply and
efficiently assess the broad capabilities of LLMs, in contrast to the existing
methods.
A benchmark evaluates large language models for feature engineering tasks using code generation and XGBoost model performance as metrics.
large language models feature engineering XGBoost code generation