TabSTAR: A Foundation Tabular Model With Semantically Target-Aware
  Representations
While deep learning has achieved remarkable success across many domains, it
has historically underperformed on tabular learning tasks, which remain
dominated by gradient boosting decision trees (GBDTs). However, recent
advancements are paving the way for Tabular Foundation Models, which can
leverage real-world knowledge and generalize across diverse datasets,
particularly when the data contains free-text. Although incorporating language
model capabilities into tabular tasks has been explored, most existing methods
utilize static, target-agnostic textual representations, limiting their
effectiveness. We introduce TabSTAR: a Foundation Tabular Model with
Semantically Target-Aware Representations. TabSTAR is designed to enable
transfer learning on tabular data with textual features, with an architecture
free of dataset-specific parameters. It unfreezes a pretrained text encoder and
takes as input target tokens, which provide the model with the context needed
to learn task-specific embeddings. TabSTAR achieves state-of-the-art
performance for both medium- and large-sized datasets across known benchmarks
of classification tasks with text features, and its pretraining phase exhibits
scaling laws in the number of datasets, offering a pathway for further
performance improvements.
6833f8b419852283c4b3bbd6
Alan Arazi
claimed_verified
2025-05-26T14:07:38.679Z
60d84af7eac5e05d4594f010
https://cdn-avatars.huggingface.co/v1/production/uploads/60d84af7eac5e05d4594f010/KnGxUR7OUOAGg0S67tRaY.png
Alan Arazi
user
alana89
6833f8b419852283c4b3bbd7
Eilam Shapira
claimed_verified
2025-05-26T08:09:08.206Z
64802fb6c57f629056c59966
/avatars/d5ecabaceeba759969855acf512b6649.svg
Eilam Shapira
user
EilamSha
6833f8b419852283c4b3bbd8
Roi Reichart
TabSTAR: A Foundation Tabular Model With Semantically Target-Aware
  Representations
While deep learning has achieved remarkable success across many domains, it
has historically underperformed on tabular learning tasks, which remain
dominated by gradient boosting decision trees (GBDTs). However, recent
advancements are paving the way for Tabular Foundation Models, which can
leverage real-world knowledge and generalize across diverse datasets,
particularly when the data contains free-text. Although incorporating language
model capabilities into tabular tasks has been explored, most existing methods
utilize static, target-agnostic textual representations, limiting their
effectiveness. We introduce TabSTAR: a Foundation Tabular Model with
Semantically Target-Aware Representations. TabSTAR is designed to enable
transfer learning on tabular data with textual features, with an architecture
free of dataset-specific parameters. It unfreezes a pretrained text encoder and
takes as input target tokens, which provide the model with the context needed
to learn task-specific embeddings. TabSTAR achieves state-of-the-art
performance for both medium- and large-sized datasets across known benchmarks
of classification tasks with text features, and its pretraining phase exhibits
scaling laws in the number of datasets, offering a pathway for further
performance improvements.
TabSTAR, a tabular foundation model with semantically target-aware representations, achieves state-of-the-art performance in classification tasks with text features through transfer learning without dataset-specific parameters.
TabSTAR foundation tabular model semantically target-aware representations transfer learning pretrained text encoder target tokens task-specific embeddings scaling laws