\section{Submissions to the NeurIPS 2022 Track on Datasets and Benchmarks}

Please read the instructions below carefully and follow them faithfully.

\subsection{Style}

Papers must be prepared according to the
instructions presented here. Papers may only be up to {\bf nine} pages long,
including figures. Additional pages \emph{containing only acknowledgments and
references} are allowed. Papers that exceed the page limit will not be
reviewed, or in any other way considered for presentation at the conference.

Authors are required to use the NeurIPS \LaTeX{} style files obtainable at the
NeurIPS website as indicated below. Please make sure you use the current files
and not previous versions. Tweaking the style files may be grounds for
rejection.

\subsection{Retrieval of style files}

The style files for the NeurIPS Track on Datasets and Benchmarks and other information are available on the World Wide Web at
\begin{center}
  \url{http://www.neurips.cc/Conferences/2022/CallForDatasetsBenchmarks}
\end{center}
The file \verb+neurips_data_2022.pdf+ contains these instructions and illustrates the
various formatting requirements your submission must satisfy.

The only supported style file for NeurIPS 2022 is \verb+neurips_data_2022.sty+,
written for \LaTeXe{}. There are no supported style sheets for Microsoft Word, RTF, or other formats. The \LaTeX{} style file contains three optional arguments: \verb+final+, which
creates a camera-ready copy, \verb+preprint+, which creates a preprint for
submission to, e.g., arXiv, and \verb+nonatbib+, which will not load the
\verb+natbib+ package for you in case of package clash.

\paragraph{Preprint option}
If you wish to post a preprint of your work online, e.g., on arXiv, using the
NeurIPS style, please use the \verb+preprint+ option. This will create a version of your work with the text ``Preprint. Work in progress.''
in the footer. This version may be distributed as you see fit. Please \textbf{do
  not} use the \verb+final+ option, which should \textbf{only} be used for
papers accepted to the NeurIPS Track on Datasets and Benchmarks.

At submission time, please omit the \verb+final+ and \verb+preprint+
options. This will add line numbers to aid
review. Please do \emph{not} refer to these line numbers in your paper as they
will be removed during generation of camera-ready copies. Note that submissions to the NeurIPS Track on Datasets and Benchmarks are reviewed in a single-blind fashion and therefore not anonymous. This is because datasets can typically not be shared in a non-anonymous way. If you feel strongly that your work should be submitted anonymously, please use the \verb+anonymous+ option. This will create a version of your work with all author names hidden.

The file \verb+neurips_data_021.tex+ may be used as a ``shell'' for writing your
paper. All you have to do is replace the author, title, abstract, and text of
the paper with your own.

The formatting instructions contained in these style files are summarized in
Sections \ref{gen_inst}, \ref{headings}, and \ref{others} below.

\section{General formatting instructions}
\label{gen_inst}

The text must be confined within a rectangle 5.5~inches (33~picas) wide and
9~inches (54~picas) long. The left margin is 1.5~inch (9~picas).  Use 10~point
type with a vertical spacing (leading) of 11~points.  Times New Roman is the
preferred typeface throughout, and will be selected for you by default.
Paragraphs are separated by \nicefrac{1}{2}~line space (5.5 points), with no
indentation.

The paper title should be 17~point, initial caps/lower case, bold, centered
between two horizontal rules. The top rule should be 4~points thick and the
bottom rule should be 1~point thick. Allow \nicefrac{1}{4}~inch space above and
below the title to rules. All pages should start at 1~inch (6~picas) from the
top of the page.

Authors' names are set in boldface, and each name is
centered above the corresponding address. The lead author's name is to be listed
first (left-most), and the co-authors' names (if different address) are set to
follow. If there is only one co-author, list both author and co-author side by
side.

Please pay special attention to the instructions in Section \ref{others}
regarding figures, tables, acknowledgments, and references.

\section{Headings: first level}
\label{headings}

All headings should be lower case (except for first word and proper nouns),
flush left, and bold.

First-level headings should be in 12-point type.

\subsection{Headings: second level}

Second-level headings should be in 10-point type.

\subsubsection{Headings: third level}

Third-level headings should be in 10-point type.

\paragraph{Paragraphs}

There is also a \verb+\paragraph+ command available, which sets the heading in
bold, flush left, and inline with the text, with the heading followed by 1\,em
of space.

\section{Citations, figures, tables, references}
\label{others}

These instructions apply to everyone.

\subsection{Citations within the text}

The \verb+natbib+ package will be loaded for you by default.  Citations may be
author/year or numeric, as long as you maintain internal consistency.  As to the
format of the references themselves, any style is acceptable as long as it is
used consistently.

The documentation for \verb+natbib+ may be found at
\begin{center}
  \url{http://mirrors.ctan.org/macros/latex/contrib/natbib/natnotes.pdf}
\end{center}
Of note is the command \verb+\citet+, which produces citations appropriate for
use in inline text.  For example,
\begin{verbatim}
   \citet{hasselmo} investigated\dots
\end{verbatim}
produces
\begin{quote}
  Hasselmo, et al.\ (1995) investigated\dots
\end{quote}

If you wish to load the \verb+natbib+ package with options, you may add the
following before loading the \verb+neurips_2022+ package:
\begin{verbatim}
   \PassOptionsToPackage{options}{natbib}
\end{verbatim}

If \verb+natbib+ clashes with another package you load, you can add the optional
argument \verb+nonatbib+ when loading the style file:
\begin{verbatim}
   \usepackage[nonatbib]{neurips_2022}
\end{verbatim}

As submission is double blind, refer to your own published work in the third
person. That is, use ``In the previous work of Jones et al.\ [4],'' not ``In our
previous work [4].'' If you cite your other papers that are not widely available
(e.g., a journal paper under review), use anonymous author names in the
citation, e.g., an author of the form ``A.\ Anonymous.''

\subsection{Footnotes}

Footnotes should be used sparingly.  If you do require a footnote, indicate
footnotes with a number\footnote{Sample of the first footnote.} in the
text. Place the footnotes at the bottom of the page on which they appear.
Precede the footnote with a horizontal rule of 2~inches (12~picas).

Note that footnotes are properly typeset \emph{after} punctuation
marks.\footnote{As in this example.}

\subsection{Figures}

\begin{figure}
  \centering
  \fbox{\rule[-.5cm]{0cm}{4cm} \rule[-.5cm]{4cm}{0cm}}
  \caption{Sample figure caption.}
\end{figure}

All artwork must be neat, clean, and legible. Lines should be dark enough for
purposes of reproduction. The figure number and caption always appear after the
figure. Place one line space before the figure caption and one line space after
the figure. The figure caption should be lower case (except for first word and
proper nouns); figures are numbered consecutively.

You may use color figures.  However, it is best for the figure captions and the
paper body to be legible if the paper is printed in either black/white or in
color.

\subsection{Tables}

All tables must be centered, neat, clean and legible.  The table number and
title always appear before the table.  See Table~\ref{sample-table}.

Place one line space before the table title, one line space after the
table title, and one line space after the table. The table title must
be lower case (except for first word and proper nouns); tables are
numbered consecutively.

Note that publication-quality tables \emph{do not contain vertical rules.} We
strongly suggest the use of the \verb+booktabs+ package, which allows for
typesetting high-quality, professional tables:
\begin{center}
  \url{https://www.ctan.org/pkg/booktabs}
\end{center}
This package was used to typeset Table~\ref{sample-table}.

\begin{table}
  \caption{Sample table title}
  \label{sample-table}
  \centering
  \begin{tabular}{lll}
    \toprule
    \multicolumn{2}{c}{Part}                   \\
    \cmidrule(r){1-2}
    Name     & Description     & Size ($\mu$m) \\
    \midrule
    Dendrite & Input terminal  & $\sim$100     \\
    Axon     & Output terminal & $\sim$10      \\
    Soma     & Cell body       & up to $10^6$  \\
    \bottomrule
  \end{tabular}
\end{table}

\section{Final instructions}

Do not change any aspects of the formatting parameters in the style files.  In
particular, do not modify the width or length of the rectangle the text should
fit into, and do not change font sizes (except perhaps in the
\textbf{References} section; see below). Please note that pages should be
numbered.

\section{Preparing PDF files}

Please prepare submission files with paper size ``US Letter,'' and not, for
example, ``A4.''

Fonts were the main cause of problems in the past years. Your PDF file must only
contain Type 1 or Embedded TrueType fonts. Here are a few instructions to
achieve this.

\begin{itemize}

\item You should directly generate PDF files using \verb+pdflatex+.

\item You can check which fonts a PDF files uses.  In Acrobat Reader, select the
  menu Files$>$Document Properties$>$Fonts and select Show All Fonts. You can
  also use the program \verb+pdffonts+ which comes with \verb+xpdf+ and is
  available out-of-the-box on most Linux machines.

\item The IEEE has recommendations for generating PDF files whose fonts are also
  acceptable for NeurIPS. Please see
  \url{http://www.emfield.org/icuwb2010/downloads/IEEE-PDF-SpecV32.pdf}

\item \verb+xfig+ "patterned" shapes are implemented with bitmap fonts.  Use
  "solid" shapes instead.

\item The \verb+\bbold+ package almost always uses bitmap fonts.  You should use
  the equivalent AMS Fonts:
\begin{verbatim}
   \usepackage{amsfonts}
\end{verbatim}
followed by, e.g., \verb+\mathbb{R}+, \verb+\mathbb{N}+, or \verb+\mathbb{C}+
for $\mathbb{R}$, $\mathbb{N}$ or $\mathbb{C}$.  You can also use the following
workaround for reals, natural and complex:
\begin{verbatim}
   \newcommand{\RR}{I\!\!R}
   \newcommand{\Nat}{I\!\!N}
   \newcommand{\CC}{I\!\!\!\!C}
\end{verbatim}
Note that \verb+amsfonts+ is automatically loaded by the \verb+amssymb+ package.

\end{itemize}

If your file contains type 3 fonts or non embedded TrueType fonts, we will ask
you to fix it.

\subsection{Margins in \LaTeX{}}

Most of the margin problems come from figures positioned by hand using
\verb+\special+ or other commands. We suggest using the command
\verb+\includegraphics+ from the \verb+graphicx+ package. Always specify the
figure width as a multiple of the line width as in the example below:
\begin{verbatim}
   \usepackage[pdftex]{graphicx} ...
   \includegraphics[width=0.8\linewidth]{myfile.pdf}
\end{verbatim}
See Section 4.4 in the graphics bundle documentation
(\url{http://mirrors.ctan.org/macros/latex/required/graphics/grfguide.pdf})

A number of width problems arise when \LaTeX{} cannot properly hyphenate a
line. Please give LaTeX hyphenation hints using the \verb+\-+ command when
necessary.

\subsection{Suppressing line numbers in supplementary materials}
If you need to suppress line numbers in the supplementary materials because
they interfere with the text, for instance because you are including a
data sheet in 2-column format, you can do so by
placing the following command before it:
\begin{verbatim}
    \let\linenumbers\nolinenumbers\nolinenumbers
\end{verbatim}

\begin{ack}
Use unnumbered first level headings for the acknowledgments. All acknowledgments
go at the end of the paper before the list of references. Moreover, you are required to declare
funding (financial activities supporting the submitted work) and competing interests (related financial activities outside the submitted work).
More information about this disclosure can be found at: \url{https://neurips.cc/Conferences/2022/PaperInformation/FundingDisclosure}.
You can use the \texttt{ack} environment provided in the style file. As opposed to the main NeurIPS track, acknowledgements do not need to be hidden.
\end{ack}

\section*{References}

References follow the acknowledgments. Use unnumbered first-level heading for
the references. Any choice of citation style is acceptable as long as you are
consistent. It is permissible to reduce the font size to \verb+small+ (9 point)
when listing the references.
Note that the Reference section does not count towards the page limit.
\medskip

{
\small

[1] Alexander, J.A.\ \& Mozer, M.C.\ (1995) Template-based algorithms for
connectionist rule extraction. In G.\ Tesauro, D.S.\ Touretzky and T.K.\ Leen
(eds.), {\it Advances in Neural Information Processing Systems 7},
pp.\ 609--616. Cambridge, MA: MIT Press.

[2] Bower, J.M.\ \& Beeman, D.\ (1995) {\it The Book of GENESIS: Exploring
  Realistic Neural Models with the GEneral NEural SImulation System.}  New York:
TELOS/Springer--Verlag.

[3] Hasselmo, M.E., Schnell, E.\ \& Barkai, E.\ (1995) Dynamics of learning and
recall at excitatory recurrent synapses and cholinergic modulation in rat
hippocampal region CA3. {\it Journal of Neuroscience} {\bf 15}(7):5249-5262.
}

\section*{Checklist}

The checklist follows the references.  Please
read the checklist guidelines carefully for information on how to answer these
questions.  For each question, change the default \answerTODO{} to \answerYes{},
\answerNo{}, or \answerNA{}.  You are strongly encouraged to include a {\bf
justification to your answer}, either by referencing the appropriate section of
your paper or providing a brief inline description.  For example:
\begin{itemize}
  \item Did you include the license to the code and datasets? \answerYes{See Section~\ref{gen_inst}.}
  \item Did you include the license to the code and datasets? \answerNo{The code and the data are proprietary.}
  \item Did you include the license to the code and datasets? \answerNA{}
\end{itemize}
Please do not modify the questions and only use the provided macros for your
answers.  Note that the Checklist section does not count towards the page
limit.  In your paper, please delete this instructions block and only keep the
Checklist section heading above along with the questions/answers below.

\begin{enumerate}

\item For all authors...
\begin{enumerate}
  \item Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
    \answerTODO{}
  \item Did you describe the limitations of your work?
    \answerTODO{}
  \item Did you discuss any potential negative societal impacts of your work?
    \answerTODO{}
  \item Have you read the ethics review guidelines and ensured that your paper conforms to them?
    \answerTODO{}
\end{enumerate}

\item If you are including theoretical results...
\begin{enumerate}
  \item Did you state the full set of assumptions of all theoretical results?
    \answerTODO{}
	\item Did you include complete proofs of all theoretical results?
    \answerTODO{}
\end{enumerate}

\item If you ran experiments (e.g. for benchmarks)...
\begin{enumerate}
  \item Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
    \answerTODO{}
  \item Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
    \answerTODO{}
	\item Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
    \answerTODO{}
	\item Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
    \answerTODO{}
\end{enumerate}

\item If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
\begin{enumerate}
  \item If your work uses existing assets, did you cite the creators?
    \answerTODO{}
  \item Did you mention the license of the assets?
    \answerTODO{}
  \item Did you include any new assets either in the supplemental material or as a URL?
    \answerTODO{}
  \item Did you discuss whether and how consent was obtained from people whose data you're using/curating?
    \answerTODO{}
  \item Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
    \answerTODO{}
\end{enumerate}

\item If you used crowdsourcing or conducted research with human subjects...
\begin{enumerate}
  \item Did you include the full text of instructions given to participants and screenshots, if applicable?
    \answerTODO{}
  \item Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
    \answerTODO{}
  \item Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
    \answerTODO{}
\end{enumerate}

\end{enumerate}

\section{Introduction}

Deep learning has enabled tremendous progress for learning on image, language, or even audio datasets. On tabular data, however, the picture is muddier and ensemble models based on decision trees like XGBoost remain the go-to tool for most practitioners \citep{StateDataScience} and data science competitions \citep{kossenSelfAttentionDatapointsGoing2021}. Indeed deep learning architectures have been crafted to create inductive biases matching invariances and spatial dependencies of the data. Finding corresponding invariances is hard in tabular data, made of heterogeneous features, small sample sizes, extreme values.

Creating tabular-specific deep learning architectures is a very active area of research (see \autoref{sec:related}) given that
tree-based models are not differentiable, and thus cannot be easily composed and jointly trained with other deep learning blocks. Most corresponding publications claim to beat or match tree-based models, but their claims have been put into question: a simple Resnet seems to be competitive with some of these new models \citep{gorishniyRevisitingDeepLearning2021}, and most of these methods seem to fail on new datasets \citep{shwartz-zivTabularDataDeep2021}.
Indeed, the lack of an established benchmark for tabular data learning
provides additional degrees of freedom to researchers when evaluating their method.
Furthermore, most tabular datasets available online are small compared to benchmarks in other machine learning subdomains, such as ImageNet~\citep{ImageNetLargescaleHierarchical}, making evaluation noisier. These issues add up to other sources of unreplicability across machine learning, such as unequal hyperparameters tuning efforts \citep{liptonTroublingTrendsMachine2019} or failure to account for statistical uncertainty in benchmarks \citep{bouthillierAccountingVarianceMachine2021a}. To alleviate these concerns, we contribute a tabular data benchmark with a precise methodology for datasets inclusion and hyperparameter tuning. This enables us to evaluate recent deep learning models which have not yet been independently evaluated, and to show that tree-based models remain state-of-the-art on medium-sized tabular datasets, even without accounting for the slower training of deep learning algorithms.

Impressed by the superiority of tree-based models on tabular data, we strive to understand which \textit{inductive biases} make them well-suited for these data. By transforming tabular datasets to modify the performances of different models, we uncover differing biases of tree-based models and deep learning algorithms which partly explain their different performances: neural networks struggle to learn irregular patterns of the target function, and their rotation invariance hurt their performance, in particular when handling the numerous uninformative features present in tabular data.

Our contributions are as follow:  \textbf{1.} We create a new benchmark for tabular data, with a precise methodology for choosing and preprocessing a large number of representative datasets. We share these datasets through OpenML \citep{vanschorenOpenMLNetworkedScience2014}, which makes them easy to use. \textbf{2.} We extensively compare deep learning models and tree-based models on generic tabular datasets in multiple settings, accounting for the cost of choosing hyperparameters. We also share the raw results of our random searches, which will enable researchers to cheaply test new algorithms for a fixed hyperparameter optimization budget.
\textbf{3.} We investigate empirically why tree-based models outperform deep learning, by finding data transformations which narrow or widen their performance gap.
This highlights desirable biases for tabular data learning, which we hope will help other researchers to successfully build deep learning models for tabular data.

In Sec.\,\ref{sec:related} we cover related work.
Sec. \ref{bench} gives a short description of our benchmark methodology, including datasets, data processing, and  hyper-parameter tuning. Then, Sec. \ref{numeric} shows our raw results on deep learning and tree-based models after an extensive random search. Finally, Sec. \ref{ablation} provides the results of an empirical study which exhibit desirable implicit biases of tabular datasets.

\section{Related work}

\label{sec:related}%

\paragraph{Deep learning for tabular data} As described by \cite{borisovDeepNeuralNetworks2021} in their review of the field, there have been various attempts to make deep learning work on tabular data: data encoding techniques to make tabular data better suited for deep learning \citep{hancockSurveyCategoricalData2020, yoonVIMEExtendingSuccess2020}, "hybrid methods" to benefit from the flexibility of NNs while keeping the inductive biases of other algorithms like tree-based models \citep{layRandomHingeForest2018, popovNeuralObliviousDecision2020,abutbulDNFNetNeuralArchitecture2020,hehnEndtoEndLearningDecision2019, tannoAdaptiveNeuralTrees2019,chenAttentionAugmentedDifferentiable2020, kontschiederDeepNeuralDecision2015, rodriguezInterpretableReinforcementLearning2019, popovNeuralObliviousDecision2020, layRandomHingeForest2018} or Factorization Machines \cite{guoDeepFMFactorizationMachineBased2017}, tabular-specific transformers architectures \cite{somepalliSAINTImprovedNeural2021,kossenSelfAttentionDatapointsGoing2021,arikTabNetAttentiveInterpretable2019,huangTabTransformerTabularData2020}, and various regularization techniques to adapt classical architectures to tabular data \citep{louniciMuddlingLabelRegularization2021, shavittRegularizationLearningNetworks2018,kadraWelltunedSimpleNets2021, fiedlerSimpleModificationsImprove2021}. In this paper, we focus on architectures directly inspired by classic deep learning models, in particular Transformers and Multi-Layer-Perceptrons (MLPs)

\paragraph{Comparisons between NNs and tree-based models} The most comprehensive comparisons of machine learning algorithms have been published before the advent of new deep learning methods, or on specific problems \citep{fernandez-delgadoWeNeedHundreds2014, sakrComparisonMachineLearning2017, korotcovComparisonDeepLearning2017,uddinComparingDifferentSupervised2019}. Recently, \cite{shwartz-zivTabularDataDeep2021} evaluated modern tabular-specific deep learning methods, but their goal was more to reveal that "New deep learning architectures fail to generalize to new datasets" than to create a comprehensive benchmark. \cite{borisovDeepNeuralNetworks2022} benchmarked recent algorithms in their review of deep learning for tabular data, but only on 3 datasets, and "highlight[ed]
the need for unified benchmarks" for tabular data. Most papers introducing a new architecture for tabular data benchmark various algorithms, but with a highly variable evaluation methodology, a small number of datasets, and the evaluation can be biased toward the authors' model \cite{shwartz-zivTabularDataDeep2021}. The paper closest to our work is \cite{gorishniyRevisitingDeepLearning2021}, benchmarking novel algorithms, on 11 tabular datasets. We provide a more comprehensive benchmark, with 45 datasets, split across different settings (medium-sized / large-size, with/without categorical features), accounting for the hyperparameter tuning cost, to establish a standard benchmark.

\paragraph{No standard benchmark for tabular data} Unlike other machine learning subfields such as computer vision \citep{ImageNetLargescaleHierarchical} or NLP  \citep{wangSuperGLUEStickierBenchmark2020}, there are no standard benchmarks for tabular data. There exist generic machine learning benchmarks, but, to the our knowledge, none are specific to tabular data. For instance, OpenML benchmarks CC-18, CC-100, \citep{bischlOpenMLBenchmarkingSuites2021} and AutoML Benchmark \cite{gijsbersOpenSourceAutoML2019} contain tabular data, but also include images and artificial datasets, which may explain why they have not been used in tabular deep learning papers.

\paragraph{Understanding the difference between NNs and tree-based models} To our knowledge, this is the first empirical investigation of \textsl{why} tree-based models outperform NNs on tabular data. Some speculative explanations, however, have been offered \citep{klambauerSelfNormalizingNeuralNetworks2017,borisovDeepNeuralNetworks2021}. \cite{kadraWelltunedSimpleNets2021} claims that searching across  13 regularization techniques for MLPs to find a dataset-specific combination gives state-of-the-art performances. This provides a partial explanation: MLPs are expressive enough for tabular data but may suffer from a lack of proper regularization.

\section{A benchmark for tabular learning}\label{bench}
\subsection{45 reference tabular datasets}%

For our benchmark, we compiled 45 tabular datasets from various domains provided mainly by OpenML, listed in \ref{supp:datasets} and selected via the following criteria:
\begin{description}[topsep=0pt,itemsep=0ex,partopsep=0ex,parsep=0ex,leftmargin=2ex]
    \item[Heterogeneous columns.] Columns should correspond to features of different nature. This excludes images or signal datasets where each column corresponds to the same signal on different sensors.
    \item[Not high dimensional.] We only keep datasets with a $d/n$ ratio below 1/10.
    \item[Undocumented datasets] We remove datasets where too little information is available. We did keep datasets with hidden column names if it was clear that the features were heterogeneous.
    \item[I.I.D. data.] We remove stream-like datasets or time series.
    \item[Real-world data.] We remove artificial datasets but keep some simulated datasets. The difference is subtle, but we try to keep simulated datasets if learning these datasets are of practical importance (like the Higgs dataset), and not just a toy example to test specific model capabilities.
    \item[Not too small.] We remove datasets with too few features (< 4) and too few samples (< 3\,000). For benchmarks on numerical features only, we remove categorical features before checking if enough features and samples are remaining.
    \item[Not too easy.] We remove datasets which are too easy. Specifically, we remove a dataset if a default Logistic Regression (or Linear Regression for regression) reach a score whose relative difference with the score of both a default Resnet (from \cite{gorishniyRevisitingDeepLearning2021}) and a default HistGradientBoosting model (from scikit learn) is below 5\%. Other benchmarks use different metrics to remove too easy datasets, like removing datasets which can be learnt perfectly by a single decision classifier \citep{bischlOpenMLBenchmarkingSuites2021}, but this does not account for different Bayes rate of different datasets. As tree-based methods have been shown to be superior to Logistic Regression \citep{fernandez-delgadoWeNeedHundreds2014} in our setting, a close score for these two types of models indicates that we might already be close to the best achievable score.
    \item[Not deterministic.] We remove datasets where the target is a deterministic function of the data. This mostly means removing datasets on games like poker and chess. Indeed, we believe that these datasets are very different from most real-world tabular datasets, and should be studied separately.
\end{description}

\subsection{Removing side issues}

To keep learning tasks as
homogeneous as possible and focus on challenges specific to tabular data, we exclude subproblems which would deserve their own analysis:

\begin{description}[topsep=0pt,itemsep=0ex,partopsep=0ex,parsep=0ex,leftmargin=2ex]
\item[Medium-sized training set] We truncate the training set to 10,000 samples for bigger datasets. This allows us to investigate the medium-sized dataset regime. We study the large-sized (50,000) regime, for which fewer datasets matching our criteria are available, in \ref{supp:benchmarks}.

\item[No missing data] We remove all missing data from the datasets. Indeed, there are numerous techniques for handling
missing data both for tree-based models and NNs, with varying
performances
\citep{perez-lebelBenchmarkingMissingvaluesApproaches2022}. In practice, we first remove columns containing many missing data, then all rows containing at least one missing entry.
\item[Balanced classes] For classification, the target is binarised if there are several classes, by taking the two most numerous classes, and we keep half of samples in each class.
\item[Low cardinality categorical features] We remove categorical features with more than 20 items.
\item[High cardinality numerical features] We remove numerical features with less than 10 unique values. Numerical features with 2 unique values are converted to categorical features.
\end{description}

\subsection{A procedure to benchmark models with hyperparameter selection}\label{hyperparameters}

Hyperparameter tuning leads to uncontrolled variance on a benchmark \citep{bouthillierAccountingVarianceMachine2021a}, especially with a small budget of model evaluations. We design a benchmarking procedure that jointly samples the variance of hyperparameter tuning and explores increasingly high budgets of model evaluations. It relies on random searches for hyper-parameter tuning \citep{hyperopt}.
We use hyperparameter search spaces from the Hyperopt-Sklearn \cite{komerHyperoptSklearnAutomaticHyperparameter2014} when available, from the original paper when possible, and from \cite{gorishniyRevisitingDeepLearning2021} for MLP, Resnet and XGBoost (see \ref{supp:benchmark_details}).
We run a random search of $\approx 400$ iterations per dataset, on CPU for tree-based models and GPU for NNs (more details in \ref{supp:benchmark_details}).

To study performance as a function of the number $n$ of random search iterations, we compute the best hyperparameter combination on the validation set on these $n$ iterations (for each model and dataset), and evaluate it on the test set. We do this 15 times while shuffling the random search order at each time. This gives us bootstrap-like estimates of the expected test score of the best (on the validation set) model after each number of random search iterations. In addition, we always start the random searches with the default hyperparameters of each model.

\paragraph{Resuable code and benchmark raw data}
The code used for all the experiments and comparisons is available at \href{https://github.com/LeoGrin/tabular-benchmark}{https://github.com/LeoGrin/tabular-benchmark}. To help researchers to cheaply add their own algorithms  to the results, we also share at the same link a data table containing results for all iterations of our 20,000 compute-hour random searches.

\subsection{Aggregating results across datasets}

We use the test set accuracy (classification) and R2 score (regression) to measure model performance. To aggregate results across datasets of varying difficulty, we use a metric similar to the distance to the minimum (or average distance to the minimum (ADTM) when averaged across datasets), used in \cite{feurerAutoSklearnHandsfreeAutoML2021} and introduced in \cite{wistubaLearningHyperparameterOptimization2015}. This metric consists in normalizing each test accuracy between 0 and 1 via an affine renormalization between the top-performing  and worse-performing models. Instead of the worse-performing model, we use models achieving the 10\% (classification) or 50\% (regression) test error quantile. Indeed, the worse scores are achieved by outlier models and are not representative of the difficulty of the dataset. For regression tasks, we clip all negative scores (i.e below 50\% scores) to 0 to reduce the influence of very low scores.

\subsection{Data preparation}

We strive for as little manual preprocessing as possible, applying only the following transformations:
\begin{description}[topsep=0pt,itemsep=0ex,partopsep=0ex,parsep=0ex,leftmargin=2ex]
\item[Gaussianized features] For NN training,  the features are
Gaussianized with Scikit-learn's \texttt{QuantileTransformer}.

\item[Transformed regression targets] In regression settings, target variables are log-transformed when their distributions are heavy-tailed (e.g house prices, see \ref{supp:datasets}). In addition, we add as an hyperparameter the possibility to Gaussienize the target variable for model fit, and transform it back for evaluation (via ScikitLearn's  TransformedTargetRegressor and QuantileTransformer).

\item[OneHotEncoder] For models which do not handle categorical variables natively, we encode categorical features using ScikitLearn's OneHotEncoder.
\end{description}

\section{Tree-based models still outperform deep learning on tabular data.}\label{numeric}

\subsection{Models benchmarked}
For tree-based models, we choose 3 state-of-the-art models used by practitioners: Scikit Learn’s RandomForest, GradientBoostingTrees (GBTs) (or HistGradientBoostingTrees when using categorical features), and XGBoost \cite{chenXGBoostScalableTree2016a}. We benchmark the following deep models:
\begin{description}[topsep=0pt,leftmargin=3ex,itemsep=0ex]
    \item[MLP]: a classical MLP from
\cite{gorishniyRevisitingDeepLearning2021}. The only improvement beyond a
simple MLP is using Pytorch's {\tt ReduceOnPlateau}
learning rate scheduler.
    \item[Resnet]: as in
\cite{gorishniyRevisitingDeepLearning2021}, similar to \textbf{MLP} with dropout, batch/layer normalization, and skip connections.
    \item[FT\_Transformer]: a simple Transformer model combined with a module embedding categorical and numerical features, created in \cite{gorishniyRevisitingDeepLearning2021}. We choose this model because it was benchmarked in a convincing way against tree-based models and other tabular-specific models. It can thus be considered a “best case” for Deep learning models on tabular data.
    \item[SAINT]: a Transformer model with an embedding module and an inter-samples attention mechanism, proposed in
\cite{somepalliSAINTImprovedNeural2021}. We include this model because it was the best performing deep model in \cite{borisovDeepNeuralNetworks2021}, and to investigate the impact of inter-sample attention, which performs well on tabular data according to \cite{kossenSelfAttentionDatapointsGoing2022a}.
\end{description}

\subsection{Results}\label{results}

Fig. \ref{fig:benchmark_numeric} and \ref{fig:benchmark_categorical} give benchmark results for different types of datasets (appendix \ref{supp:benchmarks} gives results as a function of computation \textit{time}).

\begin{figure}[tb!]
\begin{minipage}{.5\textwidth}
  \includegraphics[width=\linewidth]{figures/random_search_classif_numerical.jpg}%
  \llap{\raisebox{.8\linewidth}{\parbox{.8\linewidth}{\sffamily{\bfseries Classification} (15 datasets)}}}%
\end{minipage}%
\begin{minipage}{.5\textwidth}
  \includegraphics[width=\linewidth]{figures/random_search_regression_numerical.jpg}%
    \llap{\raisebox{.8\linewidth}{\parbox{.8\linewidth}{\sffamily{\bfseries Regression} (19 datasets)}}}%
\end{minipage}
\caption{\textbf{Benchmark on medium-sized datasets, with only numerical features}. Dotted lines correspond to the score of the default hyperparameters, which is also the first random search iteration. Each value corresponds to the test score of the best model (on the validation set) after a specific number of random search iterations, averaged on 15 shuffles of the random search order. The ribbon corresponds to the minimum and maximum scores on these 15 shuffles.%
\label{fig:benchmark_numeric}}

\bigskip

\begin{minipage}{.5\textwidth}
  \includegraphics[width=\linewidth]{figures/benchmark_categorical_classif.jpg}%
    \llap{\raisebox{.82\linewidth}{\parbox{.8\linewidth}{\sffamily{\bfseries Classification} (7 datasets)}}}%
\end{minipage}%
\begin{minipage}{.5\textwidth}
  \includegraphics[width=\linewidth]{figures/random_search_regression_categorical.jpg}%
      \llap{\raisebox{.82\linewidth}{\parbox{.8\linewidth}{\sffamily{\bfseries Regression} (14 datasets)}}}%
\end{minipage}
\caption{\textbf{Benchmark on medium-sized datasets, with both numerical and categorical features}. Dotted lines correspond to the score of the default hyperparameters, which is also the first random search iteration. Each value corresponds to the test score of the best model (on the validation set) after a specific number of random search iterations, averaged on 15 shuffles of the random search order. The ribbon corresponds to the minimum and maximum scores on these 15 shuffles.}
\label{fig:benchmark_categorical}
\end{figure}

\paragraph{Tuning hyperparameters does not make NNs state-of-the-art}
Tree-based models are superior for every random search budget, and the performance gap stays wide even after a large number of random search iterations. This does not take into account that each random search iteration is generally slower for NNs than for tree-based models (see \ref{supp:benchmarks}).

\paragraph{Categorical variables are not the main weakness of NNs}
Categorical variables are often seen as a major problem for using NNs on tabular data \citep{borisovDeepNeuralNetworks2021}. Our results on numerical variables only do reveal a narrower gap between tree-based models and NNs than including categorical variables. Still, most of this gap subsists when learning on numerical features only.

\section{Empirical investigation: \textit{why} do tree-based models still outperform deep
learning on tabular data?}\label{ablation}

\subsection{Methodology: uncovering inductive biases}

We have seen in Sec. \ref{results} that tree-based models beat NNs across a wide range of hyperparameter choices. This hints to inherent properties of these models which explains their performances on tabular data. Indeed, the best methods on tabular data share two attributes: they are \textbf{ensemble methods}, bagging (Random Forest) or boosting (XGBoost, GBTs), and the weak learner used in these ensembles is a \textbf{decision tree}. The decisive point seems to be the tree aspect:
other boosting and bagging methods with different weak learners exist but are not commonly used for tabular data. In this section, we try to understand the \textit{inductive biases} of decision trees that make them well-suited for tabular data, and how they differ from the inductive biases of NNs. This is equivalent to saying the reverse: which features of tabular data make this type of data easy to learn with tree-based methods yet more difficult with a NN?

To this aim, we apply various transformations to tabular datasets which either narrow or widen the generalization performance gap between NNs and tree-based models, and thus help us emphasize their different inductive biases.
For the sake of simplicity, we restrict our analysis to numerical variables and classification tasks on medium-sized datasets. Results are presented aggregated across datasets, and dataset-specific results are available in \ref{supp:xp_details}, along with additional details on our experiments

\subsection{Finding 1: NNs are biased to overly smooth solutions}\label{finding_1}

\begin{figure}[t!]
    \begin{minipage}{.55\linewidth}
    \includegraphics[width=\textwidth]{figures/high_frequencies.jpg}
    \end{minipage}%
        \hfill%
    \begin{minipage}{.42\linewidth}
    \caption{\textbf{Normalized test accuracy of different models for varying smoothing of the target function on the train set}. We smooth the target function through a Gaussian Kernel smoother, whose covariance matrix is the data covariance, multiplied by the (squared) lengthscale of the Gaussian kernel smoother. A lengthscale of 0 corresponds to no smoothing (the original data). All features have been Gaussienized before the smoothing through ScikitLearn's QuantileTransformer. The boxplots represent the distribution of normalized accuracies across 15 re-orderings of the random search.}
    \label{fig:high_frequencies}
    \end{minipage}%
\end{figure}

We transform each \textsl{train} set by smoothing the output with a Gaussian Kernel smoother for varying length-scale values of the kernel (more details are available in \ref{supp:xp_details}). This effectively prevents models from learning irregular patterns of the target function. Fig. \ref{fig:high_frequencies} shows model performance as a function of the length-scale of the smoothing kernel.
For small lengthscales, smoothing the target function on the train set decreases markedly the accuracy of tree-based models, but barely impacts that of
NNs.

Such results suggest that the target functions in our datasets are not smooth, and that NNs struggle to fit these irregular functions compared to tree-based models. This is in line with \cite{rahamanSpectralBiasNeural2019}, which finds that NNs are biased toward low-frequency functions.
Models based on decision trees, which learn piece-wise constant functions, do not exhibit such a bias.
Our findings do not contradict  papers claiming benefits  from regularization for tabular data \citep{shavittRegularizationLearningNetworks2018,borisovDeepNeuralNetworks2021,kadraRegularizationAllYou2021,louniciMuddlingLabelRegularization2021}, as adequate regularization and careful optimization may allow NNs to learn irregular patterns.
 In \ref{supp:xp_details}, we show some examples of non-smooth patterns which neural networks fail to learn, both in toy and real-world settings.

Note also that our observation could also explain the benefits of the ExU activation used in the Neural-GAM paper \citep{agarwalNeuralAdditiveModels2021}, and of the embeddings used in \cite{gorishniyEmbeddingsNumericalFeatures2022}: the periodic embedding might help the model to learn the high-frequency part of the target function, and the target-aware binning might make the target function smoother

\subsection{Finding 2: Uninformative features affect more MLP-like NNs}

\paragraph{Tabular datasets contain many uninformative features}

For each dataset, we drop an increasingly large fraction of features, according to feature importance (ranked by a Random Forest). Fig. \ref{fig:removed_features} shows that the classification accuracy of a GBT is not much affected by removing up to half of the features.

Furthermore, the test accuracy of a GBT trained on the removed features (i.e the features below a certain feature importance threshold) is very low up to 20\% of features removed, and quite low until 50\%, which suggests that most of these features are uninformative, and not solely redundant.

\begin{figure}[tb!]
    \begin{minipage}{.55\linewidth}
    \includegraphics[width=\textwidth]{figures/removed_features_2.jpg}
    \end{minipage}%
    \hfill%
    \begin{minipage}{.43\linewidth}
    \caption{\textbf{Test accuracy of a GBT for varying proportions of removed features}, on our classification benchmark on numerical features. Features are removed in increasing order of feature importance (computed with a Random Forest), and the two lines correspond to the accuracy using the (most important) kept features (blue) or the (least important) removed features (red). A score of 1.0 corresponds to the best score across all models and hyperparameters on each dataset, and 0.0 correspond to random chance. These scores are averaged across 30 random search orders, and the ribbons correspond to the 80\% interval among the different datasets.}
    \label{fig:removed_features}
    \end{minipage}%

\bigskip
\begin{minipage}{.5\textwidth}
    \includegraphics[width=\textwidth]{figures/useless_features.jpg}%
        \llap{\raisebox{.25\linewidth}{\parbox{.7\linewidth}{\sffamily\small{\textbf a.} Removing features}}}%
    \label{fig:remove_useless_features}
\end{minipage}%
\begin{minipage}{.5\textwidth}
  \includegraphics[width=\linewidth]{figures/add_features.jpg}%
        \llap{\raisebox{.25\linewidth}{\parbox{.7\linewidth}{\sffamily\small{\textbf b.} Adding features}}}%
  \label{fig:add_useless_features}
\end{minipage}
\caption{\textbf{Test accuracy changes when removing (a) or adding (b) uninformative features}.  Features
are removed in increasing order of feature
importance (computed with a Random Forest). Added features are sampled from standard Gaussians uncorrelated with the target and with other features. Scores
are averaged across datasets, and the ribbons correspond to the minimum and maximum score among the 30 different random search reorders (starting with the default models).%
\label{fig:features}}%
\end{figure}

\paragraph{MLP-like architectures are not robust to uninformative features} In the two experiments shown in Fig. \ref{fig:features}, we can see that \textit{removing} uninformative features (\ref{fig:features}a) reduces the performance gap between MLPs (Resnet) and the other models (FT Transformers and tree-based models), while \textit{adding} uninformative features widens the gap. This shows that MLPs are less robust to uninformative features, and, given the frequency of such features in tabular datasets, partly explain the results from Sec. \ref{results}.

In Fig. \ref{fig:features}a, we also remove informative features as we remove a larger fraction of features. Our reasoning, which is backed by \ref{fig:features}b, is that the decrease in accuracy due to the removal of these features is compensated by the removal of uninformative features, which is more helpful for MLPs than for other models (we also remove redundant features at the same time, which should not impact our models)

\subsection{Finding 3: Data are non invariant by rotation, so should be learning procedures}

Why are MLPs much more hindered by uninformative features, compared to other models? One answer is that this learner is rotationally invariant in the sense of \cite{ngFeatureSelectionVs2004}: the learning procedure which learns an MLP on a training set and evaluate it on a testing set is unchanged when applying a rotation (unitary matrix) to the features on both the training and testing set. Indeed, \cite{ngFeatureSelectionVs2004} shows that any rotationallly invariant learning procedure has a worst-case sample complexity that grows at least linearly in the number of irrelevant features. Intuitively, to remove uninformative features, a rotationaly invariant algorithm has to first find the original orientation of the features, and then select the least informative ones: the information contained in the orientation of the data is lost.

Fig. \ref{fig:rotation}a, which shows the change in test accuracy when randomly rotating our datasets, confirms that only Resnets are rotationally invariant. More striking, random rotations reverse the performance order: NNs are now above tree-based models and Resnets above FT Transformers. This suggests that rotation invariance is not desirable: similarly to vision~\citep{krizhevsky2012imagenet}, there is a natural basis (here, the original basis) which encodes best data-biases, and which can not be recovered by models invariant to rotations which potentially mixes features with very different statistical properties. Indeed, features of a tabular data typically carry meanings individually, as expressed by column names: {\tt age}, {\tt weight}. The link with uninformative features is apparent in \ref{fig:rotation}b: removing the least important half of the features in each dataset (before rotating), drops the performance of all models except Resnets, but the decrease is less significant than when using all features.

Our findings shed light on the results of \cite{somepalliSAINTImprovedNeural2021} and \cite{gorishniyEmbeddingsNumericalFeatures2022}, which add an embedding layer, even for numerical features, before MLP or Transformer models. Indeed, this layer breaks rotation invariance. The fact that very different types of embeddings seem to improve performance suggests that the sheer presence of an embedding which breaks the invariance is a key part of these improvements. We note that a promising avenue for further research would be to find other ways to break rotation invariance which might be less computationally costly than embeddings.

\begin{figure}
\begin{minipage}{.5\textwidth}

    \includegraphics[width=\linewidth]{figures/random_rotation.jpg}%
        \llap{\raisebox{.82\linewidth}{\parbox{\linewidth}{\sffamily\small{\textbf a.} With all features}}}%

    \label{fig:rotation_all_features}
\end{minipage}%
\begin{minipage}{.5\textwidth}

  \includegraphics[width=\linewidth]{figures/random_rotation_features_removed.jpg}%
        \llap{\raisebox{.82\linewidth}{\parbox{\linewidth}{\sffamily\small{\textbf b.} With 50\% features removed}}}%
  \label{fig:rotation_features_removed}
\end{minipage}
\caption{\textbf{Normalized test accuracy of different models when randomly rotating our datasets}. Here, the classification benchmark on numerical features was used. All features are Gaussianized before the random rotations. The scores are averaged across datasets, and the boxes depict the distribution across random search shuffles. Right: the features are removed before data rotation.}%
\label{fig:rotation}%
\end{figure}

\section{Discussion and conclusion}\label{limitation}

\paragraph{Limitation}
Our study leaves open questions for future work: which other inductive biases of tree-based models explain their performances on tabular data? How would our evaluation change on very small datasets?  On very large datasets? What is the best way to handle specific challenges like missing data or high-cardinality categorical features, for NNs and tree-based models? With these best methods, how would the evaluation change including missing data?

\paragraph{Conclusion} While each publication on learning architectures for tabular data comes to different results using a different benchmarking methodology, our systematic benchmark, going beyond the specificities of a handful of datasets and accounting for hyper-parameter choice, reveals clear trends. On such data, tree-based models more easily yield good predictions, with much less computational cost. This superiority is explained by specific features of tabular data: irregular patterns in the target function, uninformative features, and non rotationally-invariant data where linear combinations of features misrepresent the information. Beyond these conclusions, our benchmark is reusable, allowing researchers to use our methodology and datasets for new architectures, and to easily compare them to those we explored via the shared benchmark raw results.
We hope that this benchmark will stimulate tabular deep-learning research and foster more thorough empirical evaluation of contributions.

\begin{ack}
GV and LG acknowledge support in part by the French Agence Nationale de la Recherche under Grant ANR-20-CHIA-0026 (LearnI). EO was supported by the Project ANR-21-CE23-0030 ADONIS and EMERG-ADONIS from Alliance SU.
\end{ack}

\bibliographystyle{unsrtnat}

\section{Appendix}

Include extra information in the appendix. This section will often be part of the supplemental material. Please see the call on the NeurIPS website for links to additional guides on dataset publication.

\begin{enumerate}

\item Submission introducing new datasets must include the following in the supplementary materials:
\begin{enumerate}
  \item Dataset documentation and intended uses. Recommended documentation frameworks include datasheets for datasets, dataset nutrition labels, data statements for NLP, and accountability frameworks.
  \item URL to website/platform where the dataset/benchmark can be viewed and downloaded by the reviewers.
  \item Author statement that they bear all responsibility in case of violation of rights, etc., and confirmation of the data license.
  \item Hosting, licensing, and maintenance plan. The choice of hosting platform is yours, as long as you ensure access to the data (possibly through a curated interface) and will provide the necessary maintenance.
\end{enumerate}

\item To ensure accessibility, the supplementary materials for datasets must include the following:
\begin{enumerate}
  \item Links to access the dataset and its metadata. This can be hidden upon submission if the dataset is not yet publicly available but must be added in the camera-ready version. In select cases, e.g when the data can only be released at a later date, this can be added afterward. Simulation environments should link to (open source) code repositories.
  \item The dataset itself should ideally use an open and widely used data format. Provide a detailed explanation on how the dataset can be read. For simulation environments, use existing frameworks or explain how they can be used.
  \item Long-term preservation: It must be clear that the dataset will be available for a long time, either by uploading to a data repository or by explaining how the authors themselves will ensure this.
  \item Explicit license: Authors must choose a license, ideally a CC license for datasets, or an open source license for code (e.g. RL environments).
  \item Add structured metadata to a dataset's meta-data page using Web standards (like schema.org and DCAT): This allows it to be discovered and organized by anyone. If you use an existing data repository, this is often done automatically.
  \item Highly recommended: a persistent dereferenceable identifier (e.g. a DOI minted by a data repository or a prefix on identifiers.org) for datasets, or a code repository (e.g. GitHub, GitLab,...) for code. If this is not possible or useful, please explain why.
\end{enumerate}

\item For benchmarks, the supplementary materials must ensure that all results are easily reproducible. Where possible, use a reproducibility framework such as the ML reproducibility checklist, or otherwise guarantee that all results can be easily reproduced, i.e. all necessary datasets, code, and evaluation procedures must be accessible and documented.

\item For papers introducing best practices in creating or curating datasets and benchmarks, the above supplementary materials are not required.
\end{enumerate}

\subsection{Remark on the hyper parameter tuning }
 We claimed that tree-based models' are superior for every random search budget, and the performance gap stays wide even after a large number of random search iteration. However, this might not longer be true when adding additional regularization technique to our random search, such as data augmentation. Indeed, \cite{kadraWelltunedSimpleNets2021} find that searching through a "cocktail" of regularization on a Multi-Layer-Perceptron is competitive with XGBoost after half an hour of tuning (for both models), though the datasets considered in their paper are quite different, in particular with the presence of "deterministic" game-inspired datasets in \cite{kadraRegularizationAllYou2021}, on which their method performs very well.
\subsection{Datasets used}\label{supp:datasets}
Make tables

\subsection{More benchmarks}\label{supp:benchmarks}

\paragraph{Large-sized datasets}

\begin{figure}
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \includegraphics[width=\linewidth]{figures/random_search_classif_numerical_large.jpg}
  \caption{Classification. WARNING: only on 4 datasets, 3 of which already in the FT Transformer paper.}
  \label{fig:benchmark_categorical_classif_large}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering
  \caption{Regression}
  \label{fig:benchmark_categorical_regression_large}
\end{subfigure}
\caption{\textbf{Benchmark on large-sized datasets, with only numerical features}. Dotted lines correspond to the score of the default hyperparameters, which is also the first random search iteration. Each value corresponds to the test score of the best model (on the validation set) after a specific number of random search iterations, averaged on 15 shuffles of the random search order. The ribbon corresponds to the minimum and maximum scores on these 15 shuffles.}
\label{fig:benchmark_categorical_large}
\end{figure}

\begin{figure}
\centering
\begin{subfigure}{.5\textwidth}
  \centering
  \caption{Classification}
  \label{fig:benchmark_categorical_classif_large_2}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
  \centering

  \caption{Regression}
  \label{fig:benchmark_categorical_regression_large_2}
\end{subfigure}
\caption{\textbf{Benchmark on medium-sized datasets, with both numerical and categorical features}. Dotted lines correspond to the score of the default hyperparameters, which is also the first random search iteration. Each value corresponds to the test score of the best model (on the validation set) after a specific number of random search iterations, averaged on 15 shuffles of the random search order. The ribbon corresponds to the minimum and maximum scores on these 15 shuffles.}
\label{fig:benchmark_categorical_large_2}
\end{figure}

\subsection{More details on benchmark}\label{supp:benchmark_details}

\paragraph{Number of folds}

For each dataset and hyperparameters combination, we vary the number of folds used for our algorithms evaluation depending on the number of testing samples:

\begin{itemize}
 \item If We have more than 6000 samples, we evaluate our algorithms on 1 fold.
 \item If we have between 3000 and 6000 samples, we evaluate our algorithms on 2 folds.
 \item If we have between 1000 and 3000 samples, we evaluate our algorithms on 3 folds.
 \item If we have less than 1000 testing samples, we evaluate our algorithms on 5 folds.
\end{itemize}

Every algorithms and hyperparameters combinations are evaluated on the same folds.

\paragraph{Hardware}
GPU models
measure time difference?

\subsection{More details on experiments}\label{supp:xp_details}
In this section, we give more details on the choices we had to make when creating our experiments. As aggregated results can sometimes be hard to interpret, we also show the results of each experiment on each dataset.

\paragraph{High-frequency}

Details:

\begin{itemize}
    \item We restrict all datasets to their 5 most important features (according to a RandomForest feature importance ranking). This makes the smoothing easier, as kernel smoothing can be hard in high-dimension, while keeping enough features to produce interesting results.
    \item We estimate the covariance matrix of these features though ScikitLearn's MinCovDet, which is more robust to outliers than the emprical covariance.
\end{itemize}

\begin{figure}
    \centering
    \includegraphics[scale=0.1]{figures/high_frequencies_datasets.jpg}
    \caption{Same experiment than Fig. \ref{fig:high_frequencies}, shown for each dataset, without score normalization}
    \label{fig:high_frequency_dataset}
\end{figure}

Show example of decision boundaries for the best NNs vs tree-based models.

\subsection{Do our benchmarks depend on the hyperparameters sample space?}\label{supp:search_space}

For a given n, select the best order of n hp to try and then compare.

\subsection{How to use our benchmark?}\label{supp:howto}