Language models are weak learners
A central notion in practical and theoretical machine learning is that of a
weak learner, classifiers that achieve better-than-random
performance (on any given distribution over data), even by a small margin. Such
weak learners form the practical basis for canonical machine learning methods
such as boosting. In this work, we illustrate that prompt-based large language
models can operate effectively as said weak learners. Specifically, we
illustrate the use of a large language model (LLM) as a weak learner in a
boosting algorithm applied to tabular data. We show that by providing (properly
sampled according to the distribution of interest) text descriptions of tabular
data samples, LLMs can produce a summary of the samples that serves as a
template for classification and achieves the aim of acting as a weak learner on
this task. We incorporate these models into a boosting approach, which in some
settings can leverage the knowledge within the LLM to outperform traditional
tree-based boosting. The model outperforms both few-shot learning and
occasionally even more involved fine-tuning procedures, particularly for tasks
involving small numbers of data points. The results illustrate the potential
for prompt-based LLMs to function not just as few-shot learners themselves, but
as components of larger machine learning pipelines.
649a48b5478082a13ece75c5
Hariharan Manikandan
claimed_verified
2023-06-29T07:30:55.364Z
63e8013ee02ee67e8e56da42
/avatars/b4c514433c3c72bd3b09dbf20fb76a09.svg
Hariharan Manikandan
user
vshshv3
649a48b5478082a13ece75c6
Yiding Jiang
649a48b5478082a13ece75c7
J Zico Kolter
Language models are weak learners
A central notion in practical and theoretical machine learning is that of a
weak learner, classifiers that achieve better-than-random
performance (on any given distribution over data), even by a small margin. Such
weak learners form the practical basis for canonical machine learning methods
such as boosting. In this work, we illustrate that prompt-based large language
models can operate effectively as said weak learners. Specifically, we
illustrate the use of a large language model (LLM) as a weak learner in a
boosting algorithm applied to tabular data. We show that by providing (properly
sampled according to the distribution of interest) text descriptions of tabular
data samples, LLMs can produce a summary of the samples that serves as a
template for classification and achieves the aim of acting as a weak learner on
this task. We incorporate these models into a boosting approach, which in some
settings can leverage the knowledge within the LLM to outperform traditional
tree-based boosting. The model outperforms both few-shot learning and
occasionally even more involved fine-tuning procedures, particularly for tasks
involving small numbers of data points. The results illustrate the potential
for prompt-based LLMs to function not just as few-shot learners themselves, but
as components of larger machine learning pipelines.
weak learner boosting algorithm large language model (LLM) prompt-based large language models tabular data few-shot learning template for classifications traditional tree-based boosting fine-tuning procedures machine learning pipelines