Survey of Active Learning Hyperparameters: Insights from a Large-Scale
  Experimental Grid
Annotating data is a time-consuming and costly task, but it is inherently
required for supervised machine learning. Active Learning (AL) is an
established method that minimizes human labeling effort by iteratively
selecting the most informative unlabeled samples for expert annotation, thereby
improving the overall classification performance. Even though AL has been known
for decades, AL is still rarely used in real-world applications. As indicated
in the two community web surveys among the NLP community about AL, two main
reasons continue to hold practitioners back from using AL: first, the
complexity of setting AL up, and second, a lack of trust in its effectiveness.
We hypothesize that both reasons share the same culprit: the large
hyperparameter space of AL. This mostly unexplored hyperparameter space often
leads to misleading and irreproducible AL experiment results. In this study, we
first compiled a large hyperparameter grid of over 4.6 million hyperparameter
combinations, second, recorded the performance of all combinations in the
so-far biggest conducted AL study, and third, analyzed the impact of each
hyperparameter in the experiment results. In the end, we give recommendations
about the influence of each hyperparameter, demonstrate the surprising
influence of the concrete AL strategy implementation, and outline an
experimental study design for reproducible AL experiments with minimal
computational effort, thus contributing to more reproducible and trustworthy AL
research in the future.
68415ee454d7c6b3f9786deb
Julius Gonsior
admin_assigned
2025-06-05T12:58:30.615Z
637638fa1f0421002b42facb
/avatars/ed1e3024cd2ed7284b437db4bbeb2668.svg
Julius Gonsior
user
jgonsior
68415ee454d7c6b3f9786dec
Tim Rieß
admin_assigned
2025-06-05T12:58:21.628Z
6728a69e4d0fe229c96eea4c
/avatars/0c6ee4d8dd0752654ca5d86f27bebaea.svg
Tim Rieß
user
Timischaf
68415ee454d7c6b3f9786ded
Anja Reusch
68415ee454d7c6b3f9786dee
Claudio Hartmann
68415ee454d7c6b3f9786def
Maik Thiele
admin_assigned
2025-06-05T12:58:04.471Z
65f5af9309cf7381af5e261b
https://cdn-avatars.huggingface.co/v1/production/uploads/65f5af9309cf7381af5e261b/2ciaq2fzRwkpf7jk0k9JZ.png
Maik Thiele
user
DILHTWD
68415ee454d7c6b3f9786df0
Wolfgang Lehner
Survey of Active Learning Hyperparameters: Insights from a Large-Scale
  Experimental Grid
Annotating data is a time-consuming and costly task, but it is inherently
required for supervised machine learning. Active Learning (AL) is an
established method that minimizes human labeling effort by iteratively
selecting the most informative unlabeled samples for expert annotation, thereby
improving the overall classification performance. Even though AL has been known
for decades, AL is still rarely used in real-world applications. As indicated
in the two community web surveys among the NLP community about AL, two main
reasons continue to hold practitioners back from using AL: first, the
complexity of setting AL up, and second, a lack of trust in its effectiveness.
We hypothesize that both reasons share the same culprit: the large
hyperparameter space of AL. This mostly unexplored hyperparameter space often
leads to misleading and irreproducible AL experiment results. In this study, we
first compiled a large hyperparameter grid of over 4.6 million hyperparameter
combinations, second, recorded the performance of all combinations in the
so-far biggest conducted AL study, and third, analyzed the impact of each
hyperparameter in the experiment results. In the end, we give recommendations
about the influence of each hyperparameter, demonstrate the surprising
influence of the concrete AL strategy implementation, and outline an
experimental study design for reproducible AL experiments with minimal
computational effort, thus contributing to more reproducible and trustworthy AL
research in the future.
The study investigates the impact of hyperparameters on Active Learning performance, providing insights to improve its practical application and reproducibility.
Active Learning hyperparameter space hyperparameter grid experimental study design reproducibility trustworthiness