Feasible Learning
We introduce Feasible Learning (FL), a sample-centric learning paradigm where
models are trained by solving a feasibility problem that bounds the loss for
each training sample. In contrast to the ubiquitous Empirical Risk Minimization
(ERM) framework, which optimizes for average performance, FL demands
satisfactory performance on every individual data point. Since any model that
meets the prescribed performance threshold is a valid FL solution, the choice
of optimization algorithm and its dynamics play a crucial role in shaping the
properties of the resulting solutions. In particular, we study a primal-dual
approach which dynamically re-weights the importance of each sample during
training. To address the challenge of setting a meaningful threshold in
practice, we introduce a relaxation of FL that incorporates slack variables of
minimal norm. Our empirical analysis, spanning image classification, age
regression, and preference optimization in large language models, demonstrates
that models trained via FL can learn from data while displaying improved tail
behavior compared to ERM, with only a marginal impact on average performance.
67986d764fccd4b95149db0b
Juan Ramirez
admin_assigned
2025-01-28T16:53:24.742Z
65555c6c6947208b77271f1e
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/CQPL5xB-Y-F89BxJrB7tg.png
Juan Ramírez
user
juanramirezneilson
67986d764fccd4b95149db0c
Ignacio Hounie
admin_assigned
2025-01-28T19:40:24.939Z
6508b3f15b34509e16c78fea
/avatars/f663439c405354504b27f5ffab5c401a.svg
Ignacio Hounie
user
ihounie
67986d764fccd4b95149db0d
Juan Elenter
admin_assigned
2025-01-28T19:40:32.041Z
65a04adddb5d37ad5e6b9d29
/avatars/071a137c2c1aad0699ee1b8b001e4a58.svg
Elenter
user
juanelenter
67986d764fccd4b95149db0e
Jose Gallego-Posada
67986d764fccd4b95149db0f
Meraj Hashemizadeh
67986d764fccd4b95149db10
Alejandro Ribeiro
admin_assigned
2025-01-28T19:24:31.422Z
6724c42c055ce33014feaec2
/avatars/8854289d890b555ca9562f95edeab784.svg
Alejandro Ribeiro Prieto
user
Prieto
67986d764fccd4b95149db11
Simon Lacoste-Julien
admin_assigned
2025-01-28T16:53:32.537Z
6406bcaca577649430c6bff4
/avatars/77b386172b0d86d40cddd7a7a8744491.svg
Simon Lacoste-Julien
user
slacoste
Feasible Learning
We introduce Feasible Learning (FL), a sample-centric learning paradigm where
models are trained by solving a feasibility problem that bounds the loss for
each training sample. In contrast to the ubiquitous Empirical Risk Minimization
(ERM) framework, which optimizes for average performance, FL demands
satisfactory performance on every individual data point. Since any model that
meets the prescribed performance threshold is a valid FL solution, the choice
of optimization algorithm and its dynamics play a crucial role in shaping the
properties of the resulting solutions. In particular, we study a primal-dual
approach which dynamically re-weights the importance of each sample during
training. To address the challenge of setting a meaningful threshold in
practice, we introduce a relaxation of FL that incorporates slack variables of
minimal norm. Our empirical analysis, spanning image classification, age
regression, and preference optimization in large language models, demonstrates
that models trained via FL can learn from data while displaying improved tail
behavior compared to ERM, with only a marginal impact on average performance.
Feasible Learning trains models by ensuring satisfactory performance on each sample, using a primal-dual approach and minimal norm slack variables, improving tail behavior with slight impact on average performance.
Feasible Learning sample-centric learning feasibility problem Empirical Risk Minimization primal-dual approach slack variables