Audits Under Resource and Information Constraints: Scaling Laws for Fairness Pareto Frontiers
Sarah Cen, Salil Goyal, Zaynah Javed, Ananya Karthik, Percy Liang, Daniel Ho
Model audits play a critical role in holding AI actors accountable, and one branch of the law for which model audits are particularly salient is discrimination law. Several areas of discrimination law (including but extending beyond employment) implicate what is known as the "less discriminatory alternatives" (LDA) framework, in which a policy (i.e., model) is defensible if no LDA can be found with a reasonable amount of effort. Notably, the burden of proving that an LDA exists typically falls on the claimant (the party alleging discrimination) rather than the defendant. This creates a significant hurdle, as claimants would seemingly need to produce a less discriminatory model, a task requiring resources and expertise beyond most litigants. Moreover, model developers generally shield any information about the current model and training data as trade secrets. In this work, we propose a novel toolkit enabling claimants to establish the existence of LDAs even if they lack significant compute, training data, or information about the current model. Our method allows claimants to demonstrate that (obtainable) less discriminatory models exist without producing them, requiring substantially fewer resources and (potentially proprietary) information than model development. We do so by establishing a scaling law for the loss-fairness Pareto frontier (PF). As our main result, we provide a closed-form upper bound for the PF, where we focus on fairness as demographic parity as an illustrative example. Using this expression, the claimant can fit a curve to the PF in the "low resource-information regime," then extrapolate the PF that applies for the (large) model being contested. The expression thus serves as a scaling law for loss-fairness PFs.