Efficient and Fair Data Pruning for Deep Learning Models

Table of Links

Abstract and 1 Introduction

2 Data Pruning & Fairness

3 Method & Results

4 Theoretical Analysis

5 Discussion, Acknowledgments and Disclosure of Funding, and References

A Implementation Details

B Theoretical Analysis for a Mixture of Gaussians

Abstract

In the era of exceptionally data-hungry models, careful selection of the training data is essential to mitigate the extensive costs of deep learning. Data pruning offers a solution by removing redundant or uninformative samples from the dataset, which yields faster convergence and improved neural scaling laws. However, little is known about its impact on classification bias of the trained models. We conduct the first systematic study of this effect and reveal that existing data pruning algorithms can produce highly biased classifiers. At the same time, we argue that random data pruning with appropriate class ratios has potential to improve the worst-class performance. We propose a “fairness-aware” approach to pruning and empirically demonstrate its performance on standard computer vision benchmarks. In sharp contrast to existing algorithms, our proposed method continues improving robustness at a tolerable drop of average performance as we prune more from the datasets. We present theoretical analysis of the classification risk in a mixture of Gaussians to further motivate our algorithm and support our findings.

1 Introduction

The ever-increasing state-of-the-art performance of deep learning models requires exponentially larger volumes of training data according to neural scaling laws [Hestness et al., 2017, Kaplan et al., 2020, Rosenfeld et al., 2020, Gordon et al., 2021]. However, not all collected data is equally important for learning as it contains noisy, repetitive, or uninformative samples. A recent research thread on data pruning is concerned with removing those unnecessary data, resulting in improved convergence speed, neural scaling factors, and resource efficiency [Toneva et al., 2019, Paul et al., 2021, He et al., 2023]. These methods design scoring mechanisms to assess the utility of each sample, often measured by its difficulty or uncertainty as approximated during a preliminary training round, that guides pruning. Sorscher et al. [2022] report that selecting high-quality data using these techniques can trace a Pareto optimal frontier, beating the notorious power scaling laws.

The modus operandi of data pruning resembles algorithms that mitigate distributional bias—a well-established issue in AI systems concerning the performance disparity across protected groups of the population (e.g., race or gender) or classes [Dwork et al., 2012, Hardt et al., 2016]. The early approaches in this domain improve the worst-class performance of imbalanced datasets by subsampling the majority classes [Barua et al., 2012, Cui et al., 2019, Tan et al., 2020]. Data pruning extends this strategy by designing finer pruning criteria and working for balanced datasets, too. More recent developments apply importance weighting to emphasize under-represented or underperforming groups or classes during optimization [Sinha et al., 2022, Sagawa* et al., 2020, Liu et al., 2021, Idrissi et al., 2022, Wang et al., 2023, Lukasik et al., 2022, Chen et al., 2017]. Data pruning adheres to this paradigm as well by effectively assigning zero weights to the removed training samples. These parallels indicate that data pruning has potential to reduce classification bias in deep learning models, all while offering greater resource and memory efficiency. In Section 2, we conduct the first systematic evaluation of various pruning algorithms with respect to distributional robustness and find that this potential remains largely unrealized. For example, we find that Dynamic Uncertainty by He et al. [2023] achieves superior average test performance on CIFAR-100 with VGG-19 but fails miserably in terms of worst-class accuracy. Thus, it is imperative to benchmark pruning algorithms using a more comprehensive suite of performance metrics that reflect classification bias, and to develop solutions that address fairness directly.

In contrast to data pruning, many prior works on distributionally robust optimization compute difficulty-based importance scores at the level of groups or classes and not for individual training samples. For pruning, this would correspond to selecting appropriate class ratios but subsampling randomly within each class. To imitate this behavior, we propose to select class proportions based on the corresponding class-wise error rates computed on a hold-out validation set after a preliminary training on the full dataset, and call this procedure Metric-based Quotas (MetriQ). While these quotas already improve robustness of the existing pruning algorithms, they particularly shine when applied together with random pruning, substantially reducing the classification bias of models trained on the full dataset while offering an enhanced data efficiency as a bonus. Our pruning protocol even achieves higher robustness than one cost-sensitive learning approach that has access to the full dataset [Sinha et al., 2022], at no degradation of the average performance for low enough pruning ratios. A theoretical analysis of linear decision rules for a mixture of two univariate Gaussians presented in Section 4 illustrates why simple random subsampling according to MetriQ yields better worst-class performance compared to existing, finer pruning algorithms.

The summary of our contributions and the structure of the remainder of the paper are as follows.

• In Section 2, using a standard computer vision testbed, we conduct the first comprehensive evaluation of existing data pruning algorithms through the lens of fairness;

• In Section 3, we propose a random pruning procedure with error-based class ratios coined MetriQ, and verify its effectiveness in drastically reducing the classification bias;

• In Section 4, we back our empirical work with theoretical analysis of a toy Gaussian mixture model to illustrate the fundamental principles that contribute to the success of MetriQ.