This piece first appeared in German. The English version is a rewrite rather than a line-by-line translation, and the German original stays online in the archive: Wenn das Testset lügt: Die weitreichenden Folgen von falschen Datensets.

ImageNet sets aside 50,000 photographs for evaluation, each labeled by a person with what it shows, such as a tabby cat or a teapot. Those labels work as an answer key, called a test set, and the image-recognition model that matches the most of them tops the leaderboard. In 2021 Curtis Northcutt and two colleagues checked the key itself and found almost 3,000 wrong answers. The corrected key favors the smaller model. Once mislabeled photos make up six percentage points more of the test, ResNet-18 overtakes ResNet-50, which is nearly three times as deep and leads on the original labels.

Anyone selecting a model relies on a simple assumption: the highest accuracy on the test set marks the best model for production. That assumption can fail systematically, and the reason lies neither in overfitting nor in a weak architecture. It sits much earlier in the pipeline, in the labels of the test set itself.

Test sets are treated as ground truth and rarely audited

Arguments about data quality almost always concern training data. Test sets count as the dependable part, because they are the yardstick everything else is measured against. Northcutt, Athalye and Mueller examined that assumption systematically in a 2021 NeurIPS study, and it did not hold.

The authors went through ten of the most used benchmarks in computer vision, language and audio, ImageNet, CIFAR-10, CIFAR-100 and Amazon Reviews among them. An algorithmic framework called Confident Learning identified candidate errors, which human reviewers then validated through Amazon Mechanical Turk.

On average the test sets carried at least 3.3 percent incorrect labels. For the ImageNet validation set, the benchmark of record in image classification, the main table reports at least 6 percent. The authors then revise it. After expert review of examples the algorithm had never flagged, they put the figure closer to 20 percent.

What is Confident Learning? A framework that finds label errors by flagging cases where a model predicts a different class with high confidence than the one recorded. Put plainly: the model is more certain than the annotator was.



The studies underneath this article date from 2021. They stay relevant as evidence of how sensitive these systems are to errors in the data they are measured on, and not as an assessment of how today's models perform.

Cleaning the labels reorders the leaderboard

The error rate alone would be worth knowing. What the errors do to model selection is more serious. On corrected test data the ranking of established models partly inverts: NASNet-large drops from first place to 29th of 34 models tested, and ResNet-18 climbs from 34th to first. The same pattern shows up on CIFAR-10, where VGG-11 beats VGG-19 once the labels are fixed, having scored worse on the originals.

The mechanism is not classical overfitting. Large models have the capacity to learn subtle statistical patterns, and the systematic error structure of the annotators is one of them. They optimize for the benchmark and not for the world. Smaller models cannot memorize that structure in full, which leaves them more accurate once the labels are right. It is overfitting to a dataset's mistakes.

In classroom terms, the large model behaves like a student who has learned the grader's quirks, mistakes included. Marked by that same grader, the student scores near the top of the class. Once a careful grader rechecks the answers, that score drops, and a classmate who could never pick up every quirk, and learned the subject itself instead, moves ahead. For anyone choosing a model, this is the practical lesson: a high benchmark score shows how well a model matches the answer key, and the key itself can be wrong.

In practice you only ever see the wrong number

The problem stays hidden for a mundane reason. What a team can measure is the original accuracy, the score against the faulty labels. Corrected accuracy, the score against what is actually true, requires laborious manual cleaning before it exists at all. Select a model on conventional benchmark metrics and you might be deciding against a distorted yardstick without knowing it.

Real-world datasets are also considerably messier than the carefully curated benchmarks in the study, which makes the 3.3 percent a lower bound. In domain-specific applications, with fewer annotation resources, more subjective labeling calls and faster collection cycles, markedly higher error rates are realistic.

Finance has the labeling problem built in

Historical credit decisions serve as ground truth for scoring models, and they frequently record something other than an objective probability of default. What they record is how individual case handlers judged a file at a particular moment, under a market regime, internal guidelines and a personal appetite for risk. Train and evaluate a complex model on that base and you may be optimizing for the errors of past decisions.

Sentiment classifiers on earnings call transcripts have the same weakness. Where the training labels were assigned by analysts with different levels of experience, the test set very probably holds systematic inconsistencies. The model with the highest measured accuracy is then not the one most likely to survive contact with the work.

Conclusion

Northcutt and colleagues make an uncomfortable argument empirically watertight. Benchmark accuracy is an unreliable indicator of real model quality for as long as the test sets themselves carry errors. Choose by conventional metrics and you may be making the wrong call, with no mechanism that would ever tell you so.

Sources

Grouped by the section they support.

Opening

  • Northcutt, Athalye and Mueller (2021), Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, NeurIPS 2021 Datasets and Benchmarks Track, arXiv:2103.14749. Supports the 2,916 wrong labels among 50,000 ImageNet validation images (Table 1) and ResNet-18 overtaking ResNet-50 (Section 5.2, Figure 4).
  • Russakovsky et al. (2015), ImageNet Large Scale Visual Recognition Challenge, International Journal of Computer Vision 115, arXiv:1409.0575. Supports ImageNet and its validation set of 50,000 images.
  • He, Zhang, Ren and Sun (2016), Deep Residual Learning for Image Recognition, CVPR 2016, arXiv:1512.03385. Supports ResNet-18 and ResNet-50 and their depth of 18 and 50 layers.

Test sets are treated as ground truth and rarely audited

  • Northcutt et al. (2021). The ten benchmarks (Appendix A), the review on Mechanical Turk (Section 4), at least 3.3 percent on average (abstract), at least 6 percent for ImageNet (Table 1), and closer to 20 percent after expert review (Section 6).
  • Northcutt, Jiang and Chuang (2021), Confident Learning: Estimating Uncertainty in Dataset Labels, Journal of Artificial Intelligence Research 70, 1373–1411, arXiv:1911.00068. Supports the Confident Learning method and the box that explains it.

Cleaning the labels reorders the leaderboard

  • Northcutt et al. (2021), Section 5.1. NASNet-large falling from 1st to 29th and ResNet-18 rising from 34th to 1st, measured on the test images whose labels were corrected; VGG-11 ahead of VGG-19 on CIFAR-10; and smaller models as less prone to learning the pattern of wrong labels.
  • Zoph, Vasudevan, Shlens and Le (2018), Learning Transferable Architectures for Scalable Image Recognition, CVPR 2018, arXiv:1707.07012. Supports NASNet-large.
  • Simonyan and Zisserman (2015), Very Deep Convolutional Networks for Large-Scale Image Recognition, ICLR 2015, arXiv:1409.1556. Supports VGG-11 and VGG-19.
  • Krizhevsky (2009), Learning Multiple Layers of Features from Tiny Images, technical report, University of Toronto. Supports CIFAR-10.

In practice you only ever see the wrong number

  • Northcutt et al. (2021), Section 5, Definitions 1 and 2. Supports original and corrected accuracy. The point that real-world data is messier than curated benchmarks is the author's own assessment.

Finance has the labeling problem built in

  • No external source. Both examples apply the study's finding to finance and are the author's own analysis.

Conclusion

  • Northcutt et al. (2021), Sections 5 and 7.
The link has been copied!