This piece first appeared in German. The English version is a rewrite rather than a line-by-line translation, and the German original stays online in the archive: Warum Datenqualität über Modellqualität entscheidet.

Researchers at Google put the same questions to 53 people who build AI systems for credit scoring, health care and public safety in India, the United States, and across East and West Africa. Ninety-two percent of them had watched a problem in their data travel the length of a pipeline and surface much later, in a system already running. Almost half had watched it happen twice or more inside a single project.

That says something about where effort goes in this field. The industry argues about architectures, parameter counts and training methods, and treats the input data as preparation for the real work. For anyone choosing a model or signing off on what one produces, the finding turns the problem around. Data is the expensive half. Attention goes to the model.

None of this is new. Garbage in, garbage out has been the shorthand for decades, and what has changed is how long the garbage stays invisible. A classical machine learning system told you something was wrong through a confusion matrix, the table that shows which classes a model mixes up, or through a validation loss that drifted. Large language models hand back coherent, well-formed prose over data that is quietly broken. Quality became harder to control, and no less decisive.

A cascade is a chain, not a mistake

The term comes from a 2021 study by Sambasivan and colleagues at Google Research. A data cascade starts upstream, in collection or labeling, then compounds as it moves through the pipeline. It often shows up for the first time months or years later in production. The researchers were careful about where these come from: conventional AI practice produces them. Rushing to a proof of concept does it, so does tuning model performance while the pipeline stays broken, and so does treating data work as an operational side task somebody will get to.

The study's sharpest finding is a line from a practitioner in health care, which the authors put in their title: "Everyone wants to do the model work, not the data work."

Four things set a cascade off

The most common trigger, reported by 54.7 percent of the practitioners, is contact with the physical world. Models are developed on clean, controlled training data and then meet weather, wear and drift. Cameras shift on their mounts, sensors go past their maintenance interval, lighting changes, and the system starts receiving data of a kind it has never seen. These are also the slowest cascades to become visible, taking up to two or three years after deployment to emerge.

Next, at 43.4 percent, comes labeling without enough domain expertise. When a developer has to decide what counts as correct ground truth, whether an insurance claim was justified, whether a default was avoidable, their assumptions enter the dataset as fact. Finance is a hard case here. Historical credit decisions record the judgment of individual case handlers under the conditions of their time, and a model trained on them learns that judgment, not an objective probability of default.

Conflicting reward systems account for 32.1 percent. Doctors, case handlers and analysts collect much of it, and they are measured on their own work, not on data hygiene. The result is incomplete capture, wrong sampling frequencies and, where a target is attached, invented entries. Poor documentation across organizations accounts for the remaining 20.8 percent: without metadata, teams guess what a dataset contains, and an expensive guess can cost months of work.

The fix is the work nobody gets credit for

The incentive problem sits deeper than any single project. In academic machine learning, a new architecture becomes visible through publications, benchmark results and conference talks. Cleaning and documenting a dataset produces none of those things, so one path builds a reputation and the other is done in the dark.

Companies reproduce the pattern for their own reasons. A model upgrade can be shown in a quarterly report and data maintenance cannot. Budget goes to the visible layer of the stack, to the models and the infrastructure and the inference costs, while the base the whole thing rests on is left alone. Sambasivan and colleagues found this bias across organizations and countries, in well-run teams too. Every practitioner they interviewed already knew that data quality mattered, and the cascades happened anyway.

Bad labels can invert a benchmark ranking

A team at the Massachusetts Institute of Technology approached the problem one step earlier. Northcutt and colleagues went through the test sets of ten of the most used machine learning benchmarks, ImageNet, CIFAR-10 and Amazon Reviews among them. On average, they estimated, those sets contain at least 3.3 percent incorrect labels. For the ImageNet validation set, the set the field has treated as its yardstick for a decade, the main table reports at least 6 percent. A later section revises that upward. After expert review of examples the algorithm had never flagged, the authors put the validation set closer to 20 percent.

The error rate on its own would be worth knowing. What the errors do to model selection is the part that should change behavior. On the corrected test data the ranking of established models partly inverts: the complex NASNet-large falls from first place to 29th of 34 models tested, and the far simpler ResNet-18 climbs from 34th to first.

The mechanism is structural: large models have the capacity to learn every statistical pattern in their training data, including the systematic mistakes of the people who labeled it. They optimize for the benchmark and not for the world. Smaller models cannot memorize an error structure that size, and they do better once the labels are correct. This is not ordinary overfitting to data. It is overfitting to a dataset's mistakes.

What is a label error? A data point that has been assigned the wrong category. A photograph of a black stork carries the label "white stork". A model trained on it learns the annotator's mistake and not the bird.

Where this lands in finance

Neither study was written with finance in mind, and both apply to it directly.

Credit scoring is the first case. Take a model trained on lending decisions from before 2008, or from before the pandemic. It has a data problem that no amount of configuration will fix, because the sample describes a market regime that no longer exists. The error enters upstream and becomes visible in production, which is a cascade in its textbook form.

Sentiment analysis on earnings calls is the second. Train a language model on general text and point it at financial disclosure, and it will misread words like "liability" and "hedging", which carry a different meaning in the room than in ordinary usage. The quality of the domain-specific training data decides what the output is worth. Model size does not.

Better models buy more expensive mistakes

Spending on a better model while leaving the pipeline as it is buys errors at a higher price. The research is consistent on this point across image classification and credit risk alike. Data quality is the deciding variable in what an AI system can do, and it is no operational side task. As long as data work carries less prestige than model development, cascades will keep being the rule.

Sources

Grouped by the section they support.

Opening

  • Sambasivan, Kapania, Highfill, Akrong, Paritosh and Aroyo (2021), "Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI, CHI 2021, Yokohama, doi.org/10.1145/3411764.3445518, free PDF via research.google. Supports the 53 practitioners, their domains and regions (Table 1), and the 92 percent with at least one cascade and 45.3 percent with two or more.
  • Ji et al. (2023), Survey of Hallucination in Natural Language Generation, ACM Computing Surveys 55 (12), arXiv:2202.03629. Supports language models producing fluent text that is wrong. "Garbage in, garbage out" is a long-standing rule of thumb and needs no source.

A cascade is a chain, not a mistake

  • Sambasivan et al. (2021). Supports the definition of a data cascade, its roots in conventional AI practice, and the title quotation, which comes from a participant working in health care in India.

Four things set a cascade off

  • Sambasivan et al. (2021), Table 2. Supports the four triggers and their shares: 54.7, 43.4, 32.1 and 20.8 percent. Section 4.3.1 supports physical-world cascades taking up to two or three years to emerge. Section 4.3.3 supports incentives leading field workers to fabricate data. The remark on historical credit decisions is the author's own.

The fix is the work nobody gets credit for

  • Sambasivan et al. (2021). Supports model work as the path to prestige and publications, and the finding that practitioners knew data quality mattered and still experienced cascades. The pattern inside companies, quarterly reports and budgets, is the author's own analysis.

Bad labels can invert a benchmark ranking

  • Northcutt, Athalye and Mueller (2021), Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, NeurIPS 2021 Datasets and Benchmarks Track, arXiv:2103.14749. Supports the ten benchmarks, at least 3.3 percent on average, at least 6 percent for ImageNet (Table 1), closer to 20 percent after expert review (Section 6), and the rank changes of NASNet-large and ResNet-18, measured on the test images whose labels were corrected (Section 5.1). The stork in the box is an illustration.

Where this lands in finance

  • Loughran and McDonald (2011), When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks, Journal of Finance 66 (1), 35–65, doi.org/10.1111/j.1540-6261.2010.01625.x. Supports the finding that general-purpose word lists misread financial language: almost three quarters of the words the Harvard dictionary marks as negative are not negative in a financial context. The credit-scoring example is the author's own application of both studies.

Better models buy more expensive mistakes

  • No external source. The conclusion is the author's own.
The link has been copied!