Summa Intelligentiae: Data, Patterns, and Feedback
Quaestio TERTIA
03
Data, Patterns, and Feedback
The Fuel of Machine Learning
Before a model learns, it consumes.
Every AI system begins by ingesting trillions of words, images, and lines of code. Textbooks and tweets, Shakespeare and Reddit, all collected into a single digital pantry.
In The Mechanics of Intelligence, we explored the recipe: parameters, weights, and neuron layers that transform data into learning. But even the best recipe depends on what’s in the kitchen. Once baked, the dish is set; no plating can turn bologna into wagyu.
Yet, quality ingredients in skilled hands can create something remarkable. The data consumed during training becomes a model’s permanent flavor palette. Different AI companies may tweak the recipe, but they all cook with ingredients drawn from the same digital world. Some from pristine sources, others from the chaos of the open web.
So, what exactly are these models consuming, and why do those ingredients define everything that follows?
Insights to Expect
Why data is the fuel and a fundamental building block of AI models.
How pre-training data defines a model’s core capabilities and worldview.
What differences emerge from where data is implemented in training and refinement cycles
How vendors acquire data
How data provenance shapes trust, bias, and legality.
Data is to a model what experience is to a mind. Not yet knowledge or understanding, but merely raw material from which both emerge.
Early machine learning systems learned through supervision. Labeled data sets where every input had a corollary, correct answer. Learning was effective but slow as each label required a human teacher. Then came a breakthrough: models that could learn by tasting.
Self-supervised learning allowed networks to predict what comes next: the next pixel in a frame, the next word in a sentence. Instead of relying on instruction, they discovered structure through exposure.
Two 2012 experiments cemented this shift.
AlexNet showed how deep architectures could extract visual hierarchies — edges, shapes, objects.
DeepMind’s “Cat Paper” revealed that an unlabeled network trained on YouTube frames could recognize cats without ever being told what a cat was.
Together, these studies proved that learning could emerge from experience alone. Modern foundation models inherit that lesson: trained not by rulebooks, but by a data diet.
Visual representation of abstraction layers. Source: GPT-5.
From Ingredients to Structure
Each training example, whether labeled or not, presents a puzzle the model must solve. It predicts an output, compares it to the true or expected result, and measures the discrepancy as loss. That loss is transformed into feedback through backpropagation, a mathematical mirror that shows the network where it went wrong and how to adjust.
Through repetition, structure appears.
In vision, neurons respond to edges, then textures, then entire forms.
In language, they progress from letters to grammar to meaning.
This process is called representation learning, the art of turning data into geometry. Statistical regularities become directions in a vast mathematical space where similar meanings cluster and patterns compress (see “gradient descent” discussed in The Mechanics of Intelligence). Data becomes structure, and structure becomes understanding.
Scale and Abstraction: When Quantity Becomes Quality
As datasets and models grew, performance didn’t just improve, it evolved.
Scaling laws showed that error shrinks predictably as data and parameters expand (Hoffmann et al., 2022). Beyond certain thresholds, new abilities emerge.
Small datasets teach memorization; large, diverse ones teach generalization. Exposure to rare cases (e.g., idioms, exceptions, edge examples) helps systems infer abstract rules never directly taught. That’s why GPT-class models trained on trillions of tokens can compose poetry, translate languages, and summarize philosophy without explicit instruction.
But scale cuts both ways. More data amplifies both intelligence and bias. Overrepresentation of one tone, culture, or medium becomes the model’s gravitational center. Breadth breeds abstraction; imbalance breeds beliefs.
Provenance and Purity: The Origins of Knowledge
Not all data teaches equally. Provenance, the source and preparation of information, determines what kind of learning a model can achieve.
Public web data offers breadth but high noise — a buffet with questionable hygiene.
Curated or licensed corpora deliver precision but limit diversity — fine dining with a narrow menu.
Synthetic data expands scale cheaply but risks self-reference — GMO, not organic, and definitely not farm-to-table.
Provenance also shapes abstraction. Exact reproductions (i.e., books, code, verbatim text) create “crystallized paths” where prediction becomes recall. Imperfect data, by contrast, forces inference. Uncertainty compels the model to reason instead of recite.
Thus, provenance is both ethical and mechanical. Purity ensures traceability; diversity fuels insight; imperfection enables intelligence. Together they decide whether a system learns structure or merely mirrors it.
Poison and Integrity: The Fragility of Learning
The same feedback loop that lets data teach can also deceive. Every training sample adjusts billions of parameters. A handful of corrupted examples can influence an entire model.
In 2025, Anthropic and the UK AI Safety Institute demonstrated that as few as 250 poisoned documents could implant hidden behaviors across models ranging from 600 million to 13 billion parameters (Souly et al., 2025). The effect scaled not by proportion but by presence: one bad ingredient can spoil the pot.
Poisoning works by hijacking feedback, steering the model toward a hidden goal. Even non-malicious contamination, like benchmark leakage, can create false confidence. The system looks brilliant on paper but has only memorized the test.
Whether by poison or perfection, the result is the same: abstraction collapses. The model confuses memorization with mastery.
Guarding against this requires rigorous data filtering, deduplication, and provenance audits. Clean data keeps the kitchen safe and the learning valid.
The Limits of Data
Data teaches correlation, not comprehension. It allows machines to imitate the structure of intelligence, not its intent.
Even infinite data cannot give a model purpose. It defines what can be cooked, not why to cook it. Fine-tuning may add seasoning, but the dish remains a composition of its initial ingredients.
Understanding these limits clarifies what AI truly is: a mirror polished by exposure. Reflective, refined, but not self-aware.
Final Thoughts
Data defines the boundaries of intelligence.
Quality, diversity, and provenance determine what a model can know—and what it cannot.
Scale transforms examples into abstractions, but imbalance turns insight into bias.
