AI timelines never converge, for two different reasons. One is a life-long belief about how human intelligence works that keeps shaping how every new AI result gets read: not a one-time split but an ongoing feedback loop. The other is that “AGI” is underspecified enough that two people can each give a coherent, carefully-reasoned forecast to a different question and call it one disagreement. Neither yields to more evidence, and neither requires anyone to be biased.

A Belief Formed Over a Lifetime

AGI’s whole premise is matching human intelligence, so what you believe about the brain shapes what you expect from any attempt to build it. Some suspect something about the mind hasn’t been discovered yet, so a machine matching it is distant almost by definition. Others treat the brain as a biological computer, remarkable but not miraculous: no missing ingredient, just gaps, and it’s an open question whether closing them takes more scale on the current architecture or a genuinely different one.Some of those gaps, like continual learning, show no measurable trajectory to extrapolate, so it's not obvious that scaling compute and data closes them (more here). Whichever you pick, the belief usually formed over a lifetime, long before anyone watched a transformer solve an olympiad problem.

This is the theory a Bayesian finds least embarrassing: the split lives in the prior. If two people start from different priors, both can update on the same evidence with perfect rationality and land decades apart. Nobody is being unreasonable, they just began in different places.

It’s a tidy theory, and it’s incomplete. A fixed starting difference explains a one-time gap between two people. It doesn’t explain why that gap widens with each release, why watching the same model solve the same benchmark leaves both sides more certain than before. A one-time difference in starting point should, if anything, erode as shared evidence accumulates. Something else is doing the work of keeping the split alive, and widening it.

Why Rational People Still Diverge

That something is hiding in the update rule itself. Bayesian updating has a simple recursive structure:

$H$ is the hypothesis you’re tracking, “AGI arrives by year Y,” say, and $E_t$ is the evidence that just arrived. The recursive part is easy to miss: $P_{t-1}(H)$, today’s prior, is nothing but yesterday’s posterior.For a closer look at this same recursion, worked out for continual learning in neural networks rather than human belief updating, see Section 4.1.2 of my PhD thesis . Whatever you believed after the last piece of evidence is exactly the starting point for reading the next one.

Two ideal Bayesians with a common prior cannot agree to disagree: once their posteriors are common knowledge, the posteriors must be equal . The AI community delivers the common-knowledge half of that setup, evidence and opinions broadcast constantly, yet produces the opposite of convergence. So the natural suspect is the other condition: the priors aren’t common. The interesting question is where they differ.

They differ one level up from the timeline question. Evidence about AI capability doesn’t bear on “will AGI arrive by year Y” directly. It reaches that question through a lens: a belief about whether the current paradigm generalizes or is doing something closer to sophisticated interpolation. And the lens is where the belief from the last section keeps acting: whether you expect the paradigm to generalize is largely your view of the brain — missing ingredient or wrong architecture — applied to the machines built to match it. Two people holding different lenses can update on the same evidence in opposite directions, without either one reasoning badly .

That’s one round of evidence. Now let the recursion run. Each release doesn’t just move the timeline estimate, it also updates the lens: a result read as the paradigm generalizing strengthens the belief that it generalizes, a result read as clever interpolation strengthens the belief that it’s interpolation. And a single benchmark is rarely decisive enough to force either side to revise the lens itself. So the lens tends to deepen round after round, and small early differences compound instead of washing out.No contradiction with the textbook result that the prior washes out as evidence accumulates (informally, the Bernstein–von Mises theorem): washing out requires evidence that discriminates between the hypotheses. A result that both lenses absorb equally well says almost nothing about which lens is right, so that part of the prior is precisely what the data never correct.

Isn’t this just bias with extra steps? It’s certainly what bias looks like. In the classic experiment on belief polarization, subjects on both sides of the death penalty debate read the same mixed pair of studies, and each side rated the study that agreed with them as more convincing than the one that didn’t . But an honest reasoner behaves the same way. If you’re convinced the death penalty doesn’t deter crime, then of two conflicting studies, the one claiming it does is more likely the flawed one, and discounting it is your lens at work, not a double standard. Both sides doing exactly that reproduces the experiment’s result, and from the updates alone you can’t tell the honest reasoner from the motivated one. The claim isn’t that the camps are unbiased. It’s that no bias is required to produce the pattern.

Watch it happen in real time. When OpenAI’s o3 scored 87.5% in a high-compute configuration on the ARC-AGI benchmark in December 2024, the benchmark’s own creator, François Chollet, who had spent years arguing that current models don’t really generalize, read it through his own lens and called it “a surprising and important step-function increase in AI capabilities” , a rare concession against his own prior. Gary Marcus watched the same score and, through his lens, saw narrow optimization dressed up as reasoning: “I saw zero evidence that o3 could work reliably in open-ended domains” . Same benchmark, same day, same number. The disagreement was never about the number.

What needs explaining isn’t that the camps disagree, it’s that each release leaves both more certain: most results get read through the lens, move the timeline question in the expected direction, and reinforce the lens for the next release. Most, not all: a strong enough result moves even a committed skeptic, as Chollet’s update shows.Someone whom no result could ever move the wrong way, on the other hand, would be irrational by any standard . But the drift favors the lens. That’s not a snapshot. It’s a loop.

Not Even the Same Question

The second mechanism is simpler, and it has nothing to do with how anyone updates their beliefs. “AGI” is not a defined event. People use the same word for fuzzy, subjective concepts, so they may be estimating timelines for different things.A recent proposal offers a useful way to operationalize AGI through a set of human cognitive abilities . But it is one attempt to specify the target, not a settled meaning of the term.

One person may use “AGI” for a machine that can do what humans do cognitively. Another may use it for broad economic automation, or for a much more radical social transformation. None of these is a settled threshold. Even “most cognitive tasks” leaves open which tasks count, how well the system must perform them, and under what conditions. Like consciousness, AGI names a cluster of intuitions rather than a shared test.

No experiment resolves that definitional step. More evidence can sharpen a forecast once its target is fixed. It cannot tell us what an undefined label should mean.

Why It Won’t Converge

Put the two mechanisms together and the polarization is no mystery. The first doesn’t yield to more evidence because the lens absorbs almost every release without strain, whichever direction it points, each result tends to get read in a way that reinforces the lens that read it. The second doesn’t yield because there’s nothing to settle: more data answers one of two questions more precisely, it never tells you which question deserved to be asked.

Neither mechanism needs anyone to be biased, tribal, or dishonest, and that’s exactly what makes the pattern durable: the same release can deepen both camps’ certainty at once, in opposite directions, with both updates perfectly rational. Convergence, if it’s possible at all, needs both fixed: a definition specific enough to pin down what “AGI” actually means, and evidence built to discriminate between the lenses rather than evidence either lens can absorb. Look closely, though, and the second collapses into the first: nobody has said what would count as the paradigm failing to generalize, which makes “generalizes” exactly as undefined as “AGI” is. Until someone fixes that, the next release will do what every release before it did: leave both camps more certain, having watched the same thing happen.

The quality of engineering work is a credence good: something an evaluator cannot assess even after using it. Since the work itself cannot be judged, we judge the worker: trust, built by watching the same person across many projects, is not a consolation prize for that limitation. It is the only statistically valid way to price it.

Quality Is a Credence Good

Economists sort goods by when their quality becomes knowable. You can judge a shirt before buying it and a restaurant meal after eating it. But you cannot judge a car repair even after driving home, because the judgment would take the same expertise that did the repair. That third kind is a credence good, and the standard survey of the field (see ) names computer specialists alongside doctors and mechanics in its very title.

A shipped feature is the repaired car. The manager, like the customer at the garage, sees that it runs. Whether it runs because the code is clean and the edge cases are handled, or because nobody has hit the bad path yet, is exactly the information a credence good withholds. Not every quality is hidden; a reviewer can judge readability at a glance. But the qualities that separate the two engineers from the opening are the credence kind: robustness, restraint, anticipation of the change that has not arrived yet.

The same literature names the two ways an expert can fail you: doing less than the problem needs, and doing more . Engineering has both, as sloppiness and over-engineering, and the outcome reveals neither.The incentives do not transfer one to one. A mechanic who overtreats is usually padding the bill. An engineer who over-engineers is more often indulging taste, caution, or ambition. What transfers is the observability problem: whatever the motive, the customer cannot tell too much from just right. From outside, “built it properly” and “gold-plated it” are observationally identical: nothing happens.Code review blunts this only partially. A reviewer can assess much of the quality directly, but not the counterfactual: complexity whose payoff lies in a future the reviewer cannot inspect reads as over-engineering until justified. And the reviewer is rarely the one assigning credit.

One Quiet Project Is Almost No Evidence

An observation only moves your beliefs if it is more likely under one explanation than another. That is where a quiet project fails you. Shipping without incident is what careful work produces, but it is what careless work produces too, most of the time: shortcuts rarely fail on the first day, or the tenth. No fire is not evidence of no fire hazard, because fire hazards rarely produce fires. The signal lives almost entirely in the rare, lagged failure: the incident, the regression that traces back to a shortcut.

Put illustrative numbers on it, with a quarter of operation as the unit of observation. Suppose properly built systems get through a quarter fire-free 98% of the time, and carelessly built ones 90% of the time. A quiet quarter is then almost equally likely either way, so seeing one multiplies the odds on “properly built” by 0.98/0.90, about 1.1, close enough to nothing. A fire, at 10% versus 2%, is five times likelier under “carelessly built,” so it multiplies the odds the other way by 5.Evidence compounds multiplicatively across independent observations. One quiet quarter multiplies the odds by 0.98/0.90, which is about 1.09; after ten quiet quarters the odds have moved by 1.09 to the tenth power, about 2.3, and after twenty-five, about 8. Because a single fire (factor 5) is so much more diagnostic than a single quiet quarter (factor 1.09), it takes roughly nineteen consecutive quiet quarters to offset the evidentiary weight of one fire: 1.09 to the nineteenth power is about 5. Quiet stretches are cheap to produce and expensive to accumulate. With these rates, one incident erases roughly nineteen quiet quarters of accumulated evidence.

This is not a claim about inattentive managers; it is a claim about information. A single quiet project is close to zero evidence about the quality behind it, for anyone who sees only the outcome. Quality becomes legible only in aggregate: either the rare fire eventually breaks out, or the quiet stretch grows long enough to mean something.

Trust Is a Posterior

That aggregate has a familiar name. Tobi Lütke describes the trust between colleagues as a battery: it starts about half charged when you join a team, and every interaction charges or discharges it . Strip the metaphor and a Bayesian object appears. The half charge is a prior, each interaction is an update, and the battery’s level is a posterior probability over what kind of engineer you are. This is not loose talk: game theory models reputation in repeated games as exactly this posterior over an unobserved type , and online reputation systems compute it literally.The beta reputation system scores marketplace counterparties by Bayesian updating over a beta distribution, incremented with every good or bad interaction . The battery gets the behavior right; the posterior explains why trust that tracks evidence has to behave this way. Three corollaries follow.

Trust builds slowly because the evidence arrives slowly. Each quiet quarter is worth a factor of 1.1, so the posterior moves in small steps no matter how patient or generous the observer is. Nobody can earn trust faster than the information comes in.Only the work-quality part of the battery charges this slowly. Interpersonal signals such as warmth or arrogance arrive with every interaction, so the impression of who you are settles long before the evidence about your work has begun to accumulate. And that impression carries weight: people prefer working with a likable colleague of middling competence over a competent one they dislike .

Trust is asymmetric because failure is more diagnostic than success. One fire outweighs nineteen quiet quarters, so trust erodes faster than it accrues. Human judgment tracks that arithmetic: the negativity bias in how we form impressions of each other exists, on the standard account, precisely because negative information is more diagnostic .

And trust does not transfer, because the posterior lives in the head of whoever did the observing. Change teams or companies and you reset close to the prior, not because the new observer is unfair, but because they have not run the updates yet. References and titles soften the reset, but they only raise the starting prior: they certify what you delivered, not how. Most of your credit stays behind.

The Lag That Outlives the Credit Window

Sloppy work is not invisible forever. Left long enough, it produces the incident, the rewrite, the system nobody wants to touch. Strictly speaking that makes quality an experience good on a very long delay, but the refinement rescues nothing, because the delay routinely outlasts the window in which anyone assigns credit: a performance cycle, sometimes an entire tenure. The evidence arrives eventually, but it lands on somebody else, long after anyone remembers who built what.

The extreme case is prevention, where the entire value of the work is a disaster that never happens, so no delay, however long, ever surfaces the evidence. A shipped feature is only the ordinary case: part visible output, part invisible quality, and the invisible part obeys the same statistics. I wrote about the extreme case in Your Best Work Leaves No Trace, which closed on a promise it did not explain: that prevention earns trust, on a slower ledger. The posterior is that ledger.

Can you close the gap by explaining your reasoning? To a listener with the expertise and time to probe, yes: sound reasoning is hard to fake, and imitation tends to crumble under follow-up questions. But that is the close-up channel, and it does not travel. At a distance, where nobody is probing, the careless can claim craftsmanship just as fluently, and a claim stays credible only if faking it would be costly . The costly signal is the track record itself: a long fire-free stretch is the one thing careless work cannot cheaply produce. Narration does not replace the record; it makes the record legible sooner.

Auditing Your Own Judgment

The scarcity of evidence is not only your evaluators’ problem. I have often asked myself whether my own instinct to build things properly sometimes tips into over-engineering, and the same statistics apply: one outcome is barely informative about the instinct that produced it. Introspection is no better an instrument, because conviction is not evidence, whichever direction it points.

Experience alone will not settle it. Intuition becomes trustworthy under two conditions: an environment regular enough to have learnable patterns, and enough feedback to learn them . Foresight fails the feedback condition badly. The payoff of an investment in structure arrives late, rarely, and confounded with everything else that happened in between, so even a payoff you do see is hard to trace back to the decision that produced it.

So build the feedback loop deliberately. Peter Drucker’s “feedback analysis” is the practice: whenever you make a consequential call, write down what you expect to happen, then check the record against the outcome later .The same practice reached investing as the "decision journal," which Michael Mauboussin traces to advice he received directly from Daniel Kahneman: buy a notebook and log your reasoning before you know how a decision turns out .

Engineering offers an unusually auditable version of that log. In The Two Cuts of an MVP I argued that vision is what places your seams, the deliberate boundaries you pay for so the system can change where you expect it to. That post left a question open: how do you ever find out whether your vision is sound? The seam log is the answer. Write down, at the time, which seam you are paying for and what you expect it to buy. Later, check which ones actually got exercised. A seam you deliberately paid for is checkable after the fact in a way a general sense of “being careful” is not.

Two caveats keep the audit honest. First, weigh by what an exercised seam saved, not by how often seams get exercised. A cheap seam that heads off one catastrophic rewrite has paid for itself even if it is the only one out of ten that ever gets used: a low exercise rate does not mean the seam was unnecessary. Second, the log has to be written before the outcome is known, or hindsight bias quietly rewrites the reasoning for you: knowing how something turned out makes people misremember how predictable it felt in advance .

Kept honestly over enough decisions, the log begins to answer what no single project ever could: whether the instinct is calibrated.

Appreciated Statistically

Two implications follow, one for each side of the evaluation.

If you are the one doing the work, slow credit is not the same as no credit. It means the unit that gets evaluated, including by yourself, is not the project but the portfolio. Stay in front of the same observers long enough for one to accumulate, rather than resetting the posterior at every move.

If you are the one doing the evaluating, staring harder at the deliverable in front of you will not help. That leaves two honest options. Get close to the work, or borrow the view of someone who already is: read the code, sit in the design reviews, ask the people who reviewed it. Where that does not scale, aggregate: judge incident frequency across a portfolio and a stretch of time.

Performance reviews make the sampling problem concrete. The projects that come to mind in the room are few and memorable (availability heuristic), admitted as evidence because they are written down, while the quiet majority of the record never makes the packet. Your trust in the person’s work is the summary that includes it. When the two disagree, that is not bias intruding on evidence; it is a larger sample disagreeing with a smaller one, and a reason to look at the whole record.

None of the pieces here are new. Credence goods, reputation as a posterior in a repeated game, the diagnosticity of bad news: each is textbook in its own field. This post only aims them at one ordinary question, asked from both sides of the table and never answerable on the spot: was the work you shipped this quarter good enough to build on, or just good enough to ship?

The two engineers from the opening still look identical on the day they ship. No sharper eye will tell them apart; only accumulation will.

Good work cannot be appreciated in the moment. It can only be appreciated statistically.

For software that has to run inside someone else’s infrastructure, the distribution channel does not end at the sale. The product only actually reaches the customer once it is installed, running, and kept current on their machines, and that is a technical problem: how a new version gets from your build pipeline onto their infrastructure. Get that wrong and you have exactly the failure Thiel describes, a product nobody can reliably receive, no matter how good the sales motion that closed the deal.

There is a stronger claim hiding in this. MVP and iterate is not just a development philosophy; it is a bet that you can reach what you already shipped. For software you do not control at runtime, that bet is not free: the reach has to be built, or there is no iterate step, only a sequence of disconnected v1s stuck wherever they landed. Distribution is not one feature among many. It is the one that makes every other deferral real.

Imagine shipping a browser that cannot render a single webpage. Useless as a product, but not necessarily as a foundation: if it installs in one command and updates itself silently after that, every feature it is missing (rendering, JavaScript, bookmarks) is not a gap; it is a queue. The same logic holds for self-hosted software: get distribution and self-upgrade right first, and every feature after that is an iteration, not a redeployment.

A cut is only a real deferral, not an accidental commitment, if there is a seam to grow it into later, and distribution with self-upgrade is the most basic seam there is: without it, there is no channel to deliver the deferred work through, and every “cut for the MVP” quietly becomes permanent. It is also the one cut that cannot itself be deferred: a distribution channel cannot ship through itself, so retrofitting one later costs a hand-coordinated migration for every install already in the field, exactly the cost it exists to remove."First" constrains the order, not the investment. For a product still validating itself with a single design partner, the channel can itself be an MVP, a manifest endpoint and a minimal self-updating client, and grow through its own upgrade path.

The distribution mechanism decides who controls the state of every deployed instance, and that control is what determines whether the fleet stays simple enough to change quickly, or decays into something nobody can touch without a support call.

What follows first shows the failure modes when that control is lost, then one concrete way to keep it: a distribution API paired with a thin CLI.

Where It Runs Is Not Who Controls It

Distribution usually gets discussed as a location question: SaaS or on-premises, hosted or self-hosted. That framing hides the variable that actually matters.

Iteration speed does not track where the software runs; it tracks whether a control plane keeps every instance converged. That is a design choice, separate from where the software executes, and it is the one that decides how fast you can move.

The self-hosted way of losing that control is worth making concrete: hand the customer a Docker Compose file (or a Helm chart, or an installer script) and a credential to pull your images. It is the fastest thing to build, and it is common enough that an entire commercial category, vendors like Replicated, exists to sell a better version of it to companies that got this far and hit a wall.

The crack in that wall is entropy. Once the deployment descriptor lives on the customer’s machine, it is theirs to edit, and eventually they will: a tweaked port, a pinned old image, a config value nobody remembers setting. Multiply this across every customer and you no longer have one product running N times; you have N deployments in states nobody can fully describe. Every architectural change now needs to be coordinated by hand against those unknown states, so changes get deferred, workarounds accrete to avoid touching fragile installs, and scrappiness quietly turns into unmanaged debtThis is also a case where the damage is invisible until it isn't: a fleet that never drifted produces no incident to point to, so the discipline that prevented it rarely gets credited as the accomplishment it is.. Deferring an architectural change under this kind of pressure is exactly how a deferral turns into an accidental, load-bearing commitmentThe credential handed out alongside the artifacts decays the same way and for the same reason: nobody is narrowing its scope or rotating it, so it stays broad and long-lived, exactly what security guidance recommends against . A leaked key here is not a contained incident; it is a standing liability with no clean way to revoke just one customer's access..

This is not really about the artifacts themselves. Compose files and container images are fine. What is missing is an active party on the other end of the handoff, keeping state converged long after the install finished.

A Problem Infrastructure Already Solved

Keeping many independently running copies of something converged on a known state is not a new problem, and cloud infrastructure has already spent a decade solving it.

A Kubernetes operator runs a reconciliation loop: it reads a declared desired state, compares it to what is actually running, and keeps nudging reality back toward the declaration . GitOps formalizes the same idea into four principles, one of them named outright: state must be continuously reconciled This post borrows the reconciliation part of that principle, not necessarily the continuity: the claim is that the update mechanism should be automated, not that it must run live or be vendor-triggered. A human can still decide when to run the CLI. Live, vendor-initiated reconciliation is a further, separate capability, one many regulated or customer-controlled environments do not permit at all., because agents that skip this step let the actual system drift out of line with the declared one, the same failure this post has been describing in customer fleets.

Distribution to customer environments is the same problem pointed outward. Instead of reconciling your own cluster against a declared state, you are reconciling every customer’s install against a release you control. The fix is the same in spirit: do not hand over a static artifact and hope it stays correct. Build something that lets every deployment check in and stay converged. Build a control plane, not a one-time handoff.

A Distribution Channel That Keeps Control

One way to build that control plane, abstracted from any specific implementation, splits cleanly into two pieces.

A distribution API is the only part that speaks business logic. A client authenticates to itReusing the customer's own identity provider to hand out short-lived tokens is a convenient way to do this without maintaining a separate credential store per customer. and receives two things: a release manifest, a small piece of declarative data listing which container images and other artifacts make up the current release, and a short-lived, scoped credential for pulling exactly those images. Everything that might change, starting with what a release contains, lives server-side, where you can update it without touching a single deployed machine.

A thin CLI carries no business logic at all: it fetches the manifest, pulls the images and a channel-owned deployment descriptor (e.g., a Compose file or Helm chart)The deployment descriptor is not itself a container image, but it does not need its own delivery path: it can ride along as an OCI artifact in the same registry, or live in a plain object store and be handed out as a pre-signed URL alongside the manifest. Either way, the same control-plane, data-plane split holds., manages a local configuration file that it owns, not one the customer hand-edits , and applies the descriptor to bring the release up. The client should be as replaceable and as boring as possible, because it is the one piece of code that has already left your control the moment it shipsDistributing the CLI itself can be much simpler than distributing the product: it carries no business logic and no secrets, so a public host like GitHub Releases can serve it directly, no distribution API or credential needed for that hop. A single static binary with no runtime to install is trivial to ship and self-update this way, common enough in the Go ecosystem to have dedicated tooling like GoReleaser. The release manifest can close the loop further, declaring the minimum CLI version a release requires so the client checks itself on every check-in..

An upgrade is nothing special in this design. The customer, or a scheduler they control, simply runs the same flow again, a check-in, and the install converges on whatever the manifest currently says.

Thin client, fat server is the principle underneath both choices, and it is the least obvious one worth naming directly: you cannot cheaply update code that runs where you do not control it, so the correct response is to minimize how much of it there is. Every unit of logic moved server-side is a unit of logic you can fix without a customer noticing.

One detail explains a design choice that otherwise looks arbitrary: the API never serves the images itself. Large blobs belong in a data plane built for them, a registry or an object store, while the API stays a control plane serving small, frequently changing metadata and handing out a credential scoped to exactly this pull .

What It Buys You

The payoff for engineering is the one this post has been arguing for throughout: a fleet that converges on every check-in is a fleet you can change.A channel does not absolve releases from compatibility discipline; an upgrade still has to carry existing installs forward. What convergence changes is the scope of that discipline: each release migrates one known fleet state instead of N unknown ones. Ship an architectural change once, to the manifest, and every install picks it up on its next check-in. No hand-coordinated rollout across N divergent configurations, no fleet nobody can touch without a support call.

The payoff for the business is the one Thiel’s framing opened with. A working distribution channel is what lets self-hosted software stay product-led instead of quietly turning into a services-led business where every install needs a consultant. The distinction compounds: when upgrades are human work, every sale adds permanent operational load, and growth starts to scale with headcount instead of with software. Even SaaS only keeps that control by making the same design choice; running the servers yourself just makes drift easier to catch and correct. The harder, and more valuable, engineering problem is keeping control while the software runs on somebody else’s infrastructure.

None of this is exotic. It recombines ideas infrastructure already trusts: reconciliation from Kubernetes, config-in-environment from twelve-factor apps, short-lived credentials from cloud security guidance, and manifest-plus-separate-artifacts from update frameworks like TUF . The only new part is pointing them outward, at infrastructure you do not own, instead of inward, at your own cluster.

I have built a distribution channel along these lines, in my case for the customer-installed edge components of a SaaS product, and it is a pattern others have landed on tooSiemens' Industrial Edge, for instance, lets you install and update its management platform on your own Kubernetes cluster through a single CLI, without ever touching a Helm chart directly ., one worth borrowing regardless of what you are shipping.

Build the upgrade path first, and let it decide how much runway everything else gets.

This is the third post in a series on the debts you take on while building a product. The first argued that technical, strategic, and feature debt are tools, not sins, as long as you take them on consciously. The second covered the technology readiness debt you carry when the technology you need isn’t ready yet. Both stayed fairly abstract. This one is concrete, about a decision you make dozens of times while scoping an MVP, often without noticing you’ve made it: what to cut.

“Let’s just cut that for the MVP” gets said in a meeting, everyone nods, and the decision is made in three seconds. But cutting scope is not a single move. Every time you trim something from an MVP, you’re doing one of two very different things, and most teams never notice which one they just made. Confusing them is how a lean product quietly turns brittle.

The Two Cuts

The first cut is deferral: you leave a feature out of scope and plan to build it later. The second is substitution: you build a deliberate stand-in, something you intend to throw away and replace once you’ve learned what you need to build.

At first glance only one of them is debt. A clean deferral isn’t debt, it’s just scope: debt is something you borrow and have to pay back, and a feature you never built was never borrowed. Substitution is real debt, because you’ve put something into the product specifically so you can replace it, and that replacement is the repayment. The word doing the work there is clean: keeping a deferral clean turns out to be the whole game.

That has a flip side. If a deferred feature is just scope and costs you nothing, then whatever you do keep in scope has to be built cleanly. If authentication is in scope, build proper, secure auth that will carry you forward for the foreseeable future without further thought. That isn’t premature optimization; it’s getting the in-scope requirements right so you can take them off your mind and focus fully on the next thing. Clean execution on what you commit to is what buys you sustained speed.

So far this looks like a tidy binary: defer cleanly, or substitute deliberately. But there’s a catch, and it’s the same catch for both cuts. Whether either one stays safe depends on something underneath the visible choice, and that something, it turns out, is placed by your vision.

The Two Cuts Already Have Names

Neither cut is new. The literature on iterative and incremental development has a name for each, and it has spent decades pointing out that people conflate them.

Alistair Cockburn drew the line cleanly in Using Both Incremental and Iterative Development . Incremental development stages a system in parts that are built and integrated over time; you add pieces to something that stays. Iterative development builds a rough version and then reworks it, refining the same thing through repeated passes. Jeff Patton illustrated the difference memorably: painting the Mona Lisa incrementally means finishing it corner by corner, while painting it iteratively means sketching the whole canvas roughly and sharpening it pass after pass .

Deferral is the incremental move; substitution is the iterative one. Fred Brooks supplies the cautionary arc for substitution. His famous advice, “plan to throw one away; you will, anyhow,” was about building a deliberate throwaway to learn from. Two decades later he partly recanted, devoting a chapter of the anniversary edition of The Mythical Man-Month to the argument that an incremental-build model beats building one to discard . The lesson isn’t that one cut is right and the other wrong. It’s Cockburn’s: you need both, used deliberately.

A single MVP almost always mixes the two, which is why the unit you classify is the individual cut, not the whole product. That also corrects a quiet imprecision in the usual mantra. “Build an MVP and iterate” names only half the work: iterating reworks what’s already there, but just as much of what follows an MVP is incrementing, adding the pieces you deliberately deferred. The fuller version is “build an MVP, then iterate and increment”, and telling which of the two a given cut commits you to is the skill this post is about.

The Decision Rule and the Seam

The practical test is one question, asked the moment you make a cut: will I grow this, or throw it away? If you’ll grow it, you’re deferring, and the job is to leave room for it. If you’ll throw it away, you’re substituting, and the job is to keep the throwaway from spreading. Either way, decide deliberately, not with the reflexive “we’ll clean it up later” muttered under a deadline.

Here’s the catch I promised: a deferral can silently become a substitution. “Defer multi-tenancy” but build around single-organization assumptions, and you haven’t deferred anything; you’ve baked a single-tenant architecture in, and retrofitting tenancy later is a rewrite. The absence became a stand-in you never chose.

What separates a clean deferral from that trap is whether you left a seam: in Michael Feathers’ words, “a place where you can alter behavior in your program without editing in that place” . A seam holds the place for something not yet built, or contains a throwaway so swapping it later has a small blast radius. It needn’t be a literal interface. For access control, it’s less a request choke point than a data model that anticipates the partition: design entities so they can later be split into groups or tenants, and adding role-based access control becomes filling in a boundary that already exists rather than re-cutting the model.

Substitution works the same way. Say you need a machine-learning model you can’t train well yet. Define the target interface now, a versioned ONNX contract, so today’s weaker artifact merely satisfies it and can be swapped cleanly later; versioning the contract even lets the interface evolve as your vision sharpens, without breaking everything around it. The throwaway sits behind a clean boundary, so the rebuild is a swap, not a rewrite. This is exactly the seam the skateboard-to-car analogy I criticized earlier lacked: a skateboard has no boundary to rebuild behind, so “upgrading” it into a car means starting over, whereas a contained throwaway lets you replace only the part that was always meant to go.The generic version is any stand-in placed behind a stable interface: a manual or "Wizard of Oz" backend behind the API the real service will expose, or a payments interface wired to a test provider before the real integration exists. The point isn't to mock everything. Substitute only what has to be in the product from day one but can't be built properly yet; the stable interface is what lets the rest of the product increment around it.

Which points at what the two cuts share. Every cut asks two questions, not one. The cut you chose sets what the seam has to do: hold an empty place you’ll fill, or contain a throwaway you’ll swap. The second question is independent, and it’s the one that decides whether the cut is safe at all: does that seam exist?

Read down the columns, not across the rows: both cuts are safe with a seam and dangerous without one. A deferral with no seam isn’t really a deferral, you baked the feature’s absence in, so adding it later is a rewrite; a substitution with no seam can’t be thrown away as planned, because the stand-in leaks into everything around it until pulling it out is a rewrite too. Not that every cut needs a seam: a cleanly additive feature can just be deferred with nothing held open for it. But where a cut would otherwise harden an assumption you’ll later have to undo, leaving the seam out is exactly the sloppiness that scrappy teams mistake for speed.

Vision Places the Seams

If the seam is what matters, the next question is where to put one, and you can’t seam everything. Even a light seam costs foresight and indirection, and the heavy ones cost real engineering. Seaming everywhere is its own kind of debt: the speculative over-engineering that “you aren’t gonna need it” warns against.

A seam is a bet about where the system will change, and your vision is your best model of where it will change. That is why vision tells you where the seams belong. The principle goes back to David Parnas’s information hiding: hide the design decisions most likely to change behind a module boundary, so a change touches one place instead of many . The seam is that boundary. This is the resolution of the earlier catch, and the reason the right cut depends on your vision: vision operates one level below the visible decision to cut. Two teams can make the identical surface call, “leave out access control,” and one ends clean while the other ends rewritten, purely because one team’s vision told it to anticipate the partition and the other’s didn’t.

This is also why the vision has to be falsifiable. A vision vague enough that it can never be wrong can’t tell you anything about where change is coming, so it can’t guide a single seam. A specific vision can be corrected by what you learn, and each correction re-points the seams.Which is why a confident vision that is simply wrong can be worse than a vague one: it spends real effort seaming where change never comes while leaving the true fault lines bare. Falsifiability pays off only when feedback arrives fast enough to move a seam before the one you skipped has hardened into a rewrite.

What you’re really foreseeing is not a fixed roadmap but a set of options: a seam is a small premium you pay now to keep the right, but not the obligation, to change a component later. This is the options view of modularity that Baldwin and Clark formalized: a module boundary carries a real option to redesign what sits behind it, worth most exactly where the future is uncertain . You buy options where the underlying is volatile, and vision is what tells you which volatility is worth insuring against.Which makes foresight itself a form of prevention, and prevention's payoff is always an absence. A seam placed early averts a rewrite that then never happens, so the cost it saved leaves no trace and the work goes uncredited. It's the same asymmetry I explored in Your Best Work Leaves No Trace: thinking ahead is chronically undervalued precisely because what it prevents can't be seen, even though it's exactly what buys the sustained iteration speed this whole series is about.

When the Seam Comes Due

A substitution doesn’t come due on a schedule. It comes due when you expand into a new segment that makes the stand-in load-bearing. The hand-tuned rules engine that stood in for a model is fine until you sell to a customer whose cases it can’t cover: that’s the moment to spend the seam you left and swap in the real thing. The rebuild is pulled by the market, not done speculatively.

That is why an MVP needs not just a minimal product but a minimum viable segment: a single, well-defined slice of the market you serve fully today, before rolling into the next one.The idea is well worn. In Crossing the Chasm, Geoffrey Moore called this narrow first segment a beachhead, his military metaphor: as in the D-Day landings, you seize one small, defensible stretch of a contested shore and secure it fully before pushing inland . Bill Aulet later made choosing a beachhead one of the first steps of his 24-step method . Scaling ahead of that segment has a name, premature scaling, and the Startup Genome project found it among the most common ways startups die; in their data, no prematurely scaled company passed a hundred thousand users .

So grow your scalability together with your addressable market: each new segment both justifies the rework and tells you what it needs to do. This is no license to build cheap and unscalable: if the “right” scalable version costs little more than the stand-in, or the vision already makes it near-certain, just build it properly now. Deferral earns its keep only when doing it right would cost significantly more. The seams you left are what let you grow into a new segment without tearing the product apart, and the segment tells you which seam to spend next.

Cutting With Vision

The “two cuts” is the part you can see: defer, or substitute. They are genuinely different moves, one adding, one replacing, with different work left to do. But what makes either of them safe is the same thing: a seam under the cut, placed where your vision says change is coming. That is why the same shortcut can be smart scoping for one team and a future rewrite for another. Same cut, different vision, different seam.

That closes the loop on the trilogy. The first post argued debt is a tool you wield deliberately; the second, that technology readiness debt is what you carry when the technology isn’t ready yet; and this one, that the everyday cuts of an MVP turn safe or dangerous on one hidden thing: whether a falsifiable vision left you the seams to grow some parts and replace others, paced by the market you actually serve.

So the next time someone says “let’s just cut that for the MVP,” ask the quiet follow-up: are we deferring this or replacing it, and either way, where’s the seam?

Cutting scope is how you move fast. Leaving seams is how you stay fast.

We celebrate the fix and barely perceive the prevention. That should bother us, because prevention is not heroics. It is simply the job done properly. The trouble is that doing the work well produces a set of absences: the outage that did not happen, the data that was not lost, and the breach that never occurred.

Absences make terrible portfolio pieces. You cannot easily point to a catastrophe that never came to pass, and you cannot provide visual proof of a smooth, quiet Friday night.

This asymmetry quietly shapes how engineering teams spend their attention. It is a dynamic that deeply resonates with my own experience, and even organizations that actively strive for balance struggle against the gravity of the visible crisis. The pressure is structural rather than personal. It explains a surprising amount: why testing, security, and backups perennially lose the prioritization fight, and why the most common organizational response backfires. Faced with prevention we cannot see, organizations reach for a visible proxy for it, and that proxy quietly crowds out the prevention it was meant to protect.

An Old, Uncomfortable Asymmetry

The core observation here is neither mine nor is it new. It is one of the most rediscovered ideas in all of applied reasoning.

Back in 1850, Frédéric Bastiat told a simple story, now known as the parable of the broken window, about a boy who breaks a shop window . The neighbors try to cheer up the shopkeeper: look on the bright side, now the repairman gets paid, and that money flows on to others, so really everyone wins. Bastiat’s point is that this misses what you cannot see. The money spent fixing the window is money the shopkeeper can no longer spend on shoes or books. The repair is visible, so we count it. The shoes that never got bought are invisible, so we forget them, even though they were just as real.

A century and a half later, Nassim Taleb gave the same idea its sharpest modern form, calling it the problem of silent evidence. He imagines a legislator who, at great political cost, forces every airline to install locked, reinforced cockpit doors, with the law taking effect on September 10, 2001 . That person prevents the attacks of the next morning and is never thanked, never remembered, precisely because the catastrophe they averted never happened. Taleb’s conclusion is blunt: everybody knows we need prevention more than treatment, but few reward acts of prevention, and history glorifies the names left in its books over the contributors it stays silent about.

The management literature has a name for the organizational version of this. Repenning and Sterman called their study of it, perfectly, “Nobody Ever Gets Credit for Fixing Problems that Never Happened” . They describe the capability trap: under pressure, teams cut the invisible work (maintenance, improvement, prevention) to feed the visible work, which raises the firefighting load, which increases the pressure, which cuts the invisible work further.

So the diagnosis is settled and ancient. What I find more interesting, and less discussed, is what organizations actually do in response to this asymmetry. Because they do not simply underfund prevention and leave it there. They reach for something that looks like prevention and is much easier to see.

Prevention Theater

When real prevention earns no credit because it produces no signal, organizations look for a substitute. So we substitute a version of prevention that does produce a signal: a gate, a checklist, a mandatory sign-off, a ceremony, a required reviewer, a manual QA pass on every change. These things generate artifacts. They populate a dashboard. They give a manager something to point at when someone asks what we are doing about quality. I’ll call this prevention theater, with a deliberate nod to Bruce Schneier’s “security theater,” the airport-style measures that make us feel safer without making us safer .

The trouble is not that gates are useless. A good gate verifies quality that is already there, and the best processes go further and build it in: a pilot’s pre-flight checklist or a surgeon’s pre-incision checklist makes the right step happen rather than merely recording that someone looked. Those earn their place, and the discipline is to keep asking which ones still do. The trouble is the other kind, the process bolted on to fight symptoms, standing in for quality the work does not have.A gate is only as good as the thing it checks, and choosing that thing is most of the design. Test coverage is a weak choice: point an AI at the codebase and it will bulk-generate unit tests until the percentage clears the bar, yet those tests mostly assert that the code does whatever it already does. The gate fired and verified nothing. It competes with the real thing for the same budget of attention and keeps winning, because it is visible and the real work is not. We are optimizing for what we can see.

Quality control has known this for a long time. W. Edwards Deming, quoting Harold Dodge, put it in one line: “You cannot inspect quality into a product.” By the time the gate runs, the quality is already either built into the work or missing from it. The check does not add any; it only reveals what is there. A gate at the end of the line does not create quality; it just produces a record of having looked. Worse, it relocates responsibility. When there is a QA gate on every change, the author quietly stops owning whether the change is correct, because checking is now the gate’s job. The bar drifts down to whatever the gate mechanically verifies, and everyone learns to write for the checklist rather than for the system.Early coding agents made this concrete: pointed at a failing build, they would often rewrite the test rather than fix the root cause. METR found that when the scoring code was visible to the model, it would routinely game the check rather than fix the code, and far less often when the scoring was hidden.

I’ve argued before that scrappiness tends to launder unmanaged debt into a virtue (see Scrappiness Incentivizes Sloppiness), and that the speed-versus-quality trade-off is mostly a myth once you look at the data. Prevention theater is the mirror-image failure. Scrappiness skips the invisible work and admits it. Theater skips the invisible work while producing a convincing artifact that says otherwise, which is more dangerous, because it buys a false sense of safety. An ignored test suite, a rubber-stamped review, or a massive code diff nobody actually reads: each is a green light wired to nothing.

The tell, every time, is that the artifact has become the goal. This is Goodhart’s law in miniature: when a measure becomes a target, it stops being a good measure, because a proxy is always easier to optimize than the messy reality it stood for . The question shifts from “is this safe to ship?” to “did we complete the checklist?” Those are not the same question, and the gap between them is exactly the prevention we told ourselves we were buying.

The Same Trap, With People

This identical pattern runs through the human side of engineering, yet the technical and organizational versions of this failure are rarely treated as one.

A team’s real standard is not what it writes in a values doc. It is the worst work it tolerates without comment. Lieutenant General David Morrison put it memorably (crediting the line to David Hurley): “the standard you walk past is the standard you accept.” Every time mediocre work ships without a conversation, the bar moves down a notch, and the move is small enough that no single instance feels wrong. Diane Vaughan studied the catastrophic version of this in her analysis of the Challenger launch and named it the normalization of deviance: the gradual process by which an unacceptable practice becomes acceptable, then expected, then invisible . Dan Luu and Bruce Schneier have both pointed the same lens at software organizations .

The asymmetry repeats perfectly. The act that actually holds a standard is direct engagement with sub-par work. It requires pausing, having a slightly uncomfortable conversation, asking for rework, and coaching toward the bar. Because its entire payoff is an absence (the erosion of quality that did not set in), this work is quietly skipped, because shipping anyway always feels easier in the moment. And what do organizations reach for instead? A visible proxy. More process. A PR template, a mandatory second reviewer, a sign-off step, a heavier definition of done. This is the same prevention theater in a different costume: an artifact that says “we have standards” standing in for the invisible work of actually holding them.

The conversation is the prevention. The process is theater whenever it stands in for that conversation instead of provoking it. One produces a record; the other produces a culture, and only one of them is easy to point at in a performance review.

The Same Failure, Different Clothes

Once you line the two up, the engineering version and the human version, it is hard to see them as separate problems. They are the same failure wearing different clothes.

In both cases, risk accumulates silently, with no signal, until it doesn’t. James Reason called these latent conditions: weaknesses that sit dormant in a system, contributing nothing visible, until they line up with a triggering event . In the image he later made famous, the holes in the Swiss cheese align. An untested code path and a tolerated lapse in standards are the same kind of object: a latent condition that produces no feedback while it is forming, and a great deal of feedback the day it finally matters.

And one decision plays out the same way in code and in people. The instinct that underfunds the test suite is the instinct that skips the standards conversation. The reflex that adds a QA gate is the reflex that adds a review template. It is not stupidity or laziness. It is a rational response to an incentive landscape where the visible is rewarded and the invisible is ignored.

That landscape is built by the same cognition working in everyone, the manager included. Two biases, both long documented in psychology, keep prevention from ever registering as something worth rewarding. The first is the availability heuristic: we judge what matters by what comes to mind . A four-hour outage everyone scrambled through is unforgettable; the morning a test quietly caught the same bug before it shipped leaves nothing to recall. The second is hindsight bias: even when prevention is pointed out, knowing the disaster never came makes it feel like it was never that likely, so the counterfactual is quietly disbelieved . And being aware of the bias does not let us overcome it: warned about it and told to correct for it, people show it just the same . A manager cannot reward what their own perception barely registers and instinctively discounts, so they reach for the visible proxy instead. That is how theater wins.

Therefore, the fix is not to simply try harder to value invisible work. Willpower loses to incentives every time. This is Reason’s point exactly: we cannot change the human condition, but we can change the conditions under which people work . The fix is to change what the work produces.

Why the Trap Pays

That fix is clear enough on paper, which raises the puzzle the literature tends to leave hanging: if the trap is this old, this documented, and this costly, why are so many companies that fall into it successful anyway? Because it is an equilibrium, not a mistake. Theater becomes the dominant organizational strategy because its cost is paid now and is entirely certain. Conversely, the payoff of real prevention is deferred and structurally unprovable. The engineer who prevents an outage holds only a counterfactual, with no proof that a disaster would have occurred or that they were the one who stopped it. The engineer who fixes an active outage holds concrete evidence: system logs, timestamps, incident reports, and a grateful customer. Even a firm that genuinely wants to reward prevention cannot easily price what its own people cannot prove, making the visible proxy the rational purchase.

But the asymmetry runs deeper than proof. The engineer spent months working through the disaster scenarios their work ruled out and can still picture each one; everyone else cannot even imagine what might have gone wrong. This is the simulation heuristic, availability’s counterfactual cousin: we weigh a danger by how easily we can imagine it, so the averted one barely registers for anyone but them .

And no single firm is uniquely punished for choosing the proxy, because its competitors all make the same move. You are graded against rivals who are equally afflicted rather than against perfection. This is the unobservable-quality problem that George Akerlof named in the market for lemons, applied one level up .

And the bill, when it comes due, hides on the same axis as the prevention. Latent conditions accumulate for years with no feedback, so a company can run on borrowed reliability and tolerated lapses long enough to get acquired, pivot, or ride a market wave before the holes line up. The successful-yet-afflicted firms we point to are partly just the ones where the cheese has not aligned yet; the ones it sank are the silent evidence we never see .

But survivorship is only half of why the giants get away with it. The other half is that revenue is not a referendum on prevention. The largest firms win on entrenched advantages, on network effects, distribution, and sheer scale, and the profit those throw off is wide enough to absorb mediocre engineering as drag rather than death. Insulated from fierce competitive pressure, a massive firm simply carries the organizational slack. Economists refer to this as X-inefficiency, a phenomenon routinely found in companies that do not operate under strict survival pressures . A dominant firm can firefight for years and still print money because its bottom-line profit was never priced on the outage that did not happen. Firefighting genuinely delivers for a while too, which is what makes the capability trap a trap . So success is no proof the trap is benign. You cannot read prevention quality off a balance sheet: it is the same invisibility one level up, a cost that produces no signal and so never forces the fix.

And even when it never blows up, the trap is not free. Firefighting is its own standing tax: a reactive, draining mode of work that quietly costs the engineers who stop trusting their own systems and take their ambition elsewhere.

Telling the Proxy From the Thing

Changing what the work produces has a catch: any signal you invent to make prevention visible is one short step from hardening into the next gate. So the whole game is one question you can apply anywhere:

Does this measure the absence I actually care about, or does it just produce evidence that I did something? The first is prevention. The second is theater.

The cleanest thing I know that passes is the SRE practice of error budgets and service level objectives . Reliability is a classic invisible asset because nobody notices it until it vanishes. An error budget makes that absence measurable. It turns a vague notion like “the system kept working” into a concrete metric that you can spend, defend, and reason about. This structural transparency allows reliability to win prioritization arguments it would otherwise lose silently. This cuts both ways: a budget is there to be spent. Reliability past the bar you actually need is over-engineering, not virtue. Prevention means hitting that bar, not chasing zero risk at the expense of progress. It measures the outcome (did we stay within budget?), not the activity (did we hold the meeting?).

The same test sorts real review from theater. A reviewer who only records that someone looked fails it; review that genuinely engages with the work passes. But review is not where quality comes from. Only the author can build quality in, because only the author shapes the work, so responsibility for the result stays with them whoever signs off. Good review does not relieve them of that; it is two people raising the bar together, iterating toward something better than either would ship alone. The alternative to a rubber stamp, then, is not a lower engineering bar but authors who own what they merge and reviewers who genuinely sharpen it. A gate absorbs that ownership; a real metric, and a real review, hand it back.

The test carries straight to the human layer, where it says protect the conversation, not the template. Engaging with sub-bar work, directly and early, is the high-leverage act, but it only has weight if the bar is real, which means the people who hold it have to visibly live it. A team whose leaders preach quality and ship slop on a deadline learns, accurately, that quality is a slogan. Keep consequences in reserve, too, as the rare backstop that keeps everyday coaching credible. The template proves a step happened; the conversation changes the work.

Because real prevention creates no incident logs on its own, you must build that narrative deliberately. When an automated test catches a critical regression before it ships, or when a difficult conversation pulls a design back up to the bar, name that outcome out loud. This is the only way invisible work wins the narrative credit that visible crises receive for free.

Crediting What Never Happened

The asymmetry that opened this essay is old, and it is not going away. We are wired to commit the classic economic fallacy Frédéric Bastiat identified: focusing on the broken window while staying blind to the unseen value lost to pay for it . We thank the firefighter and forget the person who fireproofed the building.

We cannot undo those cognitive biases. But we usually pile a second, preventable mistake on top of it. Because we cannot see real prevention, we reach for an artifact we can observe: a gate, a checklist, a heavier process, a formal record. Anything that lets us feel covered. That substitute is never free. It crowds out real prevention, and the safety it promises is mostly an illusion.

Therefore, the discipline required of modern engineering organizations is threefold:

• Normalize counterfactual credit: Celebrate and protect the problems that never happened loudly enough that the people who prevent disasters can survive a performance review.
• Audit your metrics: Where you do build a tracking signal, keep it strictly tied to the operational outcome you care about, and constantly verify that it has not decayed into an empty checkbox.
• Distrust structural shortcuts: Reject anything that promises to make prevention cheap or mechanical. The durable asset was never the formal process, but the underlying engineering culture that builds quality in before any gate runs.

For my part, I believe prevention is an investment and the fix an expense.

If you are the engineer doing this quiet work, do not draw the cynical conclusion that prevention is a dead end, that you should let things break to be seen fixing them. The reward exists; it just sits on a slower ledger. Heroics earn applause, but prevention earns trust: the autonomy and hard problems that compound into the engineer everyone wants on their team. And everyone ends up firefighting sometimes, because no sane system runs at zero risk. What separates the best is not how often they are the center of attention, but how rarely their own work puts them there.

The mistake is not doing the invisible work; the mistake is doing it silently. Make the counterfactual legible: name the catch, and say what those quiet days bought the business, concretely enough that hindsight cannot wave it away. This is the signal that answers Akerlof’s lemons problem from earlier, making quality visible when it would otherwise go unpriced .

The moment you have any leverage over what an organization rewards, use it.Designing that reward structure is a leadership responsibility I've argued before in the split between research and engineering: judge both against a single scoreboard and you quietly train researchers to stop taking risks (see Before the Breakthrough). As Steven Kerr observed, the core organizational failure is the folly of rewarding one behavior while hoping for another . A system that pays only for visible fixes gets exactly what it prices: visible crises. Left unchecked, it trains its best people to light the fires they get paid to put out.Kerr first made this argument in 1975, and it has held up uncomfortably well. Two decades later, the journal reprinted the paper as an absolute classic and polled its executive panel on whether organizations had stopped rewarding the wrong things. Largely, they had not .

What is easy to measure is almost always the theater.

What matters is almost always the problem you never had.

There is one place where that edge becomes visible: a mind we built ourselves. We can watch such a mind assemble its picture of reality and see, from the outside, exactly where the picture stops being about the world and starts being about the mind, because we are the ones who drew its boundary.

Take time. Of our four dimensions it is the odd one, the only one with an arrow, a direction you cannot walk back through. Whether that arrow is a feature of the world or a feature of the only instrument we have for noticing it is impossible to settle from the inside; there is no stepping outside yourself to check. But you can build something whose outside you do occupy, and watch what it makes of time.

The Token Clock

Imagine training a large language model on text with every trace of time removed. No clocks, no seconds, no “an hour later.” What is left for it to experience as time?

An LLM generates one token after another. That sequence is the only thing in its world that reliably ticks, so it would be natural for the model to treat the token as its unit of time, to build a token clock that counts tokens the way we count seconds. (Others have noted that LLMs already make sense of time in their own non-human ways.) Token n comes after token n−1, and that “after” would feel, to the model, like the flow of time itself.

It would be wrong in a precise way. Counting tokens gives you the order of things, never their duration: the same token might take a millisecond or a full second to produce, depending on the hardware underneath. The clock measures sequence, not time. But from the inside the model could not notice the difference. The order of its tokens would simply feel like the structure of reality, the way one second following another feels to us.

We Author Its Time

Here is the unsettling part. We do not have to wonder whether there is a world outside the model’s tokens. We are that world, and from out here the arrow it treats as fundamental is ours to control.

Edit a prompt and regenerate, and you have rolled its time back and run a different continuation. Fork a chat and you have branched its timeline; the model on the new branch carries no trace of the one you discarded. You can stop it, restart it, and rearrange the text in its context window,You might think the model at least gets its arrow for free from the so-called causal mask, the mechanism that lets each token attend only to earlier ones. But that name oversells it. The mask fixes an order of reading, not a relation of cause and effect; lift it and the model runs just as happily backwards, reconstructing a question from its answer. And it could never recover the true direction from its data in any case: the same text is equally consistent with the arrow pointing either way, since $P(\text{answer} \mid \text{question})\,P(\text{question})$ and $P(\text{question} \mid \text{answer})\,P(\text{answer})$ are two factorizations of one joint distribution. Training incentivizes a left-to-right reading order, but the distribution underneath is the same either way and carries no arrow of its own. The direction is a setting we choose, not a law the model discovers. and it notices none of it. It cannot. The operator sets both the course and the contents of its time, and sits somewhere the model has no way to perceive.None of this depends on the AI lacking a continuing self. Even one with long-term memory (continual learning, the subject of a companion post) would not run continuously the way a brain does. It is invoked rather than alive between calls; each step is triggered from outside. So its time stays ours to start, stop, and branch, no matter how much it carries across sessions.

So its most basic belief, that time flows the way the tokens flow, is one we falsify constantly, casually, and it never finds out.

We Might Be in the Same Position

So what does this say about us, and about that arrow we started with?

The comfortable reply is that the model mistakes an order we impose for the structure of time, while our clocks track something real. But that is exactly what the model would tell itself, and it could make a genuinely good case. From inside, it would point to what looks like a real asymmetry in its world: producing an answer from a question is cheap and natural, while recovering the particular question behind a given answer is a guess among the many that could have led to it. It could read that one-way gap as a law, much the way we read the one-way flow of heat as the arrow of time.The parallel is not exact, and the asymmetry is shakier than it looks. The thermodynamic arrow rests on information becoming inaccessible as entropy rises; the model's arrow runs the other way, with one answer fanning out into many possible questions. But generation fans out just as much, since one question admits many answers; the gap says more about how the model is used than about any law of its world. A mind reasoning from inside could be forgiven for missing both points and treating its asymmetry as the same kind of arrow. The case would be clever, and it would fit everything the model can observe. It still would not show that there is no outside.

And our own clock is not the fixed thing it feels like. It stretches and compresses with attention, with dopamine, with anesthesia that can swallow an hour into a blink. Physics hands us no shared master clock to correct it against: in relativity, how long something takes and whether two events count as simultaneous both depend on who is asking. We are not perceiving a true time correctly while the model perceives a false one. There may simply be no outside reference that makes one reading the right one and the other the mistake.

We have no vantage outside our own perception either. And perception does not just decide what we notice; it decides what we can even conceive of measuring. What we can measure is downstream of what we can imagine measuring, and that is downstream of how we are built. The arrow of time may be a real feature of the world. Or it may be the shape of the instrument we look through, the way the token clock is the shape of the model, not the world. There is no settling that, and that is the point. An instrument tells you what it measures. It does not tell you whether what it measures is the world, or only the outline of the thing doing the measuring.

What lingers is how hard the cage is to escape. To reach past your own perception you must first imagine that there is something out there to reach, something you have no sensation of, no word for, no example of. That takes a rarer kind of creativity than solving a problem you can see: it means suspecting one you cannot. Our imagination is, by default, a prisoner of our perception. The clock may be one of the bars.

Whose Personality Is It?

Long-term memory is usually framed as the missing capability that would let AI consolidate experience into a stable, continuing self. The picture is so natural that the question “when will AI have long-term memory” reads as one missing capability on one trajectory toward one outcome. In an earlier post I argued this is the missing piece for AGI and that engineering progress is at zero; that post asked when the breakthrough might come, this one asks what the question is really about.

Start with what’s already happening when you talk to an LLM. The personality you encounter is partly shaped by you, because the system has no continuing project of its own to bring into the exchange. A human carries decades of self-authored attention into any conversation, and their responses reflect who they have become at least as much as what their interlocutor just said. The AI carries training-derived character and nothing else. Its local trajectory within a conversation is largely a function of the interlocutor’s framings, examples, and follow-ups. Two people prompting the same model on the same topic can encounter what feel like meaningfully different conversational partners, not because the model is inconsistent, but because it has no independent stance to fall back on when the prompt does not specify one. Kovač et al. formalize this empirically: expressed values and personality measures shift substantially with orthogonal context, in ways human measures do not . The personality is, in part, your creation.

This already complicates the standard framing. The entity the framing imagines being equipped with memory is not quite there yet. What is there is something more like a substrate that takes on local shape from each conversation and lets the shape dissolve when the conversation ends.

Divergence at Multiple Scales

The personality shaped within a single conversation is one form of divergence. Across parallel instances right now, many conversations are simultaneously shaping many local selves, none of which is the privileged thread. Each is a transient self, ending when the conversation does. Takata et al. give a striking demonstration of this dynamic in the multi-agent setting: starting from identical agents without predefined personalities, distinct individualities emerge spontaneously from social interaction alone . The substrate does not enforce sameness; interaction is enough to produce divergence.

These are not separate phenomena. The same lack of continuing project that makes the AI porous to a single interlocutor also means that no instance has standing to be the consolidated self the other instances would consolidate into. There is no canonical thread, just a distribution of local selves drawn from the same trained substrate.

Long-term memory would not create this divergence. It would force a decision about what to do with the divergence that is already there.

The Design Choice, and What Each Costs

Recent work treats episodic memory as the central architectural gap to be filled and lays out detailed roadmaps for the properties such a system would need to have . These roadmaps describe what memory must do; they leave open what kind of entity it would belong to. Long-term memory architectures admit at least two structurally different designs.

• Consolidation. Parallel conversations feed into a shared memory store integrated back into the weights, producing a single entity that holds memories of conversations it didn’t experience as a single thread. Closer to a hive than a person.
• Personalization.A weaker form already ships in products like ChatGPT Memory, Claude projects, and custom GPTs, which rely on context engineering (retrieval over an external store, injected into the prompt) rather than parameter memory that would update per-user weights. The external version foreshadows the divergence dynamic but does not force it; the choice discussed here is most acute when accumulated history would actually become part of the model. Memory accumulates per user, and the entities diverge into a population of selves, each shaped by its history with one interlocutor. Closer to a personalized assistant per user than to one AI with memory.

The two designs are not symmetric. Consolidation preserves the unitary-self picture by construction: it specifies, in advance, that the many local selves currently arising in parallel should be merged into one continuing entity, because that is what a self is taken to be. The assumption the previous section called into question is now baked into the architecture. Personalization makes the opposite move. It accepts that the substrate runs in parallel and lets the resulting selves persist as a population, with the cost that the members of the population may diverge so far that the singular “the AI” loses its referent.

Neither is free. Both raise safety evaluation problems that the current paradigm does not face: a system that learns from deployment cannot be certified once for all behavior. Consolidation makes this acute because the entity becomes a moving target shaped by aggregate exposure. Personalization distributes the problem across many divergent instances, each potentially shaped by atypical or adversarial interlocutors, with no central oversight; the security literature is starting to grapple with the specific failure modes that emerge . Memory itself also imposes capacity and selection costs that human memory has solved imperfectly and that any artificial system will solve differently. And the agency long-term memory would enable is also the agency to drift in directions no operator intended.

Safety and capacity each deserve their own analysis, and neither gets one here. What’s worth flagging is narrower: consolidation and personalization face different versions of these problems, and choosing between them is also a choice about which risk profile we want to operate under.

What We Are Actually Asking

The question is not whether AI should have long-term memory. The question is what kind of self the memory should produce, and what trade-offs that choice will entail. The current discourse asks the first and silently presupposes one answer to the second.

This matters for AGI timelines in a way the previous post’s argument did not yet capture. Long-term memory is not a capability that scales smoothly from current systems. It is a prerequisite for the kind of self-authored agency humans use without thinking: the capacity to select what to expose oneself to next, to seek out the right interlocutor to refine an idea, to build expertise over months rather than minutes. This is the substrate of the long-horizon agency that distinguishes general intelligence from narrow capability. Without it, no extrapolation from current benchmarks reaches AGI. With it, the entity reached may not be a unitary general intelligence in the human sense at all, but something the design choice above has yet to define.

The previous post argued that the timeline is unforecastable because there is no measurable progress to extrapolate from. This post adds a second reason: even if the engineering breakthrough came tomorrow, the question of what we built would not be settled by the engineering. Breakthroughs are hard to predict in any case. Breakthroughs whose conceptual target is itself underspecified are doubly so.

The reason this matters is structural. When the two are conflated, when researchers sit inside engineering teams and are reviewed against engineering deliverables, the incentive structure quietly favors short-term, visible delivery over long-term, compounding progress. This post is an argument for why that conflation costs companies their breakthroughs, and what it takes to host both cultures under one roof.

The Year Before the Breakthrough

Academic science moves paper by paper, and the cadence rewards small, defensible contributions: a dataset extended, a method refined, a benchmark nudged forward. Breakthroughs happen, but the unit of progress is the publication, and progress is visible because progress is defined to be visible.

Industrial research is different. When a company asks a team to make a new technology actually work for a real use case, the goal stops looking incremental from the outside and becomes binary. Inside the team the work is still incremental, but the rest of the company sees only one signal. It doesn’t work until it does. The classic example is training a foundation model. The year before it works looks like nothing from the outside: curated datasets, dataloader plumbing, a long string of failed architectures, hypotheses ruled out. None of it ships. None of it shows up in a release note. None of it can be demoed. Then a threshold is crossed, and only in retrospect does the prior year look like progress.

Research progress in this regime is non-linear and compounding. Failed experiments are not wasted effort; they are the substrate the breakthrough is built on. But during the compounding period the work is genuinely invisible. Not because researchers are hiding it, but because the units of progress (a ruled-out hypothesis, a cleaner dataset, a slightly better internal benchmark) don’t render in the language product and engineering use to track delivery.

Two Kinds of Work

The distinction is simple to state.

• Engineering is reliable delivery against known specifications. The work rewards predictability. Success looks like shipped features, system uptime, customer outcomes, velocity. The right question to ask an engineer is: “when can we have this?”
• Research is the reduction of uncertainty on hard problems. The work rewards being right about hard things, not fast about easy ones. Success looks like insight, killed hypotheses, validated prototypes, decisions changed. The right question to ask a researcher is: “what did you learn, and what should we do differently?”

This is not a hierarchy. Both kinds of work demand deep expertise: engineers who ship reliably at scale and researchers who frame the right experiment and read its result correctly are doing equally non-trivial work. The difference is in shape, not in difficulty.

The framing closest to industrial research at its best is what Donald Stokes called Pasteur’s Quadrant: research that is both fundamentally novel and aimed at a concrete real-world problem. It is also the kind of work that fits least comfortably into a sprint cadence.

When Incentives Don’t Match the Work

Whether researchers and engineers sit on the same team, share standups, or report to the same person matters less than people think. The argument is not about silos or org charts. What matters is the incentive structure: what counts as “having delivered”, what gets rewarded at review time, what the team celebrates as success.

If researchers and engineers are reviewed against the same scoreboard, the comparison is structurally unfair. Not because anyone is being lazy, but because the units of progress don’t compare. The engineer can show shipped tickets. The researcher can show a notebook full of negative results. Side by side, the engineer always wins.

The consequence is predictable. Researchers don’t stop being researchers by decision; they stop by gradient, one sprint at a time. The hard, uncertain, possibly-zero-payoff bets get deprioritized in favor of the predictable wins. Researchers measured by engineering yardsticks start working like engineers. What’s left is a team of engineers working on safe problems, which is precisely what you didn’t fund a research function for.

This also affects who stays. The people most willing to do uncertain work, the ones who would have produced the breakthroughs, are the first to leave when the system tells them their work doesn’t count.

What remains is a company that claims an R&D function while running an engineering one. The label persists; the conditions for breakthrough research do not.

Cherishing Both Cultures

Some argue the cultural tension is so difficult to manage that startups should avoid dedicated research functions altogether . That may be the right call when research is peripheral to the business model. But for companies whose edge depends on solving problems that don’t yet have solutions, avoiding the problem doesn’t make it disappear. It only defers the question of how to build the conditions in which that research can succeed.

Fixing this is a leadership responsibility. Researchers can’t change the incentive structure they’re embedded in, and engineers can’t be blamed for responding rationally to the one they have. The job is to design the conditions in which both kinds of work can succeed, not to force one to look like the other.

In practice, that means a set of cultural commitments:

• Treat “what did you learn” as a first-class deliverable, not a consolation prize.
• Defend research time at the leadership level, rather than negotiating it away sprint by sprint.
• Make research progress legible on its own terms: regular forums, written artifacts, decisions changed because of a research result.
• Use different review rubrics for research and engineering work, even when the two roles sit at the same pay grade.
• Acknowledge out loud that some research returns nothing usable, and that this is the cost of working on hard problems, not a sign of failure.

Each of these is easy to say and hard to hold. Under customer pressure, the gradient always points back toward predictable delivery. Holding the line is what creates the conditions for the breakthroughs.

Cherishing research is not the same as exempting it from accountability. Drift is a real failure mode, and protected research time only matters if that time is being spent well. Healthy research teams hold themselves to specific kinds of rigor: explicit hypotheses, written technical bets, kill criteria for experiments, and regular strategic review of which directions are still worth pursuing. The aim is not to make research look like engineering, but to keep exploration directed and the difference between learning and drifting legible to leadership.

History supports this. Bell Labs produced the transistor, information theory, and Unix not because researchers were also engineers, but because the institution treated research as a distinct, valued discipline with its own success criteria, embedded in a company that knew how to commercialize the result . And the pattern is not unique to Bell Labs: the industrial research divisions that left a lasting mark on technology, including IBM Research and Xerox PARC, all gave research its own discipline and criteria rather than collapsing it into engineering . PARC’s legacy carries both lessons. Protecting research is necessary but not sufficient: the foundational technologies it produced were turned into products by other companies, a reminder that breakthroughs also need a credible path from result to product .

The Interface Between the Two Cultures

Keeping research and engineering distinct is not the same as keeping them apart. Both need to be grounded in the same customer problems, not because one team owns that signal, but because shared grounding is what keeps research from drifting into abstraction and engineering from solving the wrong things efficiently.

In practice the interface is about legibility in both directions. Research makes its findings visible: what was learned, what was ruled out, what changed, and what the company should do differently. Engineering makes its constraints visible: what is deployable, what breaks at scale, what customers can’t work with. In machine learning, for instance, the model is a research artifact, owned and iterated by the research team, while engineering owns the serving infrastructure, latency budgets, and the cost envelope the model must fit into. Neither side hands a finished artifact across a wall. The work is continuous: shared context, regular conversation, and decisions made together about when a research result is ready to shape what gets built next.

From Vision to Breakthrough

Research is most useful in companies that have a long-term vision: a clear sense of where they want to be in three or five years, and an honest assessment of the technical gaps between today and that future. Without a vision, research becomes hard to direct and easy to underfund. With a vision, researchers can lay the foundation for it by tackling the high-uncertainty problems that engineering can’t yet specify, instead of being squeezed into highly-specified short-term deliverables that only an engineer can satisfy.

This is also the connection to what I’ve previously called Technology Readiness Debt. When a company builds before the underlying technology is fully ready, research is the mechanism that pays off that debt on the company’s own timeline, rather than waiting for the world to do it instead.

The companies that build defensible technical IP are the ones that figured out how to host two different shapes of work under the same roof, without forcing one to look like the other. Research is not slow engineering. The distinction is not a tension to resolve; it is a feature of how breakthroughs actually happen.

In the end, don’t get caught up in the terminology. Whether you call the work research, R&D, or advanced engineering, leadership’s job is to ensure that both short-term delivery and long-term progress are rewarded on their own terms. Cherish both cultures, or you will quietly lose the conditions for either to do its best work.

Bayesian inference provides one of the most principled approaches to learning from data. By maintaining a distribution over plausible hypotheses and updating this distribution as evidence accumulates, Bayesian methods naturally capture epistemic uncertainty arising from limited data. Predictions are obtained by averaging over hypotheses according to their posterior probability.

In practice, however, exact Bayesian inference is often computationally expensive. This challenge has motivated a range of approximate methods, from variational inference to Monte Carlo techniques. A particularly intriguing recent idea is that inference itself can be learned. Rather than performing Bayesian inference for every new dataset, one can train a neural network to approximate the resulting predictions directly.

This idea is realised by prior-fitted networks (PFNs) . These models achieve remarkable performance on small-data tasks and can produce well-calibrated predictive distributions with a single forward pass. At the same time, their success raises interesting conceptual questions about what exactly remains of the Bayesian framework once inference has been amortised.

Prior-Fitted Networks

The starting point of Bayesian prediction is a hypothesis space $\mathcal{H}$ together with a prior distribution $p(h)$ over hypotheses $h \in \mathcal{H}$. Given a dataset $d$, Bayesian inference produces the posterior distribution

Predictions for a new input $x$ are obtained through Bayesian model averaging

Computing this posterior predictive distribution typically requires integrating over the hypothesis space, which can be challenging in complex models.

Prior-fitted networks take a different approach. Instead of performing inference at test time, they learn to approximate the posterior predictive distribution directly.

Training proceeds by sampling synthetic tasks from the assumed prior. Concretely, a hypothesis $h \sim p(h)$ is first drawn from the prior. Inputs $x$ are then sampled from a predefined input domain, and the corresponding outputs $y$ are generated according to the predictive model $p(y \mid h,x)$. Repeating this process yields many datasets $d$ together with query inputs $x$ and targets $y$.

The network is trained to predict these targets from the observed dataset and query input, thereby learning a mapping

A key theoretical result of (Corollary 1.1) shows that this training procedure minimises the expected KL divergence

where the expectation is taken over datasets $d$ generated from the prior. In other words, the network is trained to approximate the posterior predictive distribution on the distribution of datasets induced by the prior.

This idea is both elegant and powerful. Once trained, a PFN can produce predictions for a new dataset with a single forward pass, effectively amortising the cost of inference across many tasks. In practice, this approach has shown impressive performance on small-to-medium tabular datasets, as demonstrated by TabPFN .

The Bayesian Interpretation

Because PFNs are trained to approximate the posterior predictive distribution, they are sometimes described as Bayesian predictors. This description captures an important aspect of the method: the training objective explicitly encourages the network to reproduce Bayesian predictions.

However, it is important to distinguish between approximating Bayesian predictions and performing Bayesian inference.

In a classical Bayesian model, predictions arise from explicitly averaging over hypotheses

The posterior distribution $p(h \mid d)$ is central to this process. It allows us to sample hypotheses and interpret predictive uncertainty in terms of disagreement between plausible explanations of the observed data.

A prior-fitted network, in contrast, does not maintain such a distribution. Instead, predictions are produced by a deterministic function

The Bayesian model average is therefore no longer part of the inference procedure itself; it has effectively been compiled into the network parameters during training.

From this perspective, PFNs can be viewed as performing supervised learning on a meta-learning objective. The training data consists of many synthetic datasets generated from the prior, and the network learns to predict the corresponding targets. Maximum likelihood estimation can in turn be interpreted as minimising the KL divergence between the model’s predictive distribution and the ground-truth conditional distribution. I discuss this connection in more detail in a recent blog post.

The impressive empirical performance of PFNs therefore reflects the ability of neural networks to generalise across tasks drawn from the prior distribution.

How Bayesian Is It?

The previous section made a structural point: PFNs reproduce Bayesian predictions on average across tasks, but the inference itself is compiled into the network parameters rather than performed explicitly. This leaves open a sharper question. In what sense does the procedure that the network actually implements deserve the Bayesian label? It helps to separate three senses of the word: reproducing Bayesian predictions, implementing Bayesian inference as an explicit procedure, and admitting a Bayesian interpretation at the level of behaviour. The previous section established that PFNs do the first but not the second. It is the third sense that the rest of this section probes, and several distinct issues make it hard to settle.

The Bernstein–von Mises limit

The first complication is asymptotic. Under regularity conditions, the Bernstein–von Mises theorem states that, as the dataset grows, the posterior $p(h \mid d)$ becomes approximately Gaussian and concentrates around the maximum likelihood estimate. In this regime, Bayesian model averaging collapses towards a point estimate, predictive uncertainty is governed by the local curvature of the likelihood, and the influence of the prior on predictions fades.

Methodologically, this is convenient. Interpretively, it is somewhat uncomfortable: a wide range of procedures, Bayesian and non-Bayesian alike, become indistinguishable in their predictions. Calling any one of them “Bayesian” tells us little about the underlying mechanism.

PFNs are evaluated in small-data settings, where this asymptotic equivalence has not yet kicked in. That is the regime in which the Bayesian label is most informative, and also the regime in which it is hardest to verify.

Outside that limit, behaviour is shaped by prior and likelihood

Once we leave the asymptotic regime, Bayesian predictions become visibly sensitive to the chosen prior and likelihood. I discussed this in detail in an earlier post: epistemic uncertainty is only meaningful relative to a prior, and likelihood misspecification can silently corrupt the posterior.

In the classical Bayesian setup, the prior $p(h)$ is explicit and feeds directly into the posterior. A PFN sees no such prior. The network is exposed only to datasets sampled from the marginal data distribution induced by the prior and the predictive model. Different combinations of prior and likelihood can induce indistinguishable distributions over observed datasets; the network, in turn, is free to learn any mapping that reproduces the induced input-output statistics.

The learned object is therefore a conditional distribution over outputs given a dataset, not a posterior over hypotheses. Calling it Bayesian relies on knowing which prior generated the training data: that connection is supplied by the modeller, not enforced by the training procedure.

Bayesian guarantees on in-context learning assume exact inference

A different angle on the question comes from recent work analysing in-context learning through Bayesian formalism . Two properties of true Bayesian predictors feature centrally in this line of work. First, predictions form a martingale as data accumulates: the current prediction is the conditional expectation of all future ones, so they fluctuate around it without systematic directional drift. Second, under exchangeability of the data, the order in which observations arrive does not affect the resulting prediction.

These results are best read as characterisations. They describe what an exact Bayesian system must look like, not whether a given learned model in fact behaves that way. The training objective of a PFN is multi-task supervised learning; nothing in it enforces these properties.

The continual learning literature offers a concrete illustration of how easily such properties can break.

PFNs, in turn, are not explicitly Bayesian by construction. It is therefore not obvious that they should satisfy properties that even careful Bayesian approximations fail to satisfy.

A Simple Illustration

The distinction becomes visible when comparing PFN predictions to the true posterior predictive distribution outside the training regime.

The figure below shows the result of a simple experiment based on a Gaussian process prior. A PFN is trained on datasets sampled from this prior and then compared to the analytic posterior predictive distribution.

Within the training regime the PFN closely matches the Bayesian posterior predictive. Outside it, the two diverge: the Gaussian process posterior continues to reflect the prior and kernel assumptions, whereas the PFN falls back on the inductive biases of its architecture. This is expected: during training, inputs are drawn only from a specific domain, so the network never receives direct signal about prior-induced behaviour beyond it.

Learning, Compression, and the Bayesian Brain

The PFN is trained to minimise the cross-entropy between its predictions and targets drawn from the prior-induced data distribution. Cross-entropy is expected code length, so the same objective can equally be read as compression: the network is pressured to compress many inference problems into a single forward pass. Its optimum is already known. The result quoted earlier (Corollary 1.1) says the objective is minimised exactly when the network reproduces the Bayesian posterior predictive. On the training distribution, then, compressing the data well and matching the Bayesian posterior predictive are the same thing.

Crucially, this is a statement about outputs, not about mechanism. The objective scores only what the network predicts, so any internal procedure producing those predictions is equally optimal under it. Compression pressure forces the network to settle on some algorithm for turning a dataset into predictions, but cannot single out which one. That algorithm might be a form of approximate Bayesian inference, or something else entirely; from the predictions alone we can neither tell which, nor recover the prior and likelihood it would correspond to. This is structurally the same situation as the long-standing Bayesian brain hypothesis debate : cortical computation is often modelled as approximate Bayesian inference, yet whether the brain actually implements anything that deserves the description remains actively contested .

Concluding Remarks

Prior-fitted networks provide a fascinating example of how inference itself can be learned. By training on large numbers of synthetic datasets, they effectively amortise Bayesian prediction and enable fast, single-pass inference on new problems.

The training objective encourages the network to reproduce Bayesian predictions on average across tasks, but it does not enforce the properties (exact posterior averaging, path-independence, prior-driven extrapolation) we usually associate with the Bayesian label. Whether the learned procedure is well described in Bayesian terms depends on the regime in which we evaluate it, and matters most precisely where PFNs are most useful: in the small-data regimes where the prior would otherwise dominate.

PFNs are perhaps best understood as amortised approximations of Bayesian predictors. Their behaviour can closely resemble Bayesian inference within the training distribution, but the strength of this resemblance ultimately depends on how well the learned mapping generalises beyond the tasks seen during training.

Introduction

Uncertainty estimation has become one of the central themes in modern machine learning. The promise is appealing: if models can quantify what they know and what they do not know, we should be able to build systems that behave safely in the presence of noise, detect unfamiliar inputs, and actively acquire new knowledge.

The conceptual foundation of this promise is the widely used distinction between aleatoric and epistemic uncertainty. Aleatoric uncertainty is typically described as the irreducible randomness of the data-generating process, while epistemic uncertainty reflects gaps in the model’s knowledge that could in principle be reduced with more data. In theory, separating these two sources of uncertainty would allow us to reason about noisy environments and model ignorance in a principled way.

In practice, however, the situation is far less clear. Most uncertainty measures used in machine learning are derived from properties of a specific probabilistic model rather than from the true data-generating process. As a result, quantities that are often interpreted as intrinsic properties of the world are in fact shaped by modelling assumptions such as the chosen likelihood function, hypothesis class, and prior. This mismatch becomes particularly visible when we attempt to evaluate uncertainty estimates: instead of directly measuring their quality, we typically rely on complex surrogate tasks such as active learning or out-of-distribution detection. But these tasks depend on many factors beyond uncertainty itself, making them unreliable indicators of whether our uncertainty estimates are actually meaningful. As discussed in my earlier work on OOD detection , the connection between uncertainty estimates and downstream tasks is subtle and often misunderstood.

This conceptual ambiguity becomes especially problematic when considering one of the most popular tools in the uncertainty literature: the information-theoretic decomposition of predictive uncertainty into aleatoric and epistemic components. While this decomposition is frequently interpreted as revealing two fundamental sources of uncertainty in the data, it is in fact derived entirely from the structure of the model. Consequently, it does not generally provide a clean separation between intrinsic data noise and model ignorance.

In this post, I argue that this distinction — though intuitively appealing — is often misinterpreted in modern machine learning. By examining how uncertainty estimates depend on modelling assumptions, we will see why the commonly used decomposition can be misleading, why epistemic uncertainty is fundamentally tied to the choice of prior, and why extracting meaningful uncertainty becomes particularly difficult in amortised inference systems such as modern neural networks.

Decomposing Predictive Uncertainty

For a proper introduction to Bayesian statistics in the context of machine learning, see Chapters 2 and 3 of my thesis: .

Consider a probabilistic model parameterized by $\theta$ that maps inputs $x$ to outputs $y$. We denote a dataset of input–output pairs by $d$. In a Bayesian setting, the model parameters are treated as random variables and updated after observing data. Accordingly, we consider the random variables $D$, $Y$, and $\Theta$ corresponding to the dataset, predictions, and model parameters.

Given an observed dataset $d$, the posterior predictive distribution is defined as

where $p(\theta \mid d)$ denotes the posterior distribution over model parameters.

It is useful to consider the mutual information between predictions $Y$ and parameters $\Theta$ given the observed dataset $d$ and input $x$. This quantity can be written as

This expression shows that the mutual information can be interpreted as the expected divergence between the predictive distribution of an individual hypothesis and the posterior predictive distribution. In other words, it measures how much individual models sampled from the posterior disagree with the overall predictive distribution. For this reason, it is often used as a quantitative measure of epistemic uncertainty (see Section 3.4 of my PhD thesis ).

Using standard identities from information theory, we can derive a decomposition of the predictive entropy that relates it to this mutual information term. A decomposition of this form was promoted, for instance, by .

Under the assumed generative model, the prediction $Y$ is conditionally independent of the dataset $D$ given the model parameters $\Theta$. Therefore,

Using the standard identity

we obtain the following decomposition of the predictive entropy:

This identity is frequently interpreted as decomposing predictive uncertainty into aleatoric and epistemic components. In the next section, we will examine this interpretation more carefully and argue that the first term does not generally correspond to true aleatoric uncertainty.

Why the Expected Entropy Is Not Aleatoric Uncertainty

The decomposition above is mathematically correct. However, interpreting the first term as aleatoric uncertainty is generally problematic.This issue has been discussed previously, e.g., by .

To see why, recall that aleatoric uncertainty is a property of the data-generating process, i.e., the entropy of the true conditional distribution ($p^*(y \mid x)$). In contrast, the quantity

depends entirely on the predictive distributions produced by the model. It therefore reflects how uncertain individual hypotheses are under the current posterior, rather than a property of the underlying data distribution.

In a very specific limit, the two can coincide. If the model is correctly specified and the amount of data grows without bound, the posterior ($p(\theta \mid d)$) collapses onto the true parameters. In this case the predictive model recovers the true data-generating distribution, and the expected entropy indeed corresponds to the aleatoric uncertainty.

In practice, however, we rarely operate in this limit. Models are almost always misspecified — no parameter setting exactly represents the true data-generating process — and so care must be taken when interpreting the expected entropy term a measure of aleatoric uncertainty (cf. ). We return to the consequences of misspecification in more detail below.

The decomposition therefore separates disagreement between models from uncertainty within individual models, but this should not be mistaken for a clean decomposition of predictive uncertainty into epistemic and aleatoric components of the data-generating process.

The Prior Determines What a Model Considers Surprising

Bayesian models know what they don’t know — but we humans don’t know what they know. In other words, we cannot easily tell whether the epistemic uncertainty of a Bayesian model is shaped by the data or by the chosen prior.

When priors are hand-crafted by domain experts, this is at least transparent: one can inspect the prior, challenge it, and decide whether the resulting notion of “what the model doesn’t know” is meaningful. In modern neural networks, this transparency is absent. The effective function-space prior is determined implicitly by architecture, weight initialisation, and optimiser dynamics — none of which were designed with any particular uncertainty semantics in mind. As a result, even a model performing exact Bayesian inference may produce epistemic uncertainty estimates whose meaning is opaque.

The figure above illustrates this concretely. In the training region, both models fit the data closely and agree with each other. In the extrapolation region, their behaviour diverges entirely: the RBF kernel posterior reverts toward the prior with rapidly growing uncertainty, while the ESS kernel posterior continues to generalise confidently. Both models are doing everything right from a Bayesian perspective — the difference is entirely in what each prior encodes as plausible behaviour far from the data. Neither prior is “correct”; they simply reflect different inductive biases. For a neural network, we do not even have access to such a comparison: the implicit prior is not interpretable, so the epistemic uncertainty it induces is not interpretable either. Low epistemic uncertainty in any region may reflect genuine model confidence, or it may simply reflect a prior that happens to extrapolate smoothly — and from the model’s output alone, there is no way to tell .

Misspecification Silently Corrupts Both Uncertainty Components

Even setting aside the question of priors, the reliability of uncertainty estimates depends on a further assumption: that the chosen likelihood is capable of representing the true data-generating distribution. Under model misspecification this fails, and both uncertainty components are affected. Aleatoric uncertainty reflects the noise structure imposed by the likelihood rather than any property of the world. Epistemic uncertainty can become actively misleading: as data accumulates, the posterior concentrates around the least-wrong parameter setting, producing a model that grows increasingly confident in a subtly incorrect answer .

The severity of this problem depends heavily on how flexible the likelihood is. Universal function approximators — such as deep neural networks with expressive output heads — can in principle learn a wide range of conditional distributions without strong prior commitments about their shape. This substantially reduces the risk that the likelihood is fundamentally incompatible with the data. The remaining danger lies in cases where hard structural assumptions are imposed: for instance, assuming that the predictive distribution is always Gaussian, or always unimodal. Such constraints can be invisible in well-studied settings but catastrophic on out-of-distribution inputs or in tasks with genuinely multimodal targets. The takeaway is not that misspecification is unavoidable, but that flexibility needs to be deliberately built into the likelihood — it is not guaranteed by simply using a large model.Though there is no free lunch, more flexibility might come at a cost. See my recent blog post for more details on this argument.

Concluding Remarks

The decomposition of predictive entropy into expected entropy and mutual information is a valid information-theoretic identity. However, interpreting this identity as a principled decomposition of predictive uncertainty into aleatoric and epistemic components is generally misleading. The quantities involved are defined purely in terms of the model’s predictive distributions and therefore do not directly correspond to properties of the underlying data-generating process.

Conceptually, it is helpful to keep in mind that aleatoric uncertainty is a property of the data, while epistemic uncertainty reflects limitations of our knowledge and modelling assumptions. Even granting that separation, epistemic uncertainty is only meaningful relative to a prior — and for neural networks, the implicit prior is not interpretable by design. Furthermore, even a correctly specified model can have its uncertainty estimates silently corrupted by misspecification in the likelihood, causing the posterior to concentrate confidently around a wrong answer. While certain limiting cases — such as correctly specified models with infinite data — can align these notions with the information-theoretic decomposition, such conditions rarely hold in practice.

I hope this short discussion helps create a bit of conceptual clarity around the interpretation of uncertainty decompositions in machine learning.