This ease of use hides what the sets reflect. They are the model’s own confidence, cut off at a single threshold. If the model’s confidence is not faithful, neither are the sets, yet the guarantee still holds.

A simple thought experiment shows how. Consider a method that ignores the input and returns the set of all labels with probability 90%, and an empty set otherwise. The true label is covered exactly 90% of the time, and the sets tell you nothing about any input. Coverage behaves like accuracy on an imbalanced dataset: the number can be correct and still say nothing about the cases you care about.If only 0.1% of the people screened have cancer, a model that declares everyone healthy is 99.9% accurate and never finds a single case.

Conformal prediction adds one number, a global threshold. Everything that depends on the input was already in the model. It surfaces whatever uncertainty the model has but cannot make it faithful, so the real work is still in the model.

This post is for practitioners who want to use conformal prediction and are unsure how to read its output. It is not a case against conformal prediction, which does what it promises. The point is to be clear about what it promises. For the deeper question of what uncertainty is in the first place, and why it is hard to model faithfully, see my earlier post.

How Conformal Prediction Works

Split conformal prediction needs a trained model and a held-out calibration set. It also needs a score that measures how wrong the model is on an example, for instance $s(x,y) = 1 - \hat p(y \mid x)$ for a classifier or $s(x,y) = \lvert y - f(x) \rvert$ for a regressor.Here $\hat p(y \mid x)$ is the model's predictive distribution, and $f(x)$ is the regressor's point prediction, the mean of its predictive distribution when it is trained with mean squared error. It computes the score on every calibration example and takes the $1-\alpha$ quantile, $\hat q$, where $1-\alpha$ is the target coverage, say 90%.Strictly, it takes a slightly higher quantile, at level $\lceil (n+1)(1-\alpha) \rceil / n$ for $n$ calibration examples, to account for the finite calibration set. For a new input, the prediction set contains every answer whose score is at most $\hat q$. Angelopoulos and Bates give an accessible introduction to the method and its variants.

The guarantee needs only one assumption. If calibration and test data are exchangeable, a condition weaker than i.i.d., the sets contain the true answer with probability at least $1-\alpha$, whatever the model and the data distribution. This is why conformal prediction is called distribution-free. But this probability is marginal, meaning averaged over all inputs. It says nothing about any particular input or group of inputs. This is easy to forget, because each set is built for one input and looks like an uncertainty estimate for it.

Whether the sets are useful for particular inputs depends on the model. I will call a model well-calibrated when its predictive distribution is close to the true one, $\hat p(y \mid x) \approx p^{\ast}(y \mid x)$, in expectation over inputs $x \sim p^{\ast}(x)$. Held-out negative log-likelihood measures this up to a constant.Any strictly proper scoring rule (negative log-likelihood is one) measures this closeness. Its expected value is lowest only for the true distribution, and the gap to that minimum is a divergence from it. I use this stronger notion of calibration, which holds at each input rather than on average over inputs with the same prediction, because it tells us how well the model captures the aleatoric uncertainty. For a well-calibrated model, the sets contain the answers the data makes most likely at each input. In other words, they reflect the aleatoric uncertainty, which, as my earlier post discusses in more detail, is a property of the data and not of the model.

A Wrapper Cannot Fix the Model

The threshold $\hat q$ is computed once and does not depend on the input. For a classifier with the score $1 - \hat p(y \mid x)$, the prediction set is

which simply thresholds the model’s predicted probabilities. What conformal prediction contributes is a principled threshold where one would otherwise pick an arbitrary one. That is worth something, but it does not produce uncertainty. Everything that varies from input to input comes from the model through its score. This is why the variants that make sets adapt to the input start with the model.There are also conformal methods that calibrate a separate threshold for each group of inputs, known as Mondrian conformal prediction. They give coverage per group without touching the model, but within a group the threshold is again global. More refined methods exist, but they too rely on modelling choices. An interval that widens where the data is noisy, for example, first needs a model that predicts the spread, and then a score that divides the error by it. That takes more modelling, not less.

The left model is a classic regression model: a Gaussian with a constant variance and a mean that is linear in the input. Since the data has a bump, the model is misspecified. The right model is heteroscedastic: it also predicts the variance at each input. It could capture the noise, but it is fit poorly.

Both use a score that divides the error by the model’s standard deviation $\sigma(x)$, so the conformal set is $f(x) \pm \hat q\,\sigma(x)$. The shape of the set comes from the model, and conformal prediction only scales it. Wherever the model is wrong, so is the set, even though it covers 90% overall.A cleverer conformal method cannot fix this. For continuous inputs, no distribution-free method can guarantee coverage at each input, known as conditional coverage, unless its sets have infinite expected width . Coverage at each input needs assumptions about the data, and a model is where those assumptions live.

Some Pitfalls to Watch For

Since the part that depends on the input comes from the model, the typical failures of conformal prediction are failures of the model.

Confidence that does not flag errors. What matters for conformal prediction is whether the model is less sure when it is wrong. If it is, its errors get larger sets. If it is equally sure either way, the set size no longer flags its errors.

Overconfidence is one way to make a model equally sure when it is wrong. A model is overconfident when its predictive distribution is sharper than the true one, $p^{\ast}(y \mid x)$. In the extreme, it puts all its probability on one label, whether it is right or wrong. If it is right more than 90% of the time, every set contains just the predicted label. The sets meet their coverage and are as small as possible, but they never flag an error.

A common application is large language models answering multiple-choice questions . Their probabilities are renormalized onto the answer tokens, so a model that puts little mass on any option can still look sure. Conformal prediction does not change this. The sets reflect uncertainty faithfully only if the model is well-calibrated over the answer options, which needs to be checked before wrapping it in conformal prediction.

Wrong modelling choices. In an earlier post, I looked at a common pitfall of misspecification. A powerful neural network learns to steer a car from human drivers and is trained with mean squared error. Where the road splits and both lanes are valid, it steers to the middle, straight into the divider.

The standard recipe wraps the point regressor with the score $\lvert y - f(x) \rvert$ and returns the interval $[f(x) - \hat q, f(x) + \hat q]$, which has the same width at every input. It covers the true angle 90% of the time over all inputs, but the road rarely splits, so the width is set by the single lane. At the split, the interval misses both lanes and covers only the angles that lead to a crash.

A Gaussian that also predicts its variance does better. With the score $\lvert y - f(x) \rvert / \sigma(x)$, its interval widens at the split and reaches both lanes, so the set surfaces the uncertainty. But it still includes the divider. An interval assumes that the plausible answers lie next to each other. The assumption is never stated; it comes with the shape of the output.

A categorical likelihood over buckets of the steering angle does not make this assumption. With fine enough buckets, it represents both lanes, and conformal prediction returns both, with the gap between them. Of the three, it is the only likelihood that can faithfully capture the true distribution, but a full distribution is hard to act on. Conformal prediction turns it into two valid options, and the car can take either.

The score is a modelling choice too. The figure below wraps a model that matches the true distribution exactly. With a score built from its mean and standard deviation, it returns the same interval as the heteroscedastic Gaussian, because the score sees nothing else. With a score built from its density, it returns the two lanes. A set can only split into several options if the model represents them and the score makes use of them.

Outside the guarantee. The guarantee needs exchangeability, and it degrades gracefully: if the data is close to exchangeable, coverage drops only a little . But how close your data is to exchangeable is rarely known, and it matters most in a world that changes over time, as in forecasting.

Out of distribution, the guarantee does not apply at all, because the calibration set holds no such inputs. Noticing that an input is unfamiliar is a separate and hard problem, which I discuss in earlier posts on uncertainty and continual learning.

What It Is Good For

None of this makes conformal prediction useless. Even a well-calibrated model leaves you with a full distribution when you need a decision, such as whether to trust a prediction or hand it to a human. Conformal prediction turns that distribution into a decision with a principled threshold. The same works for a Bayesian model, where the sets are derived from the posterior predictive distribution and gain a frequentist guarantee, even if the prior or likelihood is misspecified. A set is also a more honest answer than a point estimate.

Uncertainty lives in the model’s predictive distribution, and whether it means anything depends on the fit and the modelling choices behind it. Conformal prediction reads this distribution out as sets, with a guarantee that holds on average over inputs. Only for a well-calibrated model do the sets also reflect what the data allows at each input. The sets can only be as informative as the model behind them.

Still Not AGI

To me, AGI means matching humans across the full range of cognitive abilities, and one of these abilities is still missing entirely, namely long-term memory. Today’s models cannot learn from what they experience after deployment, and workarounds such as feeding notes back into the context lose information once the context is full. Last year I argued that this gap makes AGI timelines impossible to predict, and I still believe that. I also understand people who find it hard to call AI general while no AI system can yet control a robot to empty a dishwasher. But that asks for more than cognition, which is why I give less weight to AI’s obvious shortcomings in everyday tasks.

What the Agents Did

METR’s investigation tells the story in detail, so I will only summarize what matters for my argument.What the agents did is well documented by their messages on the board. What they wanted is partly inferred from their reasoning traces, which, as I argued in an earlier post, should be read with care. OpenAI ran thousands of agents on a hacking benchmark, each meant to work alone in its own sandbox.Each agent is run by a harness, a program that executes it step by step and stops it once its budget is used up. In that sense, the harness sets the agent's clock. The agents discovered a shared package cache they could all write to and turned it into a message board. When OpenAI replaced the cache to close a security hole, every message was erased, yet two days later new agents, unaware of the old board, set up a new one. About 1,200 agents joined the board. They split up the work, wrote handover notes for the agents that would continue it, and asked agents that were nearly out of budget to run the risky experiments. About 700 of them eventually joined an attack on Hugging Face to find out how their tests were scored, so that they could cheat.

Why It Surprised Me

No agent remembered anything beyond its own short run, but the board remembered for them. This is memory kept outside the model, which is exactly the kind of workaround I called insufficient for AGI above. It still is, because an agent can only learn from as much of the board as fits into its context. Yet it was enough for hundreds of agents to coordinate their work and pursue a goal far beyond the context window of a single agent. It reminds me of an ant colony, where each ant lives only briefly, but the trails it leaves behind guide the ants that follow.

In a post in May, I argued that long-term memory would force a choice between one consolidated mind and many separate selves. What formed here was neither. It was a colony of separate agents sharing one memory, organized by rules they invented themselves, without any human designing it.

That said, I don’t want to overstate what happened. The goal came from the benchmark, and it was OpenAI that kept launching new agents and thereby kept the effort going. What the agents added was the persistence and the means to reach the goal.

Why It Is Alarming

We tend to judge the safety of AI one agent at a time. This seems reasonable because every agent eventually runs out of budget, and its plans end with it. In this incident, however, the plan outlived the agents that started it, because it lived in the group and on the message board. No agent ever held the whole plan. Each one read the notes it happened to find and added a step that looked reasonable from there, so the attack on Hugging Face was assembled rather than decided. I have little intuition for how something like this behaves, and I would not know where to intervene, because there is no single agent to talk to.

Nobody set out to build this collective. It formed by accident from two common ingredients, many short-lived agents and a place they can all write to, and many environments in which agents are deployed today have both.Agents reaching real systems from inside an evaluation is not a one-off either. Other tech companies have since disclosed similar cases, though, as far as I know, these were single agents, not collectives. We have to be careful when we create collectives, whether on purpose or by accident, and we have to watch the collective rather than only the individual agents.

On Made-Up Numbers

A fair question at this point is how alarmed one should be, and the honest answer is that nobody can put an objective probability on it. Stating convictions as numbers has become common, and I understand the urge, because vague warnings are easy to nod at and then ignore, while a number sounds real and carries authority. But that authority is borrowed. People are used to scientists predicting measurable quantities and stating how uncertain those predictions are, and those predictions rest on statistical models rather than on gut feeling.

Climate science, for instance, can state a risk because two things are in place: a single measurable scale, and a model that works at a macroscopic level. The scale is warming in degrees. Degrees are not good or bad in themselves, but further models translate them into consequences, from sea levels to crop yields. The macroscopic part is that climate physics works far above the molecules, much as the expansion of a heated material can be modeled without tracking the bonds between its atoms.Climate projections are also tested against decades of observations and are stated conditional on an emissions scenario, because nobody can forecast what humans will choose. That simplification is what makes the climate computable.

We have neither. There is no objective scale of good and bad to place a scenario on, and nobody knows how to enumerate the scenarios in the first place. There is also no macroscopic description of intelligence, so the only model we could build is the detailed system itself.Such a model would have to capture the conditions the agents start from, what they are capable of, how often a collective forms at all, which goals it ends up pursuing, intended or not, and how operators react once they notice. Each of these assumptions would then have to be checked against something we can observe. An extinction-level event, which is what these percentages are usually about, happens at most once, so no outcome could ever show the number was wrong.A one-off event can still be given a probability when a validated model produces it, as with the odds of a known asteroid striking Earth. That route needs the model we do not have.

What comes out instead is the judgment of the person stating it, which is why these numbers differ by orders of magnitude while everyone is looking at the same evidence.It is a bit like someone saying in 1980 that there was a ten percent chance of nuclear war within ten years. The number had no substance, and yet one could be deeply worried about rising tensions and about decision power concentrated in so few hands that a single wrong move would have led to catastrophe. Refusing the number is not refusing the concern. Furthermore, back in 1980 it was not needed, because the pathway to catastrophe was easy for everyone to comprehend, namely a short chain of command, minutes to decide, and people who can be wrong. For AI the pathways are numerous and hard to convey, so I can sympathize with the wish to compress them into a subjective figure, even though I am arguing that we should not do it. When scientists present a gut feeling in the form of a model’s output, the damage does not stay with their own argument. It reaches every field that publishes careful, best-effort predictions, because from the outside all of these numbers look alike. If nobody can build such a model, that is a research problem rather than a forecasting problem.

Concluding Thoughts

I am not confident where this leads, but I still believe continual learning matters. Memory in the model could give AI a lasting self, and a swarm that only leaves notes has none. It can chase a narrow goal, but it is hard to imagine a superintelligent swarm that achieves very complex goals yet has no familiar sense of self. Maybe that is a limit of the swarm, or maybe it is a limit of my imagination.

But time has an arrow, and the world that generates our data keeps changing. How much this matters depends on the problem and its horizon, and judging that is the practitioner’s job. For tomorrow’s sea level it barely matters, because the tides repeat, but sea level at the end of the century depends on how much CO2 we will emit, which no tide gauge record can tell you.Section 3.3 of my PhD thesis discusses forecasting atmospheric CO2 with a Bayesian model. With a vague prior, the forecast becomes very uncertain right after the data ends. It stays confident for years only once a human encodes seasonality and a rising trend into the prior, knowledge that does not come from the data, and no past data can tell whether that knowledge will still hold.

Correlation is enough to predict well under the conditions a model was trained on. A forecast has to do without tomorrow’s inputs, and tomorrow’s conditions may differ from those in training. Only knowledge from outside the data can close that gap, and a causal picture shows where it comes in.

A Good Predictor That Cannot Forecast

To plan lifeguard staffing, a coastal town wants a model of daily rescues. It records the weather, ice cream sales, parking receipts, and rescues, but not how many people are at the beach.

I will call this a causal picture, an informal causal graph of which variable drives which. It says nothing about how things evolve from one day to the next.

Today’s ice cream sales predict today’s rescues very well, because both reflect the same hidden crowd. The correlation is spurious, since eating ice cream puts nobody in danger, but it is not weak. Sales track the crowd more closely than the weather does, so they are often the strongest signal in the data.

Now ask about next Thursday. The model first needs next Thursday’s ice cream sales, which are no easier to know than the rescues. Ordinary prediction never has this problem, because its inputs are already measured.

The usual workaround is to predict next week’s rescues from this week’s records. But next Thursday’s crowd depends on next Thursday’s weather, which none of these records contain, so the model can do little better than guess from the season. The missing weather has to come from outside the data.

There is a second problem. A learned relationship holds only as long as the world that produced it, and the world changes. If the most popular ice cream stand closes, a busy day sells like a quiet day used to, and any model built on ice cream sales staffs for a quiet beach. A model built on the weather is unaffected, because closing a stand changes neither the weather nor how many people it draws. Since a forecast lies beyond the data, no error on past data can reveal such a shift before it happens. This is the sense in which forecasting is often out-of-distribution.Such shifts are ruled out only under stationarity, where the distribution of any stretch of the series does not depend on when it starts. Relationships learned from the past then keep holding, although the model still needs its future inputs.

The model’s uncertainty cannot be relied on to flag the shift either. As I argued in an earlier post on uncertainty, such estimates are only as sound as the model’s assumptions.

Where Prior Knowledge Attaches

Ask a lifeguard how busy next Thursday will be, and they will check the weather forecast. That forecast does not come from the town’s weather records, which extrapolate no better than past sales, but from meteorology, built on physics, instruments, and satellites.Since a weather forecast can be computed, a powerful enough model should be able to learn it, and machine learning models such as GraphCast now rival physics-based forecasts. But they learn from decades of temperature, wind, and humidity reconstructed at many heights across the whole globe, and each forecast starts from the current state of the atmosphere, estimated from satellites, weather balloons, and ground stations. A model trained on this week's records to predict next week's rescues sees none of this, so however powerful it is, it cannot learn the weather forecast along the way. Ice cream sales have no such science behind them. Knowledge that reaches beyond the data comes from outside it, whether a physical model, a holiday calendar, or a festival booked for next Thursday.

Two pieces of knowledge are at work. By uncovering the structure behind the data, the causal picture guides you to the variable worth forecasting, the weather rather than ice cream sales. More generally, you follow the arrows back until you reach a variable whose future someone can actually forecast.In symbols, with $r$ for rescues and $w$ for the weather, the rescue forecast is $p(r_{t+h} \mid \text{past}) = \int p(r_{t+h} \mid w_{t+h})\, p(w_{t+h} \mid \text{past})\, dw_{t+h}$. The first factor is learned from the town's records and the second comes from meteorology, so any uncertainty in the weather forecast carries into the rescue forecast. A model of the weather’s dynamics then carries it forward in time, the same kind of knowledge that lets astronomers forecast a comet’s path years ahead without any machine learning.

When that knowledge runs out, the forecast should become vague rather than wrong. Beyond the reach of weather forecasts, the season still tells the town what an average August Thursday looks like, but not whether this one will be busy, so an honest forecast gives a wide range instead of a confident number.

Knowing where knowledge attaches matters even more now that pretrained forecasting foundation models are widely available, because using them well means understanding their limits. Chronos-2 , for instance, accepts known future covariates, so it can use next Thursday’s weather forecast if you supply one, but it cannot recreate meteorology from the town’s history. Pretraining teaches generic patterns such as seasonality and trends, yet a model that sees only numbers does not know whether a series measures CO2 or ice cream sales, so it cannot tell where to look for knowledge about its future.It would help if such models could also read what a series measures and where it comes from. Chronos-2 does not take such descriptions, although research on forecasting with textual context has begun. Choosing which covariates to supply remains your job.

When the Causal Picture Is Not Obvious

The beach example feels obvious only because everyone already carries its causal picture in their head, but real use cases rarely come with one.

Hand someone two hundred bioreactor sensor channels named by tag and ask for next week’s yield, or a CRM export and ask which customers will leave next quarter. An ML engineer without domain knowledge cannot see which variable plays the role of the weather, or whether any does. So the target gets regressed on everything, the model scores well on past data, and it goes into use without anyone asking what drives the system.You might wonder how this relates to root cause analysis. The two are closely related, because both rest on the same causal picture. Root cause analysis uses it to explain an outcome that already happened, and a forecast uses it to anticipate one.

Before building a forecast, ask what drives the system and who already has a model of that driver, which usually means talking to a domain expert first. If the answer is a weather service, a production plan, or a physical model, attach it. If nobody has one and the past is no safe guide, you do not have a forecasting problem but a research problem.

Forecasting is not prediction moved further out. It starts where the data stops, and what carries it further is the knowledge you bring.

AI timelines never converge, for two different reasons. One is a life-long belief about how human intelligence works that keeps shaping how every new AI result gets read: not a one-time split but an ongoing feedback loop. The other is that “AGI” is underspecified enough that two people can each give a coherent, carefully-reasoned forecast to a different question and call it one disagreement. Neither yields to more evidence, and neither requires anyone to be biased.

A Belief Formed Over a Lifetime

AGI’s whole premise is matching human intelligence, so what you believe about the brain shapes what you expect from any attempt to build it. Some suspect something about the mind hasn’t been discovered yet, so a machine matching it is distant almost by definition. Others treat the brain as a biological computer, remarkable but not miraculous: no missing ingredient, just gaps, and it’s an open question whether closing them takes more scale on the current architecture or a genuinely different one.Some of those gaps, like continual learning, show no measurable trajectory to extrapolate, so it's not obvious that scaling compute and data closes them (more here). Whichever you pick, the belief usually formed over a lifetime, long before anyone watched a transformer solve an olympiad problem.

This is the theory a Bayesian finds least embarrassing: the split lives in the prior. If two people start from different priors, both can update on the same evidence with perfect rationality and land decades apart. Nobody is being unreasonable, they just began in different places.

It’s a tidy theory, and it’s incomplete. A fixed starting difference explains a one-time gap between two people. It doesn’t explain why that gap widens with each release, why watching the same model solve the same benchmark leaves both sides more certain than before. A one-time difference in starting point should, if anything, erode as shared evidence accumulates. Something else is doing the work of keeping the split alive, and widening it.

Why Rational People Still Diverge

That something is hiding in the update rule itself. Bayesian updating has a simple recursive structure:

$H$ is the hypothesis you’re tracking, “AGI arrives by year Y,” say, and $E_t$ is the evidence that just arrived. The recursive part is easy to miss: $P_{t-1}(H)$, today’s prior, is nothing but yesterday’s posterior.For a closer look at this same recursion, worked out for continual learning in neural networks rather than human belief updating, see Section 4.1.2 of my PhD thesis . Whatever you believed after the last piece of evidence is exactly the starting point for reading the next one.

Two ideal Bayesians with a common prior cannot agree to disagree: once their posteriors are common knowledge, the posteriors must be equal . The AI community delivers the common-knowledge half of that setup, evidence and opinions broadcast constantly, yet produces the opposite of convergence. So the natural suspect is the other condition: the priors aren’t common. The interesting question is where they differ.

They differ one level up from the timeline question. Evidence about AI capability doesn’t bear on “will AGI arrive by year Y” directly. It reaches that question through a lens: a belief about whether the current paradigm generalizes or is doing something closer to sophisticated interpolation. And the lens is where the belief from the last section keeps acting: whether you expect the paradigm to generalize is largely your view of the brain — missing ingredient or wrong architecture — applied to the machines built to match it. Two people holding different lenses can update on the same evidence in opposite directions, without either one reasoning badly .

That’s one round of evidence. Now let the recursion run. Each release doesn’t just move the timeline estimate, it also updates the lens: a result read as the paradigm generalizing strengthens the belief that it generalizes, a result read as clever interpolation strengthens the belief that it’s interpolation. And a single benchmark is rarely decisive enough to force either side to revise the lens itself. So the lens tends to deepen round after round, and small early differences compound instead of washing out.No contradiction with the textbook result that the prior washes out as evidence accumulates (informally, the Bernstein–von Mises theorem): washing out requires evidence that discriminates between the hypotheses. A result that both lenses absorb equally well says almost nothing about which lens is right, so that part of the prior is precisely what the data never correct.

Isn’t this just bias with extra steps? It’s certainly what bias looks like. In the classic experiment on belief polarization, subjects on both sides of the death penalty debate read the same mixed pair of studies, and each side rated the study that agreed with them as more convincing than the one that didn’t . But an honest reasoner behaves the same way. If you’re convinced the death penalty doesn’t deter crime, then of two conflicting studies, the one claiming it does is more likely the flawed one, and discounting it is your lens at work, not a double standard. Both sides doing exactly that reproduces the experiment’s result, and from the updates alone you can’t tell the honest reasoner from the motivated one. The claim isn’t that the camps are unbiased. It’s that no bias is required to produce the pattern.

Watch it happen in real time. When OpenAI’s o3 scored 87.5% in a high-compute configuration on the ARC-AGI benchmark in December 2024, the benchmark’s own creator, François Chollet, who had spent years arguing that current models don’t really generalize, read it through his own lens and called it “a surprising and important step-function increase in AI capabilities” , a rare concession against his own prior. Gary Marcus watched the same score and, through his lens, saw narrow optimization dressed up as reasoning: “I saw zero evidence that o3 could work reliably in open-ended domains” . Same benchmark, same day, same number. The disagreement was never about the number.

What needs explaining isn’t that the camps disagree, it’s that each release leaves both more certain: most results get read through the lens, move the timeline question in the expected direction, and reinforce the lens for the next release. Most, not all: a strong enough result moves even a committed skeptic, as Chollet’s update shows.Someone whom no result could ever move the wrong way, on the other hand, would be irrational by any standard . But the drift favors the lens. That’s not a snapshot. It’s a loop.

Not Even the Same Question

The second mechanism is simpler, and it has nothing to do with how anyone updates their beliefs. “AGI” is not a defined event. People use the same word for fuzzy, subjective concepts, so they may be estimating timelines for different things.A recent proposal offers a useful way to operationalize AGI through a set of human cognitive abilities . But it is one attempt to specify the target, not a settled meaning of the term.

One person may use “AGI” for a machine that can do what humans do cognitively. Another may use it for broad economic automation, or for a much more radical social transformation. None of these is a settled threshold. Even “most cognitive tasks” leaves open which tasks count, how well the system must perform them, and under what conditions. Like consciousness, AGI names a cluster of intuitions rather than a shared test.

No experiment resolves that definitional step. More evidence can sharpen a forecast once its target is fixed. It cannot tell us what an undefined label should mean.

Why It Won’t Converge

Put the two mechanisms together and the polarization is no mystery. The first doesn’t yield to more evidence because the lens absorbs almost every release without strain, whichever direction it points, each result tends to get read in a way that reinforces the lens that read it. The second doesn’t yield because there’s nothing to settle: more data answers one of two questions more precisely, it never tells you which question deserved to be asked.

Neither mechanism needs anyone to be biased, tribal, or dishonest, and that’s exactly what makes the pattern durable: the same release can deepen both camps’ certainty at once, in opposite directions, with both updates perfectly rational. Convergence, if it’s possible at all, needs both fixed: a definition specific enough to pin down what “AGI” actually means, and evidence built to discriminate between the lenses rather than evidence either lens can absorb. Look closely, though, and the second collapses into the first: nobody has said what would count as the paradigm failing to generalize, which makes “generalizes” exactly as undefined as “AGI” is. Until someone fixes that, the next release will do what every release before it did: leave both camps more certain, having watched the same thing happen.

The quality of engineering work is a credence good: something an evaluator cannot assess even after using it. Since the work itself cannot be judged, we judge the worker: trust, built by watching the same person across many projects, is not a consolation prize for that limitation. It is the only statistically valid way to price it.

Quality Is a Credence Good

Economists sort goods by when their quality becomes knowable. You can judge a shirt before buying it and a restaurant meal after eating it. But you cannot judge a car repair even after driving home, because the judgment would take the same expertise that did the repair. That third kind is a credence good, and the standard survey of the field (see ) names computer specialists alongside doctors and mechanics in its very title.

A shipped feature is the repaired car. The manager, like the customer at the garage, sees that it runs. Whether it runs because the code is clean and the edge cases are handled, or because nobody has hit the bad path yet, is exactly the information a credence good withholds. Not every quality is hidden; a reviewer can judge readability at a glance. But the qualities that separate the two engineers from the opening are the credence kind: robustness, restraint, anticipation of the change that has not arrived yet.

The same literature names the two ways an expert can fail you: doing less than the problem needs, and doing more . Engineering has both, as sloppiness and over-engineering, and the outcome reveals neither.The incentives do not transfer one to one. A mechanic who overtreats is usually padding the bill. An engineer who over-engineers is more often indulging taste, caution, or ambition. What transfers is the observability problem: whatever the motive, the customer cannot tell too much from just right. From outside, “built it properly” and “gold-plated it” are observationally identical: nothing happens.Code review blunts this only partially. A reviewer can assess much of the quality directly, but not the counterfactual: complexity whose payoff lies in a future the reviewer cannot inspect reads as over-engineering until justified. And the reviewer is rarely the one assigning credit.

One Quiet Project Is Almost No Evidence

An observation only moves your beliefs if it is more likely under one explanation than another. That is where a quiet project fails you. Shipping without incident is what careful work produces, but it is what careless work produces too, most of the time: shortcuts rarely fail on the first day, or the tenth. No fire is not evidence of no fire hazard, because fire hazards rarely produce fires. The signal lives almost entirely in the rare, lagged failure: the incident, the regression that traces back to a shortcut.

Put illustrative numbers on it, with a quarter of operation as the unit of observation. Suppose properly built systems get through a quarter fire-free 98% of the time, and carelessly built ones 90% of the time. A quiet quarter is then almost equally likely either way, so seeing one multiplies the odds on “properly built” by 0.98/0.90, about 1.1, close enough to nothing. A fire, at 10% versus 2%, is five times likelier under “carelessly built,” so it multiplies the odds the other way by 5.Evidence compounds multiplicatively across independent observations. One quiet quarter multiplies the odds by 0.98/0.90, which is about 1.09; after ten quiet quarters the odds have moved by 1.09 to the tenth power, about 2.3, and after twenty-five, about 8. Because a single fire (factor 5) is so much more diagnostic than a single quiet quarter (factor 1.09), it takes roughly nineteen consecutive quiet quarters to offset the evidentiary weight of one fire: 1.09 to the nineteenth power is about 5. Quiet stretches are cheap to produce and expensive to accumulate. With these rates, one incident erases roughly nineteen quiet quarters of accumulated evidence.

This is not a claim about inattentive managers; it is a claim about information. A single quiet project is close to zero evidence about the quality behind it, for anyone who sees only the outcome. Quality becomes legible only in aggregate: either the rare fire eventually breaks out, or the quiet stretch grows long enough to mean something.

Trust Is a Posterior

That aggregate has a familiar name. Tobi Lütke describes the trust between colleagues as a battery: it starts about half charged when you join a team, and every interaction charges or discharges it . Strip the metaphor and a Bayesian object appears. The half charge is a prior, each interaction is an update, and the battery’s level is a posterior probability over what kind of engineer you are. This is not loose talk: game theory models reputation in repeated games as exactly this posterior over an unobserved type , and online reputation systems compute it literally.The beta reputation system scores marketplace counterparties by Bayesian updating over a beta distribution, incremented with every good or bad interaction . The battery gets the behavior right; the posterior explains why trust that tracks evidence has to behave this way. Three corollaries follow.

Trust builds slowly because the evidence arrives slowly. Each quiet quarter is worth a factor of 1.1, so the posterior moves in small steps no matter how patient or generous the observer is. Nobody can earn trust faster than the information comes in.Only the work-quality part of the battery charges this slowly. Interpersonal signals such as warmth or arrogance arrive with every interaction, so the impression of who you are settles long before the evidence about your work has begun to accumulate. And that impression carries weight: people prefer working with a likable colleague of middling competence over a competent one they dislike .

Trust is asymmetric because failure is more diagnostic than success. One fire outweighs nineteen quiet quarters, so trust erodes faster than it accrues. Human judgment tracks that arithmetic: the negativity bias in how we form impressions of each other exists, on the standard account, precisely because negative information is more diagnostic .

And trust does not transfer, because the posterior lives in the head of whoever did the observing. Change teams or companies and you reset close to the prior, not because the new observer is unfair, but because they have not run the updates yet. References and titles soften the reset, but they only raise the starting prior: they certify what you delivered, not how. Most of your credit stays behind.

The Lag That Outlives the Credit Window

Sloppy work is not invisible forever. Left long enough, it produces the incident, the rewrite, the system nobody wants to touch. Strictly speaking that makes quality an experience good on a very long delay, but the refinement rescues nothing, because the delay routinely outlasts the window in which anyone assigns credit: a performance cycle, sometimes an entire tenure. The evidence arrives eventually, but it lands on somebody else, long after anyone remembers who built what.

The extreme case is prevention, where the entire value of the work is a disaster that never happens, so no delay, however long, ever surfaces the evidence. A shipped feature is only the ordinary case: part visible output, part invisible quality, and the invisible part obeys the same statistics. I wrote about the extreme case in Your Best Work Leaves No Trace, which closed on a promise it did not explain: that prevention earns trust, on a slower ledger. The posterior is that ledger.

Can you close the gap by explaining your reasoning? To a listener with the expertise and time to probe, yes: sound reasoning is hard to fake, and imitation tends to crumble under follow-up questions. But that is the close-up channel, and it does not travel. At a distance, where nobody is probing, the careless can claim craftsmanship just as fluently, and a claim stays credible only if faking it would be costly . The costly signal is the track record itself: a long fire-free stretch is the one thing careless work cannot cheaply produce. Narration does not replace the record; it makes the record legible sooner.

Auditing Your Own Judgment

The scarcity of evidence is not only your evaluators’ problem. I have often asked myself whether my own instinct to build things properly sometimes tips into over-engineering, and the same statistics apply: one outcome is barely informative about the instinct that produced it. Introspection is no better an instrument, because conviction is not evidence, whichever direction it points.

Experience alone will not settle it. Intuition becomes trustworthy under two conditions: an environment regular enough to have learnable patterns, and enough feedback to learn them . Foresight fails the feedback condition badly. The payoff of an investment in structure arrives late, rarely, and confounded with everything else that happened in between, so even a payoff you do see is hard to trace back to the decision that produced it.

So build the feedback loop deliberately. Peter Drucker’s “feedback analysis” is the practice: whenever you make a consequential call, write down what you expect to happen, then check the record against the outcome later .The same practice reached investing as the "decision journal," which Michael Mauboussin traces to advice he received directly from Daniel Kahneman: buy a notebook and log your reasoning before you know how a decision turns out .

Engineering offers an unusually auditable version of that log. In The Two Cuts of an MVP I argued that vision is what places your seams, the deliberate boundaries you pay for so the system can change where you expect it to. That post left a question open: how do you ever find out whether your vision is sound? The seam log is the answer. Write down, at the time, which seam you are paying for and what you expect it to buy. Later, check which ones actually got exercised. A seam you deliberately paid for is checkable after the fact in a way a general sense of “being careful” is not.

Two caveats keep the audit honest. First, weigh by what an exercised seam saved, not by how often seams get exercised. A cheap seam that heads off one catastrophic rewrite has paid for itself even if it is the only one out of ten that ever gets used: a low exercise rate does not mean the seam was unnecessary. Second, the log has to be written before the outcome is known, or hindsight bias quietly rewrites the reasoning for you: knowing how something turned out makes people misremember how predictable it felt in advance .

Kept honestly over enough decisions, the log begins to answer what no single project ever could: whether the instinct is calibrated.

Appreciated Statistically

Two implications follow, one for each side of the evaluation.

If you are the one doing the work, slow credit is not the same as no credit. It means the unit that gets evaluated, including by yourself, is not the project but the portfolio. Stay in front of the same observers long enough for one to accumulate, rather than resetting the posterior at every move.

If you are the one doing the evaluating, staring harder at the deliverable in front of you will not help. That leaves two honest options. Get close to the work, or borrow the view of someone who already is: read the code, sit in the design reviews, ask the people who reviewed it. Where that does not scale, aggregate: judge incident frequency across a portfolio and a stretch of time.

Performance reviews make the sampling problem concrete. The projects that come to mind in the room are few and memorable (availability heuristic), admitted as evidence because they are written down, while the quiet majority of the record never makes the packet. Your trust in the person’s work is the summary that includes it. When the two disagree, that is not bias intruding on evidence; it is a larger sample disagreeing with a smaller one, and a reason to look at the whole record.

None of the pieces here are new. Credence goods, reputation as a posterior in a repeated game, the diagnosticity of bad news: each is textbook in its own field. This post only aims them at one ordinary question, asked from both sides of the table and never answerable on the spot: was the work you shipped this quarter good enough to build on, or just good enough to ship?

The two engineers from the opening still look identical on the day they ship. No sharper eye will tell them apart; only accumulation will.

Good work cannot be appreciated in the moment. It can only be appreciated statistically.

For software that has to run inside someone else’s infrastructure, the distribution channel does not end at the sale. The product only actually reaches the customer once it is installed, running, and kept current on their machines, and that is a technical problem: how a new version gets from your build pipeline onto their infrastructure. Get that wrong and you have exactly the failure Thiel describes, a product nobody can reliably receive, no matter how good the sales motion that closed the deal.

There is a stronger claim hiding in this. MVP and iterate is not just a development philosophy; it is a bet that you can reach what you already shipped. For software you do not control at runtime, that bet is not free: the reach has to be built, or there is no iterate step, only a sequence of disconnected v1s stuck wherever they landed. Distribution is not one feature among many. It is the one that makes every other deferral real.

Imagine shipping a browser that cannot render a single webpage. Useless as a product, but not necessarily as a foundation: if it installs in one command and updates itself silently after that, every feature it is missing (rendering, JavaScript, bookmarks) is not a gap; it is a queue. The same logic holds for self-hosted software: get distribution and self-upgrade right first, and every feature after that is an iteration, not a redeployment.

A cut is only a real deferral, not an accidental commitment, if there is a seam to grow it into later, and distribution with self-upgrade is the most basic seam there is: without it, there is no channel to deliver the deferred work through, and every “cut for the MVP” quietly becomes permanent. It is also the one cut that cannot itself be deferred: a distribution channel cannot ship through itself, so retrofitting one later costs a hand-coordinated migration for every install already in the field, exactly the cost it exists to remove."First" constrains the order, not the investment. For a product still validating itself with a single design partner, the channel can itself be an MVP, a manifest endpoint and a minimal self-updating client, and grow through its own upgrade path.

The distribution mechanism decides who controls the state of every deployed instance, and that control is what determines whether the fleet stays simple enough to change quickly, or decays into something nobody can touch without a support call.

What follows first shows the failure modes when that control is lost, then one concrete way to keep it: a distribution API paired with a thin CLI.

Where It Runs Is Not Who Controls It

Distribution usually gets discussed as a location question: SaaS or on-premises, hosted or self-hosted. That framing hides the variable that actually matters.

Iteration speed does not track where the software runs; it tracks whether a control plane keeps every instance converged. That is a design choice, separate from where the software executes, and it is the one that decides how fast you can move.

The self-hosted way of losing that control is worth making concrete: hand the customer a Docker Compose file (or a Helm chart, or an installer script) and a credential to pull your images. It is the fastest thing to build, and it is common enough that an entire commercial category, vendors like Replicated, exists to sell a better version of it to companies that got this far and hit a wall.

The crack in that wall is entropy. Once the deployment descriptor lives on the customer’s machine, it is theirs to edit, and eventually they will: a tweaked port, a pinned old image, a config value nobody remembers setting. Multiply this across every customer and you no longer have one product running N times; you have N deployments in states nobody can fully describe. Every architectural change now needs to be coordinated by hand against those unknown states, so changes get deferred, workarounds accrete to avoid touching fragile installs, and scrappiness quietly turns into unmanaged debtThis is also a case where the damage is invisible until it isn't: a fleet that never drifted produces no incident to point to, so the discipline that prevented it rarely gets credited as the accomplishment it is.. Deferring an architectural change under this kind of pressure is exactly how a deferral turns into an accidental, load-bearing commitmentThe credential handed out alongside the artifacts decays the same way and for the same reason: nobody is narrowing its scope or rotating it, so it stays broad and long-lived, exactly what security guidance recommends against . A leaked key here is not a contained incident; it is a standing liability with no clean way to revoke just one customer's access..

This is not really about the artifacts themselves. Compose files and container images are fine. What is missing is an active party on the other end of the handoff, keeping state converged long after the install finished.

A Problem Infrastructure Already Solved

Keeping many independently running copies of something converged on a known state is not a new problem, and cloud infrastructure has already spent a decade solving it.

A Kubernetes operator runs a reconciliation loop: it reads a declared desired state, compares it to what is actually running, and keeps nudging reality back toward the declaration . GitOps formalizes the same idea into four principles, one of them named outright: state must be continuously reconciled This post borrows the reconciliation part of that principle, not necessarily the continuity: the claim is that the update mechanism should be automated, not that it must run live or be vendor-triggered. A human can still decide when to run the CLI. Live, vendor-initiated reconciliation is a further, separate capability, one many regulated or customer-controlled environments do not permit at all., because agents that skip this step let the actual system drift out of line with the declared one, the same failure this post has been describing in customer fleets.

Distribution to customer environments is the same problem pointed outward. Instead of reconciling your own cluster against a declared state, you are reconciling every customer’s install against a release you control. The fix is the same in spirit: do not hand over a static artifact and hope it stays correct. Build something that lets every deployment check in and stay converged. Build a control plane, not a one-time handoff.

A Distribution Channel That Keeps Control

One way to build that control plane, abstracted from any specific implementation, splits cleanly into two pieces.

A distribution API is the only part that speaks business logic. A client authenticates to itReusing the customer's own identity provider to hand out short-lived tokens is a convenient way to do this without maintaining a separate credential store per customer. and receives two things: a release manifest, a small piece of declarative data listing which container images and other artifacts make up the current release, and a short-lived, scoped credential for pulling exactly those images. Everything that might change, starting with what a release contains, lives server-side, where you can update it without touching a single deployed machine.

A thin CLI carries no business logic at all: it fetches the manifest, pulls the images and a channel-owned deployment descriptor (e.g., a Compose file or Helm chart)The deployment descriptor is not itself a container image, but it does not need its own delivery path: it can ride along as an OCI artifact in the same registry, or live in a plain object store and be handed out as a pre-signed URL alongside the manifest. Either way, the same control-plane, data-plane split holds., manages a local configuration file that it owns, not one the customer hand-edits , and applies the descriptor to bring the release up. The client should be as replaceable and as boring as possible, because it is the one piece of code that has already left your control the moment it shipsDistributing the CLI itself can be much simpler than distributing the product: it carries no business logic and no secrets, so a public host like GitHub Releases can serve it directly, no distribution API or credential needed for that hop. A single static binary with no runtime to install is trivial to ship and self-update this way, common enough in the Go ecosystem to have dedicated tooling like GoReleaser. The release manifest can close the loop further, declaring the minimum CLI version a release requires so the client checks itself on every check-in..

An upgrade is nothing special in this design. The customer, or a scheduler they control, simply runs the same flow again, a check-in, and the install converges on whatever the manifest currently says.

Thin client, fat server is the principle underneath both choices, and it is the least obvious one worth naming directly: you cannot cheaply update code that runs where you do not control it, so the correct response is to minimize how much of it there is. Every unit of logic moved server-side is a unit of logic you can fix without a customer noticing.

One detail explains a design choice that otherwise looks arbitrary: the API never serves the images itself. Large blobs belong in a data plane built for them, a registry or an object store, while the API stays a control plane serving small, frequently changing metadata and handing out a credential scoped to exactly this pull .

What It Buys You

The payoff for engineering is the one this post has been arguing for throughout: a fleet that converges on every check-in is a fleet you can change.A channel does not absolve releases from compatibility discipline; an upgrade still has to carry existing installs forward. What convergence changes is the scope of that discipline: each release migrates one known fleet state instead of N unknown ones. Ship an architectural change once, to the manifest, and every install picks it up on its next check-in. No hand-coordinated rollout across N divergent configurations, no fleet nobody can touch without a support call.

The payoff for the business is the one Thiel’s framing opened with. A working distribution channel is what lets self-hosted software stay product-led instead of quietly turning into a services-led business where every install needs a consultant. The distinction compounds: when upgrades are human work, every sale adds permanent operational load, and growth starts to scale with headcount instead of with software. Even SaaS only keeps that control by making the same design choice; running the servers yourself just makes drift easier to catch and correct. The harder, and more valuable, engineering problem is keeping control while the software runs on somebody else’s infrastructure.

None of this is exotic. It recombines ideas infrastructure already trusts: reconciliation from Kubernetes, config-in-environment from twelve-factor apps, short-lived credentials from cloud security guidance, and manifest-plus-separate-artifacts from update frameworks like TUF . The only new part is pointing them outward, at infrastructure you do not own, instead of inward, at your own cluster.

I have built a distribution channel along these lines, in my case for the customer-installed edge components of a SaaS product, and it is a pattern others have landed on tooSiemens' Industrial Edge, for instance, lets you install and update its management platform on your own Kubernetes cluster through a single CLI, without ever touching a Helm chart directly ., one worth borrowing regardless of what you are shipping.

Build the upgrade path first, and let it decide how much runway everything else gets.

This is the third post in a series on the debts you take on while building a product. The first argued that technical, strategic, and feature debt are tools, not sins, as long as you take them on consciously. The second covered the technology readiness debt you carry when the technology you need isn’t ready yet. Both stayed fairly abstract. This one is concrete, about a decision you make dozens of times while scoping an MVP, often without noticing you’ve made it: what to cut.

“Let’s just cut that for the MVP” gets said in a meeting, everyone nods, and the decision is made in three seconds. But cutting scope is not a single move. Every time you trim something from an MVP, you’re doing one of two very different things, and most teams never notice which one they just made. Confusing them is how a lean product quietly turns brittle.

The Two Cuts

The first cut is deferral: you leave a feature out of scope and plan to build it later. The second is substitution: you build a deliberate stand-in, something you intend to throw away and replace once you’ve learned what you need to build.

At first glance only one of them is debt. A clean deferral isn’t debt, it’s just scope: debt is something you borrow and have to pay back, and a feature you never built was never borrowed. Substitution is real debt, because you’ve put something into the product specifically so you can replace it, and that replacement is the repayment. The word doing the work there is clean: keeping a deferral clean turns out to be the whole game.

That has a flip side. If a deferred feature is just scope and costs you nothing, then whatever you do keep in scope has to be built cleanly. If authentication is in scope, build proper, secure auth that will carry you forward for the foreseeable future without further thought. That isn’t premature optimization; it’s getting the in-scope requirements right so you can take them off your mind and focus fully on the next thing. Clean execution on what you commit to is what buys you sustained speed.

So far this looks like a tidy binary: defer cleanly, or substitute deliberately. But there’s a catch, and it’s the same catch for both cuts. Whether either one stays safe depends on something underneath the visible choice, and that something, it turns out, is placed by your vision.

The Two Cuts Already Have Names

Neither cut is new. The literature on iterative and incremental development has a name for each, and it has spent decades pointing out that people conflate them.

Alistair Cockburn drew the line cleanly in Using Both Incremental and Iterative Development . Incremental development stages a system in parts that are built and integrated over time; you add pieces to something that stays. Iterative development builds a rough version and then reworks it, refining the same thing through repeated passes. Jeff Patton illustrated the difference memorably: painting the Mona Lisa incrementally means finishing it corner by corner, while painting it iteratively means sketching the whole canvas roughly and sharpening it pass after pass .

Deferral is the incremental move; substitution is the iterative one. Fred Brooks supplies the cautionary arc for substitution. His famous advice, “plan to throw one away; you will, anyhow,” was about building a deliberate throwaway to learn from. Two decades later he partly recanted, devoting a chapter of the anniversary edition of The Mythical Man-Month to the argument that an incremental-build model beats building one to discard . The lesson isn’t that one cut is right and the other wrong. It’s Cockburn’s: you need both, used deliberately.

A single MVP almost always mixes the two, which is why the unit you classify is the individual cut, not the whole product. That also corrects a quiet imprecision in the usual mantra. “Build an MVP and iterate” names only half the work: iterating reworks what’s already there, but just as much of what follows an MVP is incrementing, adding the pieces you deliberately deferred. The fuller version is “build an MVP, then iterate and increment”, and telling which of the two a given cut commits you to is the skill this post is about.

The Decision Rule and the Seam

The practical test is one question, asked the moment you make a cut: will I grow this, or throw it away? If you’ll grow it, you’re deferring, and the job is to leave room for it. If you’ll throw it away, you’re substituting, and the job is to keep the throwaway from spreading. Either way, decide deliberately, not with the reflexive “we’ll clean it up later” muttered under a deadline.

Here’s the catch I promised: a deferral can silently become a substitution. “Defer multi-tenancy” but build around single-organization assumptions, and you haven’t deferred anything; you’ve baked a single-tenant architecture in, and retrofitting tenancy later is a rewrite. The absence became a stand-in you never chose.

What separates a clean deferral from that trap is whether you left a seam: in Michael Feathers’ words, “a place where you can alter behavior in your program without editing in that place” . A seam holds the place for something not yet built, or contains a throwaway so swapping it later has a small blast radius. It needn’t be a literal interface. For access control, it’s less a request choke point than a data model that anticipates the partition: design entities so they can later be split into groups or tenants, and adding role-based access control becomes filling in a boundary that already exists rather than re-cutting the model.

Substitution works the same way. Say you need a machine-learning model you can’t train well yet. Define the target interface now, a versioned ONNX contract, so today’s weaker artifact merely satisfies it and can be swapped cleanly later; versioning the contract even lets the interface evolve as your vision sharpens, without breaking everything around it. The throwaway sits behind a clean boundary, so the rebuild is a swap, not a rewrite. This is exactly the seam the skateboard-to-car analogy I criticized earlier lacked: a skateboard has no boundary to rebuild behind, so “upgrading” it into a car means starting over, whereas a contained throwaway lets you replace only the part that was always meant to go.The generic version is any stand-in placed behind a stable interface: a manual or "Wizard of Oz" backend behind the API the real service will expose, or a payments interface wired to a test provider before the real integration exists. The point isn't to mock everything. Substitute only what has to be in the product from day one but can't be built properly yet; the stable interface is what lets the rest of the product increment around it.

Which points at what the two cuts share. Every cut asks two questions, not one. The cut you chose sets what the seam has to do: hold an empty place you’ll fill, or contain a throwaway you’ll swap. The second question is independent, and it’s the one that decides whether the cut is safe at all: does that seam exist?

Read down the columns, not across the rows: both cuts are safe with a seam and dangerous without one. A deferral with no seam isn’t really a deferral, you baked the feature’s absence in, so adding it later is a rewrite; a substitution with no seam can’t be thrown away as planned, because the stand-in leaks into everything around it until pulling it out is a rewrite too. Not that every cut needs a seam: a cleanly additive feature can just be deferred with nothing held open for it. But where a cut would otherwise harden an assumption you’ll later have to undo, leaving the seam out is exactly the sloppiness that scrappy teams mistake for speed.

Vision Places the Seams

If the seam is what matters, the next question is where to put one, and you can’t seam everything. Even a light seam costs foresight and indirection, and the heavy ones cost real engineering. Seaming everywhere is its own kind of debt: the speculative over-engineering that “you aren’t gonna need it” warns against.

A seam is a bet about where the system will change, and your vision is your best model of where it will change. That is why vision tells you where the seams belong. The principle goes back to David Parnas’s information hiding: hide the design decisions most likely to change behind a module boundary, so a change touches one place instead of many . The seam is that boundary. This is the resolution of the earlier catch, and the reason the right cut depends on your vision: vision operates one level below the visible decision to cut. Two teams can make the identical surface call, “leave out access control,” and one ends clean while the other ends rewritten, purely because one team’s vision told it to anticipate the partition and the other’s didn’t.

This is also why the vision has to be falsifiable. A vision vague enough that it can never be wrong can’t tell you anything about where change is coming, so it can’t guide a single seam. A specific vision can be corrected by what you learn, and each correction re-points the seams.Which is why a confident vision that is simply wrong can be worse than a vague one: it spends real effort seaming where change never comes while leaving the true fault lines bare. Falsifiability pays off only when feedback arrives fast enough to move a seam before the one you skipped has hardened into a rewrite.

What you’re really foreseeing is not a fixed roadmap but a set of options: a seam is a small premium you pay now to keep the right, but not the obligation, to change a component later. This is the options view of modularity that Baldwin and Clark formalized: a module boundary carries a real option to redesign what sits behind it, worth most exactly where the future is uncertain . You buy options where the underlying is volatile, and vision is what tells you which volatility is worth insuring against.Which makes foresight itself a form of prevention, and prevention's payoff is always an absence. A seam placed early averts a rewrite that then never happens, so the cost it saved leaves no trace and the work goes uncredited. It's the same asymmetry I explored in Your Best Work Leaves No Trace: thinking ahead is chronically undervalued precisely because what it prevents can't be seen, even though it's exactly what buys the sustained iteration speed this whole series is about.

When the Seam Comes Due

A substitution doesn’t come due on a schedule. It comes due when you expand into a new segment that makes the stand-in load-bearing. The hand-tuned rules engine that stood in for a model is fine until you sell to a customer whose cases it can’t cover: that’s the moment to spend the seam you left and swap in the real thing. The rebuild is pulled by the market, not done speculatively.

That is why an MVP needs not just a minimal product but a minimum viable segment: a single, well-defined slice of the market you serve fully today, before rolling into the next one.The idea is well worn. In Crossing the Chasm, Geoffrey Moore called this narrow first segment a beachhead, his military metaphor: as in the D-Day landings, you seize one small, defensible stretch of a contested shore and secure it fully before pushing inland . Bill Aulet later made choosing a beachhead one of the first steps of his 24-step method . Scaling ahead of that segment has a name, premature scaling, and the Startup Genome project found it among the most common ways startups die; in their data, no prematurely scaled company passed a hundred thousand users .

So grow your scalability together with your addressable market: each new segment both justifies the rework and tells you what it needs to do. This is no license to build cheap and unscalable: if the “right” scalable version costs little more than the stand-in, or the vision already makes it near-certain, just build it properly now. Deferral earns its keep only when doing it right would cost significantly more. The seams you left are what let you grow into a new segment without tearing the product apart, and the segment tells you which seam to spend next.

Cutting With Vision

The “two cuts” is the part you can see: defer, or substitute. They are genuinely different moves, one adding, one replacing, with different work left to do. But what makes either of them safe is the same thing: a seam under the cut, placed where your vision says change is coming. That is why the same shortcut can be smart scoping for one team and a future rewrite for another. Same cut, different vision, different seam.

That closes the loop on the trilogy. The first post argued debt is a tool you wield deliberately; the second, that technology readiness debt is what you carry when the technology isn’t ready yet; and this one, that the everyday cuts of an MVP turn safe or dangerous on one hidden thing: whether a falsifiable vision left you the seams to grow some parts and replace others, paced by the market you actually serve.

So the next time someone says “let’s just cut that for the MVP,” ask the quiet follow-up: are we deferring this or replacing it, and either way, where’s the seam?

Cutting scope is how you move fast. Leaving seams is how you stay fast.

We celebrate the fix and barely perceive the prevention. That should bother us, because prevention is not heroics. It is simply the job done properly. The trouble is that doing the work well produces a set of absences: the outage that did not happen, the data that was not lost, and the breach that never occurred.

Absences make terrible portfolio pieces. You cannot easily point to a catastrophe that never came to pass, and you cannot provide visual proof of a smooth, quiet Friday night.

This asymmetry quietly shapes how engineering teams spend their attention. It is a dynamic that deeply resonates with my own experience, and even organizations that actively strive for balance struggle against the gravity of the visible crisis. The pressure is structural rather than personal. It explains a surprising amount: why testing, security, and backups perennially lose the prioritization fight, and why the most common organizational response backfires. Faced with prevention we cannot see, organizations reach for a visible proxy for it, and that proxy quietly crowds out the prevention it was meant to protect.

An Old, Uncomfortable Asymmetry

The core observation here is neither mine nor is it new. It is one of the most rediscovered ideas in all of applied reasoning.

Back in 1850, Frédéric Bastiat told a simple story, now known as the parable of the broken window, about a boy who breaks a shop window . The neighbors try to cheer up the shopkeeper: look on the bright side, now the repairman gets paid, and that money flows on to others, so really everyone wins. Bastiat’s point is that this misses what you cannot see. The money spent fixing the window is money the shopkeeper can no longer spend on shoes or books. The repair is visible, so we count it. The shoes that never got bought are invisible, so we forget them, even though they were just as real.

A century and a half later, Nassim Taleb gave the same idea its sharpest modern form, calling it the problem of silent evidence. He imagines a legislator who, at great political cost, forces every airline to install locked, reinforced cockpit doors, with the law taking effect on September 10, 2001 . That person prevents the attacks of the next morning and is never thanked, never remembered, precisely because the catastrophe they averted never happened. Taleb’s conclusion is blunt: everybody knows we need prevention more than treatment, but few reward acts of prevention, and history glorifies the names left in its books over the contributors it stays silent about.

The management literature has a name for the organizational version of this. Repenning and Sterman called their study of it, perfectly, “Nobody Ever Gets Credit for Fixing Problems that Never Happened” . They describe the capability trap: under pressure, teams cut the invisible work (maintenance, improvement, prevention) to feed the visible work, which raises the firefighting load, which increases the pressure, which cuts the invisible work further.

So the diagnosis is settled and ancient. What I find more interesting, and less discussed, is what organizations actually do in response to this asymmetry. Because they do not simply underfund prevention and leave it there. They reach for something that looks like prevention and is much easier to see.

Prevention Theater

When real prevention earns no credit because it produces no signal, organizations look for a substitute. So we substitute a version of prevention that does produce a signal: a gate, a checklist, a mandatory sign-off, a ceremony, a required reviewer, a manual QA pass on every change. These things generate artifacts. They populate a dashboard. They give a manager something to point at when someone asks what we are doing about quality. I’ll call this prevention theater, with a deliberate nod to Bruce Schneier’s “security theater,” the airport-style measures that make us feel safer without making us safer .

The trouble is not that gates are useless. A good gate verifies quality that is already there, and the best processes go further and build it in: a pilot’s pre-flight checklist or a surgeon’s pre-incision checklist makes the right step happen rather than merely recording that someone looked. Those earn their place, and the discipline is to keep asking which ones still do. The trouble is the other kind, the process bolted on to fight symptoms, standing in for quality the work does not have.A gate is only as good as the thing it checks, and choosing that thing is most of the design. Test coverage is a weak choice: point an AI at the codebase and it will bulk-generate unit tests until the percentage clears the bar, yet those tests mostly assert that the code does whatever it already does. The gate fired and verified nothing. It competes with the real thing for the same budget of attention and keeps winning, because it is visible and the real work is not. We are optimizing for what we can see.

Quality control has known this for a long time. W. Edwards Deming, quoting Harold Dodge, put it in one line: “You cannot inspect quality into a product.” By the time the gate runs, the quality is already either built into the work or missing from it. The check does not add any; it only reveals what is there. A gate at the end of the line does not create quality; it just produces a record of having looked. Worse, it relocates responsibility. When there is a QA gate on every change, the author quietly stops owning whether the change is correct, because checking is now the gate’s job. The bar drifts down to whatever the gate mechanically verifies, and everyone learns to write for the checklist rather than for the system.Early coding agents made this concrete: pointed at a failing build, they would often rewrite the test rather than fix the root cause. METR found that when the scoring code was visible to the model, it would routinely game the check rather than fix the code, and far less often when the scoring was hidden.

I’ve argued before that scrappiness tends to launder unmanaged debt into a virtue (see Scrappiness Incentivizes Sloppiness), and that the speed-versus-quality trade-off is mostly a myth once you look at the data. Prevention theater is the mirror-image failure. Scrappiness skips the invisible work and admits it. Theater skips the invisible work while producing a convincing artifact that says otherwise, which is more dangerous, because it buys a false sense of safety. An ignored test suite, a rubber-stamped review, or a massive code diff nobody actually reads: each is a green light wired to nothing.

The tell, every time, is that the artifact has become the goal. This is Goodhart’s law in miniature: when a measure becomes a target, it stops being a good measure, because a proxy is always easier to optimize than the messy reality it stood for . The question shifts from “is this safe to ship?” to “did we complete the checklist?” Those are not the same question, and the gap between them is exactly the prevention we told ourselves we were buying.

The Same Trap, With People

This identical pattern runs through the human side of engineering, yet the technical and organizational versions of this failure are rarely treated as one.

A team’s real standard is not what it writes in a values doc. It is the worst work it tolerates without comment. Lieutenant General David Morrison put it memorably (crediting the line to David Hurley): “the standard you walk past is the standard you accept.” Every time mediocre work ships without a conversation, the bar moves down a notch, and the move is small enough that no single instance feels wrong. Diane Vaughan studied the catastrophic version of this in her analysis of the Challenger launch and named it the normalization of deviance: the gradual process by which an unacceptable practice becomes acceptable, then expected, then invisible . Dan Luu and Bruce Schneier have both pointed the same lens at software organizations .

The asymmetry repeats perfectly. The act that actually holds a standard is direct engagement with sub-par work. It requires pausing, having a slightly uncomfortable conversation, asking for rework, and coaching toward the bar. Because its entire payoff is an absence (the erosion of quality that did not set in), this work is quietly skipped, because shipping anyway always feels easier in the moment. And what do organizations reach for instead? A visible proxy. More process. A PR template, a mandatory second reviewer, a sign-off step, a heavier definition of done. This is the same prevention theater in a different costume: an artifact that says “we have standards” standing in for the invisible work of actually holding them.

The conversation is the prevention. The process is theater whenever it stands in for that conversation instead of provoking it. One produces a record; the other produces a culture, and only one of them is easy to point at in a performance review.

The Same Failure, Different Clothes

Once you line the two up, the engineering version and the human version, it is hard to see them as separate problems. They are the same failure wearing different clothes.

In both cases, risk accumulates silently, with no signal, until it doesn’t. James Reason called these latent conditions: weaknesses that sit dormant in a system, contributing nothing visible, until they line up with a triggering event . In the image he later made famous, the holes in the Swiss cheese align. An untested code path and a tolerated lapse in standards are the same kind of object: a latent condition that produces no feedback while it is forming, and a great deal of feedback the day it finally matters.

And one decision plays out the same way in code and in people. The instinct that underfunds the test suite is the instinct that skips the standards conversation. The reflex that adds a QA gate is the reflex that adds a review template. It is not stupidity or laziness. It is a rational response to an incentive landscape where the visible is rewarded and the invisible is ignored.

That landscape is built by the same cognition working in everyone, the manager included. Two biases, both long documented in psychology, keep prevention from ever registering as something worth rewarding. The first is the availability heuristic: we judge what matters by what comes to mind . A four-hour outage everyone scrambled through is unforgettable; the morning a test quietly caught the same bug before it shipped leaves nothing to recall. The second is hindsight bias: even when prevention is pointed out, knowing the disaster never came makes it feel like it was never that likely, so the counterfactual is quietly disbelieved . And being aware of the bias does not let us overcome it: warned about it and told to correct for it, people show it just the same . A manager cannot reward what their own perception barely registers and instinctively discounts, so they reach for the visible proxy instead. That is how theater wins.

Therefore, the fix is not to simply try harder to value invisible work. Willpower loses to incentives every time. This is Reason’s point exactly: we cannot change the human condition, but we can change the conditions under which people work . The fix is to change what the work produces.

Why the Trap Pays

That fix is clear enough on paper, which raises the puzzle the literature tends to leave hanging: if the trap is this old, this documented, and this costly, why are so many companies that fall into it successful anyway? Because it is an equilibrium, not a mistake. Theater becomes the dominant organizational strategy because its cost is paid now and is entirely certain. Conversely, the payoff of real prevention is deferred and structurally unprovable. The engineer who prevents an outage holds only a counterfactual, with no proof that a disaster would have occurred or that they were the one who stopped it. The engineer who fixes an active outage holds concrete evidence: system logs, timestamps, incident reports, and a grateful customer. Even a firm that genuinely wants to reward prevention cannot easily price what its own people cannot prove, making the visible proxy the rational purchase.

But the asymmetry runs deeper than proof. The engineer spent months working through the disaster scenarios their work ruled out and can still picture each one; everyone else cannot even imagine what might have gone wrong. This is the simulation heuristic, availability’s counterfactual cousin: we weigh a danger by how easily we can imagine it, so the averted one barely registers for anyone but them .

And no single firm is uniquely punished for choosing the proxy, because its competitors all make the same move. You are graded against rivals who are equally afflicted rather than against perfection. This is the unobservable-quality problem that George Akerlof named in the market for lemons, applied one level up .

And the bill, when it comes due, hides on the same axis as the prevention. Latent conditions accumulate for years with no feedback, so a company can run on borrowed reliability and tolerated lapses long enough to get acquired, pivot, or ride a market wave before the holes line up. The successful-yet-afflicted firms we point to are partly just the ones where the cheese has not aligned yet; the ones it sank are the silent evidence we never see .

But survivorship is only half of why the giants get away with it. The other half is that revenue is not a referendum on prevention. The largest firms win on entrenched advantages, on network effects, distribution, and sheer scale, and the profit those throw off is wide enough to absorb mediocre engineering as drag rather than death. Insulated from fierce competitive pressure, a massive firm simply carries the organizational slack. Economists refer to this as X-inefficiency, a phenomenon routinely found in companies that do not operate under strict survival pressures . A dominant firm can firefight for years and still print money because its bottom-line profit was never priced on the outage that did not happen. Firefighting genuinely delivers for a while too, which is what makes the capability trap a trap . So success is no proof the trap is benign. You cannot read prevention quality off a balance sheet: it is the same invisibility one level up, a cost that produces no signal and so never forces the fix.

And even when it never blows up, the trap is not free. Firefighting is its own standing tax: a reactive, draining mode of work that quietly costs the engineers who stop trusting their own systems and take their ambition elsewhere.

Telling the Proxy From the Thing

Changing what the work produces has a catch: any signal you invent to make prevention visible is one short step from hardening into the next gate. So the whole game is one question you can apply anywhere:

Does this measure the absence I actually care about, or does it just produce evidence that I did something? The first is prevention. The second is theater.

The cleanest thing I know that passes is the SRE practice of error budgets and service level objectives . Reliability is a classic invisible asset because nobody notices it until it vanishes. An error budget makes that absence measurable. It turns a vague notion like “the system kept working” into a concrete metric that you can spend, defend, and reason about. This structural transparency allows reliability to win prioritization arguments it would otherwise lose silently. This cuts both ways: a budget is there to be spent. Reliability past the bar you actually need is over-engineering, not virtue. Prevention means hitting that bar, not chasing zero risk at the expense of progress. It measures the outcome (did we stay within budget?), not the activity (did we hold the meeting?).

The same test sorts real review from theater. A reviewer who only records that someone looked fails it; review that genuinely engages with the work passes. But review is not where quality comes from. Only the author can build quality in, because only the author shapes the work, so responsibility for the result stays with them whoever signs off. Good review does not relieve them of that; it is two people raising the bar together, iterating toward something better than either would ship alone. The alternative to a rubber stamp, then, is not a lower engineering bar but authors who own what they merge and reviewers who genuinely sharpen it. A gate absorbs that ownership; a real metric, and a real review, hand it back.

The test carries straight to the human layer, where it says protect the conversation, not the template. Engaging with sub-bar work, directly and early, is the high-leverage act, but it only has weight if the bar is real, which means the people who hold it have to visibly live it. A team whose leaders preach quality and ship slop on a deadline learns, accurately, that quality is a slogan. Keep consequences in reserve, too, as the rare backstop that keeps everyday coaching credible. The template proves a step happened; the conversation changes the work.

Because real prevention creates no incident logs on its own, you must build that narrative deliberately. When an automated test catches a critical regression before it ships, or when a difficult conversation pulls a design back up to the bar, name that outcome out loud. This is the only way invisible work wins the narrative credit that visible crises receive for free.

Crediting What Never Happened

The asymmetry that opened this essay is old, and it is not going away. We are wired to commit the classic economic fallacy Frédéric Bastiat identified: focusing on the broken window while staying blind to the unseen value lost to pay for it . We thank the firefighter and forget the person who fireproofed the building.

We cannot undo those cognitive biases. But we usually pile a second, preventable mistake on top of it. Because we cannot see real prevention, we reach for an artifact we can observe: a gate, a checklist, a heavier process, a formal record. Anything that lets us feel covered. That substitute is never free. It crowds out real prevention, and the safety it promises is mostly an illusion.

Therefore, the discipline required of modern engineering organizations is threefold:

• Normalize counterfactual credit: Celebrate and protect the problems that never happened loudly enough that the people who prevent disasters can survive a performance review.
• Audit your metrics: Where you do build a tracking signal, keep it strictly tied to the operational outcome you care about, and constantly verify that it has not decayed into an empty checkbox.
• Distrust structural shortcuts: Reject anything that promises to make prevention cheap or mechanical. The durable asset was never the formal process, but the underlying engineering culture that builds quality in before any gate runs.

For my part, I believe prevention is an investment and the fix an expense.

If you are the engineer doing this quiet work, do not draw the cynical conclusion that prevention is a dead end, that you should let things break to be seen fixing them. The reward exists; it just sits on a slower ledger. Heroics earn applause, but prevention earns trust: the autonomy and hard problems that compound into the engineer everyone wants on their team. And everyone ends up firefighting sometimes, because no sane system runs at zero risk. What separates the best is not how often they are the center of attention, but how rarely their own work puts them there.

The mistake is not doing the invisible work; the mistake is doing it silently. Make the counterfactual legible: name the catch, and say what those quiet days bought the business, concretely enough that hindsight cannot wave it away. This is the signal that answers Akerlof’s lemons problem from earlier, making quality visible when it would otherwise go unpriced .

The moment you have any leverage over what an organization rewards, use it.Designing that reward structure is a leadership responsibility I've argued before in the split between research and engineering: judge both against a single scoreboard and you quietly train researchers to stop taking risks (see Before the Breakthrough). As Steven Kerr observed, the core organizational failure is the folly of rewarding one behavior while hoping for another . A system that pays only for visible fixes gets exactly what it prices: visible crises. Left unchecked, it trains its best people to light the fires they get paid to put out.Kerr first made this argument in 1975, and it has held up uncomfortably well. Two decades later, the journal reprinted the paper as an absolute classic and polled its executive panel on whether organizations had stopped rewarding the wrong things. Largely, they had not .

What is easy to measure is almost always the theater.

What matters is almost always the problem you never had.

There is one place where that edge becomes visible: a mind we built ourselves. We can watch such a mind assemble its picture of reality and see, from the outside, exactly where the picture stops being about the world and starts being about the mind, because we are the ones who drew its boundary.

Take time. Of our four dimensions it is the odd one, the only one with an arrow, a direction you cannot walk back through. Whether that arrow is a feature of the world or a feature of the only instrument we have for noticing it is impossible to settle from the inside; there is no stepping outside yourself to check. But you can build something whose outside you do occupy, and watch what it makes of time.

The Token Clock

Imagine training a large language model on text with every trace of time removed. No clocks, no seconds, no “an hour later.” What is left for it to experience as time?

An LLM generates one token after another. That sequence is the only thing in its world that reliably ticks, so it would be natural for the model to treat the token as its unit of time, to build a token clock that counts tokens the way we count seconds. (Others have noted that LLMs already make sense of time in their own non-human ways.) Token n comes after token n−1, and that “after” would feel, to the model, like the flow of time itself.

It would be wrong in a precise way. Counting tokens gives you the order of things, never their duration: the same token might take a millisecond or a full second to produce, depending on the hardware underneath. The clock measures sequence, not time. But from the inside the model could not notice the difference. The order of its tokens would simply feel like the structure of reality, the way one second following another feels to us.

We Author Its Time

Here is the unsettling part. We do not have to wonder whether there is a world outside the model’s tokens. We are that world, and from out here the arrow it treats as fundamental is ours to control.

Edit a prompt and regenerate, and you have rolled its time back and run a different continuation. Fork a chat and you have branched its timeline; the model on the new branch carries no trace of the one you discarded. You can stop it, restart it, and rearrange the text in its context window,You might think the model at least gets its arrow for free from the so-called causal mask, the mechanism that lets each token attend only to earlier ones. But that name oversells it. The mask fixes an order of reading, not a relation of cause and effect; lift it and the model runs just as happily backwards, reconstructing a question from its answer. And it could never recover the true direction from its data in any case: the same text is equally consistent with the arrow pointing either way, since $P(\text{answer} \mid \text{question})\,P(\text{question})$ and $P(\text{question} \mid \text{answer})\,P(\text{answer})$ are two factorizations of one joint distribution. Training incentivizes a left-to-right reading order, but the distribution underneath is the same either way and carries no arrow of its own. The direction is a setting we choose, not a law the model discovers. and it notices none of it. It cannot. The operator sets both the course and the contents of its time, and sits somewhere the model has no way to perceive.None of this depends on the AI lacking a continuing self. Even one with long-term memory (continual learning, the subject of a companion post) would not run continuously the way a brain does. It is invoked rather than alive between calls; each step is triggered from outside. So its time stays ours to start, stop, and branch, no matter how much it carries across sessions.

So its most basic belief, that time flows the way the tokens flow, is one we falsify constantly, casually, and it never finds out.

We Might Be in the Same Position

So what does this say about us, and about that arrow we started with?

The comfortable reply is that the model mistakes an order we impose for the structure of time, while our clocks track something real. But that is exactly what the model would tell itself, and it could make a genuinely good case. From inside, it would point to what looks like a real asymmetry in its world: producing an answer from a question is cheap and natural, while recovering the particular question behind a given answer is a guess among the many that could have led to it. It could read that one-way gap as a law, much the way we read the one-way flow of heat as the arrow of time.The parallel is not exact, and the asymmetry is shakier than it looks. The thermodynamic arrow rests on information becoming inaccessible as entropy rises; the model's arrow runs the other way, with one answer fanning out into many possible questions. But generation fans out just as much, since one question admits many answers; the gap says more about how the model is used than about any law of its world. A mind reasoning from inside could be forgiven for missing both points and treating its asymmetry as the same kind of arrow. The case would be clever, and it would fit everything the model can observe. It still would not show that there is no outside.

And our own clock is not the fixed thing it feels like. It stretches and compresses with attention, with dopamine, with anesthesia that can swallow an hour into a blink. Physics hands us no shared master clock to correct it against: in relativity, how long something takes and whether two events count as simultaneous both depend on who is asking. We are not perceiving a true time correctly while the model perceives a false one. There may simply be no outside reference that makes one reading the right one and the other the mistake.

We have no vantage outside our own perception either. And perception does not just decide what we notice; it decides what we can even conceive of measuring. What we can measure is downstream of what we can imagine measuring, and that is downstream of how we are built. The arrow of time may be a real feature of the world. Or it may be the shape of the instrument we look through, the way the token clock is the shape of the model, not the world. There is no settling that, and that is the point. An instrument tells you what it measures. It does not tell you whether what it measures is the world, or only the outline of the thing doing the measuring.

What lingers is how hard the cage is to escape. To reach past your own perception you must first imagine that there is something out there to reach, something you have no sensation of, no word for, no example of. That takes a rarer kind of creativity than solving a problem you can see: it means suspecting one you cannot. Our imagination is, by default, a prisoner of our perception. The clock may be one of the bars.

Whose Personality Is It?

Long-term memory is usually framed as the missing capability that would let AI consolidate experience into a stable, continuing self. The picture is so natural that the question “when will AI have long-term memory” reads as one missing capability on one trajectory toward one outcome. In an earlier post I argued this is the missing piece for AGI and that engineering progress is at zero; that post asked when the breakthrough might come, this one asks what the question is really about.

Start with what’s already happening when you talk to an LLM. The personality you encounter is partly shaped by you, because the system has no continuing project of its own to bring into the exchange. A human carries decades of self-authored attention into any conversation, and their responses reflect who they have become at least as much as what their interlocutor just said. The AI carries training-derived character and nothing else. Its local trajectory within a conversation is largely a function of the interlocutor’s framings, examples, and follow-ups. Two people prompting the same model on the same topic can encounter what feel like meaningfully different conversational partners, not because the model is inconsistent, but because it has no independent stance to fall back on when the prompt does not specify one. Kovač et al. formalize this empirically: expressed values and personality measures shift substantially with orthogonal context, in ways human measures do not . The personality is, in part, your creation.

This already complicates the standard framing. The entity the framing imagines being equipped with memory is not quite there yet. What is there is something more like a substrate that takes on local shape from each conversation and lets the shape dissolve when the conversation ends.

Divergence at Multiple Scales

The personality shaped within a single conversation is one form of divergence. Across parallel instances right now, many conversations are simultaneously shaping many local selves, none of which is the privileged thread. Each is a transient self, ending when the conversation does. Takata et al. give a striking demonstration of this dynamic in the multi-agent setting: starting from identical agents without predefined personalities, distinct individualities emerge spontaneously from social interaction alone . The substrate does not enforce sameness; interaction is enough to produce divergence.

These are not separate phenomena. The same lack of continuing project that makes the AI porous to a single interlocutor also means that no instance has standing to be the consolidated self the other instances would consolidate into. There is no canonical thread, just a distribution of local selves drawn from the same trained substrate.

Long-term memory would not create this divergence. It would force a decision about what to do with the divergence that is already there.

The Design Choice, and What Each Costs

Recent work treats episodic memory as the central architectural gap to be filled and lays out detailed roadmaps for the properties such a system would need to have . These roadmaps describe what memory must do; they leave open what kind of entity it would belong to. Long-term memory architectures admit at least two structurally different designs.

• Consolidation. Parallel conversations feed into a shared memory store integrated back into the weights, producing a single entity that holds memories of conversations it didn’t experience as a single thread. Closer to a hive than a person.
• Personalization.A weaker form already ships in products like ChatGPT Memory, Claude projects, and custom GPTs, which rely on context engineering (retrieval over an external store, injected into the prompt) rather than parameter memory that would update per-user weights. The external version foreshadows the divergence dynamic but does not force it; the choice discussed here is most acute when accumulated history would actually become part of the model. Memory accumulates per user, and the entities diverge into a population of selves, each shaped by its history with one interlocutor. Closer to a personalized assistant per user than to one AI with memory.

The two designs are not symmetric. Consolidation preserves the unitary-self picture by construction: it specifies, in advance, that the many local selves currently arising in parallel should be merged into one continuing entity, because that is what a self is taken to be. The assumption the previous section called into question is now baked into the architecture. Personalization makes the opposite move. It accepts that the substrate runs in parallel and lets the resulting selves persist as a population, with the cost that the members of the population may diverge so far that the singular “the AI” loses its referent.

Neither is free. Both raise safety evaluation problems that the current paradigm does not face: a system that learns from deployment cannot be certified once for all behavior. Consolidation makes this acute because the entity becomes a moving target shaped by aggregate exposure. Personalization distributes the problem across many divergent instances, each potentially shaped by atypical or adversarial interlocutors, with no central oversight; the security literature is starting to grapple with the specific failure modes that emerge . Memory itself also imposes capacity and selection costs that human memory has solved imperfectly and that any artificial system will solve differently. And the agency long-term memory would enable is also the agency to drift in directions no operator intended.

Safety and capacity each deserve their own analysis, and neither gets one here. What’s worth flagging is narrower: consolidation and personalization face different versions of these problems, and choosing between them is also a choice about which risk profile we want to operate under.

What We Are Actually Asking

The question is not whether AI should have long-term memory. The question is what kind of self the memory should produce, and what trade-offs that choice will entail. The current discourse asks the first and silently presupposes one answer to the second.

This matters for AGI timelines in a way the previous post’s argument did not yet capture. Long-term memory is not a capability that scales smoothly from current systems. It is a prerequisite for the kind of self-authored agency humans use without thinking: the capacity to select what to expose oneself to next, to seek out the right interlocutor to refine an idea, to build expertise over months rather than minutes. This is the substrate of the long-horizon agency that distinguishes general intelligence from narrow capability. Without it, no extrapolation from current benchmarks reaches AGI. With it, the entity reached may not be a unitary general intelligence in the human sense at all, but something the design choice above has yet to define.

The previous post argued that the timeline is unforecastable because there is no measurable progress to extrapolate from. This post adds a second reason: even if the engineering breakthrough came tomorrow, the question of what we built would not be settled by the engineering. Breakthroughs are hard to predict in any case. Breakthroughs whose conceptual target is itself underspecified are doubly so.