Reflections 2nd August
When the walls move
Eighteen months ago I set myself a small question about AI. The loud questions were being asked well enough by other people: the financing, the jobs, and whether any of it is safe. Mine was simpler: since the technology is not going away, how do we use it, and when do we use it?
It has been amongst the most interesting work I have done for a long time, and the most frustrating, because the ground will not stay still long enough to be mapped. You build a way of working with the thing, test it, find where it breaks, and get close to being able to say something useful about it. Then a new release lands and what you learned describes a tool that no longer exists in that form.
So, for a while I just ran faster, but what I eventually noticed is that I had been standing on a wall, taking my bearings from it, but that the wall had moved. Not once, and not by accident; by design.
Once you see that, the question changes. If the wall is going to keep moving, the bearings we take have to come from somewhere else. They have to come from us, where we find ourselves, and not a proxy.
Suppose, I thought, AI disappeared from the planet tomorrow, what would actually change?
A great many investors would be extremely, if not terminally annoyed. In some fields the loss would be real and measurable; protein folding and drug discovery would slow, and that matters a great deal. Elsewhere, though, the ordinary business of being human and doing what humans do would carry on much as before. More slowly in some places, more accurately in others. Removing the technology would not subtract a fixed quantity of output. It would restore slower and more manual ways of working, and it would also remove an emerging category of confident mistake that exists only because a fluent machine made it seem credible.
This is not a Luddite position. The technology is remarkable, and much of it will prove to be of enormous benefit, and the counterfactual tells you where the value actually sits; at the edges: in the thinking that decides what to ask of the AI, and in the judgement that evaluates what comes back. The middle, the part everyone is excited about, is on its way to becoming a commodity. Chinese open source models are only months behind the frontier, and that gap looks likely to close. And in many cases, the open source models are good enough to make a big enough difference.
I also think that we find it very hard, given the way we have arranged our economies, to recognise when we have enough. AI is a more-than-enough technology, and we have no institutional grammar for that.
Which brings me to back to the walls, and why I think they are the real story. An organisation used to be a place with an edge. You could tell where it stopped and you began. That edge did a great deal of quiet work: it told you what was your business and what was not, whose judgement counted on which question, and when you had done enough to go home. James Scott’s argument in Seeing Like a State is that institutions make the world legible in order to govern it, and that the simplification is never neutral. Part of what they made legible was you. In exchange for being legible, you were told where you stood.
That edge is going soft, and it was going that way before any of this arrived. Some of it is technological. Work gets distributed, teams assemble across firms, and the tools a person uses at home are now routinely better than the ones they are allowed at work. Some of it is economic. Firms have spent thirty years converting employment relationships into transactional contracts and overhead into somebody else’s problem.
Consulting is a visible case. AI is hollowing out the junior layer by automating research, analysis and slide production, and that pushes firms away from the billable-hours pyramid towards flatter teams and outcome-based pricing. The job losses are the part that gets reported. The part worth remembering is that the pyramid was also the training system. Take it out and the firm has no obvious way left of manufacturing the judgement it sells. And for consulting, read any business that improved its margins by treating people as a commodity and now proposes to replace the commodity with software. Banks. Games studios. Agencies of every kind.
So the walls move, and they are becoming permeable at the same time. The institution that used to tell you where you stood is no longer standing anywhere in particular itself.
At the individual level the picture is close to the Wild West. Different people, different models, different purposes, very little of it examined, and all of it conducted in the relentlessly agreeable, empathetic register the models are tuned to produce.
The research supports the assertion. Lee and colleagues at Microsoft Research, surveying knowledge workers, found that higher confidence in the tool went with less critical thinking, while higher confidence in one’s own expertise went with more. The effort does not disappear. It moves, from producing the work to overseeing it, and it only moves successfully for people who had the judgement to start with. METR’s usage survey arrives at the same place from the other direction: reported gains run well ahead of gains anyone can verify, once you account for checking, rework, and output that looked finished and was not.
The human-like cues make this worse. First-person language, apparent empathy, the rhythm of ordinary conversation; these raise perceived accuracy quite independently of actual accuracy. People are not, on the whole, confused about whether the thing is sentient. They are poorly calibrated about whether it is right.
For anyone old enough to remember it, the business has the flavour of double glazing in the 1980s. A plausible man at the door, a product that was genuinely useful, a demonstration that was hard to argue with in the moment, and a price that made sense only because nobody in the room was equipped to evaluate the claim. The product was not a fraud. Some of those windows are still in. The failure was in the conversation, in the absence of anyone able to ask the second question.
Frank Knight drew the distinction we need here more than a century ago. Risk is the situation you can put a number on. Uncertainty is the situation you cannot, and Knight’s point was that judgement is the faculty that works in the second case and earns its keep there. John Kay and Mervyn King take the same ground in Radical Uncertainty and add the modern failure: when we cannot compute, we build something with the appearance of a computation and trust that instead.
Which is precisely what a fluent machine offers. Not an answer. The shape of one. So the discipline is unglamorous, and it comes down to two habits. Think critically on the way in, about what is actually being asked and whether it is the right question. Evaluate hard on the way out, about whether the polished thing in front of you is true. Neither habit is new. Both are what people who are good at their work have always done. The difference is that the organisation used to do some of it on your behalf, and it no longer is.
That leaves the question of who is served by a wall that keeps moving. I have come round to thinking that the greater risk to how we think sits less in the technology than in the scale of the organisations now proposing to administer it: centralised, process-bound, governed at a distance, and increasingly incurious about anything they cannot count. Ivan Illich called the point at which a tool stops serving its users and begins to require that they serve it the second watershed. He was writing about schools and medicine in 1973, and the test he proposed still works. Ask whether the tool enlarges what a person can do on their own account, or whether it enlarges the institution’s claim on them.
Both are still on offer. What decides between them is not in the model; it is in us, and specifically in whether we are prepared to keep hold of the harder part of the work: knowing what we are asking, and being able to tell whether the answer is any good.
That is not a defensive crouch. It is closer to the opposite. If the walls are going to move anyway, what matters is our perspective:
We shall not cease from exploration And the end of all our exploring Will be to arrive where we started And know the place for the first time.
T. S. Eliot, “Little Gidding”, Four Quartets
Appendix: sources
On the consulting model and firm economics
AI Is Changing the Structure of Consulting Firms, Harvard Business Review, 2025. The clearest single account of the pressure on the pyramid.
How AI is depressing entry-level wages and hiring, IESE. Evidence that AI exposure is linked to weaker demand and pay at the junior end.
On judgement and over-reliance
Lee et al., The Impact of Generative AI on Critical Thinking, Microsoft Research, 2025. Confidence in the tool suppresses critical thinking; confidence in one’s own expertise restores it.
Measuring the Self-Reported Impact of Early-2026 AI on Productivity and Value, METR, 2026. The gap between reported and verifiable gains.
On whether this time is different
Is generative AI a General Purpose Technology?, OECD. The case for GenAI as a distinctively broad general-purpose technology, and why the payoff still depends on diffusion and complementary capability.
The New Future of Work Report, Microsoft Research, 2025. On how AI changes the shape of work rather than only its volume.
From my bookshelf
Frank Knight, Risk, Uncertainty and Profit (1921). The founding distinction between what can be calculated and what must be judged.
John Kay and Mervyn King, Radical Uncertainty (2020). What happens when we substitute the appearance of calculation for judgement.
James C. Scott, Seeing Like a State (1998). Legibility, simplification, and what institutions require of the people inside them.
Ivan Illich, Tools for Conviviality (1973). The second watershed, and the test of whether a tool serves its user.
Iain McGilchrist, Ways of Attending (2018). Attention as the thing actually at stake.