Gell-Mann amnesia
You catch the agent's mistakes where you are an expert and trust it everywhere else. Its confidence is uniform. Your ability to check is not.
The writer Michael Crichton noticed something about how we read the newspaper, and named it, half as a joke, after the physicist Murray Gell-Mann. You open the paper to a story on a subject you happen to know cold. You see it is wrong. Not a little wrong, but backwards, the reporter has confused cause and effect, quoted the wrong number, missed the thing that actually mattered. You shake your head. Then you turn the page to a story about something you do not know, foreign policy, the economy, a court case, and you read it as though the paper had suddenly become reliable. You forget, in the turn of a page, what you just learned about the source.
The effect, and why it is so hard to feel
Crichton called it the Gell-Mann amnesia effect, and the joke in the name was that Gell-Mann had little to do with it, the physicist's name just lent the idea a borrowed authority, which is itself a small demonstration of the point. The mechanics are simple and they are about you, not the newspaper. Your trust in a source is supposed to track how accurate the source is. Instead, it rises or falls with whether you catch an error, and catching one depends on how well you can check. Where you are an expert, you catch the errors and your trust drops. Where you are not, you cannot catch anything, so nothing lowers your trust, and you read on, reassured by the very silence that should worry you.
The source did not get more accurate when you turned the page. Your ability to audit it fell to zero, and that felt like reassurance. This is not a mistake careless people make. It is a mistake everyone makes, because there is no signal to warn you. Being unable to spot errors feels exactly like there being no errors.
Now watch yourself review your agent
A coding agent hands you a change in a part of the system you know well. You read it the way you read the newspaper story on your own subject. You see the off-by-one, the assumption that does not hold for your data, the helper it reinvented instead of reusing, the name that will mislead the next person. You fix it, maybe with a little amusement at how confidently wrong it was. Good. That is review working.
Then the same agent hands you a change in a corner you do not know well. The billing edge cases. The retry logic in a service you inherited. The infrastructure you touch twice a year. It reads fluently. It compiles. The tests, such as they are, pass. And you approve it, because nothing in it tripped an alarm.
But the agent did not get more careful on the second change. Its confidence was no different. Its odds of being subtly wrong were no lower, and in an unfamiliar, load-bearing corner they may well be higher. The only thing that changed was you, and specifically your ability to catch it, which fell to zero exactly the way it did when you turned the newspaper page. You just had a Gell-Mann amnesia moment, and the code went in.
The trap is that trust and verification point opposite ways
Here is the uncomfortable shape of it. The agent writes with roughly uniform fluency across your whole codebase, in your domain and outside it, in the parts that are well understood and the parts that are load-bearing and strange. Your ability to verify that code is the opposite of uniform: high where you are expert, and collapsing everywhere else.
So your trust, which rides on whether anything set off an alarm, ends up highest exactly where your ability to check is lowest. The places you wave through fastest are the places you understand least. That is not a risk at the margins. That is the center of the thing, and an agent that can touch every corner of the system in an afternoon turns it from an occasional lapse into a daily one.
Why you could catch it at all
It is worth asking why you caught the error in your own domain, because the answer points at the fix. You caught it because you were holding context the agent did not have. You knew why the code needed to handle that case, what the number was supposed to be, which constraint was really binding. That knowledge is mostly tacit, and it is exactly what the agent was missing when it reached for a default that no one chose. In your domain, you supplied the missing context yourself, at review time, out of your own head. That is why the review worked.
Outside your domain, you cannot supply it, because you do not have it either. It lives in someone else's head, or in a decision made three years ago, or nowhere at all. So the agent guesses, and you nod, and the guess ships, and it is the information gap doing its quiet work in the one place no human in the room can see it.
The fix is not to trust less
The obvious response is to trust the agent less and review harder. It does not work, and not only because you do not have the hours. Reviewing harder in a domain you do not understand does not manufacture the understanding. You can stare at the billing edge cases all afternoon and still not know which one your biggest customer depends on, because that fact was never yours to know. Vigilance cannot close a gap that is made of missing information.
What closes it is putting the missing context where the agent can use it, so the agent's work in the domains you cannot personally audit is anchored to your team's real decisions instead of the internet's average guess. That is the job Brief does. It is a product navigator: it holds the decisions your code cannot, the real constraint on the billing path, the real reason the retry logic is shaped the way it is, the real limit that a contract set, and it puts them in front of the agent at the fork. It does not ask you to become an expert in every corner before you are allowed to trust anything. It makes the agent's output in those corners deserve the trust you were going to extend anyway.
Why the enterprise is one long turned page
An enterprise is a stack of domains no single person understands end to end. Everyone is an expert in a narrow slice and a trusting amnesiac across the rest, and that has always been true. It was survivable when changes were slow and each one passed through the few people who did hold the relevant context. Now an agent writes across every slice at once, at a speed no chain of reviewers can gate, and each change lands in front of someone whose ability to check it is mostly a function of luck. The enterprise does not have a Gell-Mann amnesia problem occasionally. It is a Gell-Mann amnesia machine, and it just handed everyone a tool that fills every page at the same time.
What this does and does not claim
A few limits, because the effect is about calibration, not doom. I am not claiming the agent is usually wrong outside your domain. It is often right, the same way the newspaper is often right on the page you cannot check. The claim is narrower and stranger: your confidence rises or falls with whether you catch an error, and catching one depends on your ability to verify, not the agent's actual accuracy. So the two come apart precisely where it matters most, and you get no signal when they do. I am also not saying you should distrust everything, which is just paralysis wearing the costume of rigor. And Brief does not make the agent correct everywhere, nothing does. It makes the input right in the domains you cannot personally vet, which is where the amnesia bites, so that trusting the output is a reasonable bet instead of a blind one. The Gell-Mann framing is borrowed from newspapers, and what transfers is the mechanism, uniform fluency meeting uneven verification, not the specifics of journalism.
So the next time you approve a clean, confident change in a part of the system you do not really know, do not ask whether it looks right, because it will. Ask the question the amnesia is built to make you skip: if this were wrong, in this exact corner, is there anyone who would have caught it?
Frequently asked questions
What is Gell-Mann amnesia? It is the tendency to distrust a source on the subjects you know well, where you can catch its errors, and then trust the same source on subjects you do not know, where you cannot. The writer Michael Crichton named it, half-jokingly, after the physicist Murray Gell-Mann. The key point is that trust tends to stay highest where you are least able to check, because those are the places you are least likely to catch an error; that is not the same as the source being more reliable.
How does Gell-Mann amnesia apply to AI coding agents? An agent writes with roughly uniform fluency and confidence across your whole codebase, but your ability to review it is high only in your areas of expertise and low everywhere else. So you catch its mistakes in the code you know and wave through its work in the code you do not, even though the agent was no more reliable there. Your trust ends up highest exactly where your ability to verify is lowest.
Why not just review AI-generated code more carefully? Because vigilance cannot manufacture understanding you do not have. In a part of the system you do not know, reviewing harder will not tell you which edge case a major customer depends on or why a limit was set where it was, since that context was never yours. The gap is made of missing information, not missing effort, so the fix is to supply the information, not to stare longer.
How do you make AI output trustworthy in areas you cannot personally check? By giving the agent the context those areas depend on, so its output is grounded in your team's real decisions rather than a plausible default. A product navigator like Brief holds those decisions, the real constraints, limits, and reasons behind unfamiliar parts of the system, and supplies them to the agent at the moment it would otherwise guess, so the work you cannot audit yourself is anchored to something real.
← Back to Blog