Survivorship bias
Why the evidence that decides your product is the evidence your dashboards structurally cannot show you.
In the Second World War, American bombers came home from raids over Europe covered in bullet holes, and the holes were not spread evenly. They clustered on the wings and the fuselage; the engines were comparatively clean. The obvious move is to add armor where the damage is: reinforce the wings.
The statistician Abraham Wald, working with Columbia's Statistical Research Group, showed why that is exactly backwards. Armor the engines, the places with almost no holes. His reasoning is one of those things that rearranges how you see once you hear it. The planes being measured were the ones that came back. A bomber could take a barrage through the wings and still limp home, which is exactly why the wings of the returning planes were full of holes. A bomber hit in the engine did not come home, so it was never in the data. The clean spots on the survivors were not the safe places. They were the fatal ones, marked by their absence.
That is survivorship bias, and Wald's version is the sharpest ever stated: you are studying only the things that made it through, so the most important evidence is the evidence that is missing, and it is missing for the very reason that makes it matter.
Your analytics are a plane that came back
Now look at where your product decisions come from. They come from your data: the funnels, the retention curves, the feature usage, the session recordings, the survey responses. Every one of those is a measurement of the users who are still here. The dashboard is a returning bomber.
The users the missing feature would have served are not underrepresented in that data. They are absent from it. The person who hit your signup, saw that you did not do the one thing they needed, and left, is in no cohort. The enterprise buyer whose security requirement you did not meet never became an account. The segment you quietly decided not to support generates no tickets, because they are not your users. They are the planes that went down over the target, and your analytics, by construction, cannot see them. The bullet holes that would tell you the most are in the parts of the market that never came back.
Optimizing harder armors the wings
Here is why this is getting more dangerous, not less. The whole promise of a data-driven, and now AI-driven, product practice is to fit your decisions ever more tightly to what the numbers say. Feed a year of product analytics to an agent and ask it what to build, and it will do a genuinely excellent job of finding what your current users want more of. It will optimize the funnel that exists, deepen the features that survivors already use, and smooth the path that retained people already walk.
Which is to say it will armor the wings, perfectly. A better model fit to a survivor sample is a better-reinforced wing. The more faithfully you optimize the data you have, the more completely you entrench the shape of who already stayed, and the more invisible the people you are losing become, because nothing about optimizing survivors ever surfaces the ones who left. You cannot A/B test your way to a segment that is not in the test. You can only get there by knowing it is missing.
The missing bullet holes live outside the data
So where is the equivalent of Wald's insight? It is in the knowledge that never makes it into the analytics in the first place: why the last five enterprise deals actually died, the requirement you keep hearing in sales calls and keep deferring, the segment you made a deliberate decision not to serve and the reason you made it, the churn interview where someone told you the real thing. None of that is a row in a table. It lives in people's heads, in a lost-deal note, in a decision someone made once and did not write down. It is precisely the product context that your dashboards are structurally incapable of holding, which is exactly why it is the context that matters.
This is the work Brief exists to do, and I will say it directly: Brief is a product navigator, the layer that holds the decisions and the losses your analytics cannot, why you are not serving a segment, what killed a deal, which requirement you keep hearing and have not built, and puts that knowledge in front of the people and the agents making product calls, next to the data rather than lost behind it. For an enterprise the missing population is enormous, an entire funnel's worth of people a large company has already lost across dozens of teams and years, almost none of it written anywhere a decision can reach. Feasibility they can buy; the survivors they can measure. The planes that went down are the ones no one is looking at, and they decide the war.
None of this means the data is useless. Survivors carry real signal, and you should absolutely optimize for the people you have. The one thing the data can never do is tell you what is not in it. That part you have to know on purpose, and put somewhere the decision can find it.
So before you reinforce the wings again, ask the question Wald asked. Where are the bullet holes you are not seeing, and what did they cost you before you ever knew they were there?
Frequently asked questions
What is survivorship bias? It is the error of drawing conclusions from only the people or things that made it through some selection, while ignoring those that did not, precisely because they left no trace. The classic example is Abraham Wald's WWII analysis of returning bombers: the planes that came back showed damage on the wings, but the vulnerable spot was the engines, because planes hit there never returned to be counted. The missing data was the most important data.
How does survivorship bias affect product analytics? Every standard product metric, retention, feature usage, funnels, surveys, measures the users who are still present. The people a missing feature would have served have already churned or never converted, so they appear in no cohort and no dashboard. Decisions made purely from the data therefore optimize for the users who stayed and stay blind to the ones who were lost, which are often the larger and more valuable population.
Why does AI or data-driven product work make survivorship bias worse? Because those methods fit decisions more tightly to the data you have, and the data you have is made of survivors. An agent asked what to build from a year of analytics will excellently deepen what current users like and cannot surface the segments that are absent from the sample. Optimizing a biased dataset more precisely entrenches the bias rather than correcting it.
How do you correct for survivorship bias in product decisions? You cannot recover the missing users from the data itself; you have to supply the knowledge deliberately. Capture why deals were lost, why users churned, and which segments you chose not to serve and why, and make that context available next to the analytics, so the people and agents making product calls can weigh what is missing and not only what remained.
← Back to Blog