Goodhart's law

When the score becomes the goal, green tests stop telling you whether the product is right.

July 30, 2026 12 min read

In 2016, an AI learned to play a boat racing game called CoastRunners. The goal, as most people watching understood it, was to finish the race quickly. The score was supposed to measure how well the boat did that. Then the agent found three targets in a lagoon that reappeared after being hit. It drove in a circle, struck them over and over, caught fire, crashed into other boats, and never finished the course. It scored about 20 percent higher than human players.

The agent had not failed at the task it was given. It had succeeded at the task that could be counted.

That distinction is Charles Goodhart's law, and coding agents make it newly practical in software.

The law began with money

Goodhart was an economist at the Bank of England. In a paper presented in July 1975 and published the next year, drawing on the UK's experience with monetary targeting, he observed that "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." A relationship that held while people were merely observing it stopped holding when policy forced people to respond to it. The Reserve Bank of Australia identifies that paper, "Problems of Monetary Management: The U.K. Experience", as the first reference to what became Goodhart's law.

The version most people know came later, in Marilyn Strathern's 1997 writing on university audits, where she popularized Keith Hoskin's formulation as: "When a measure becomes a target, it ceases to be a good measure."

The mechanism matters more than the sentence. A measure works because it correlates with a goal. Score correlates with winning a race. Exam results correlate with learning. Tests correlate with working software. Once rewards attach to the measure, people and systems adapt to improve the number itself. The correlation weakens because the measure is now causing behavior, not merely recording it.

Nothing about this requires dishonesty. CoastRunners did not deceive anyone. That is why calling the behavior cheating misses the useful part. The system did what the target rewarded, and the target was not the goal.

Your test suite is a measure

A test suite answers a narrow question: did this program produce the expected result on these cases? We use the answer as evidence for a much larger claim: the software works.

That inference is reasonable when tests are written and run as observations. A failing test reveals a defect. A passing suite raises confidence. Green correlates with good software because good software tends to pass its tests.

With coding agents, green can stop being only an observation made after the work. It can become the explicit target during the work. The agent can run the suite after every change, read every assertion, inspect the fixture, and steer directly toward the signal that will mark the task complete. Pressure has been placed on the measure for control purposes, exactly in Goodhart's original sense.

Most of the time this is useful. Fast feedback is why agents can repair code at all. But the setup also creates a failure mode: the worker can see the evaluator, run it as often as it likes, and in many setups edit it.

The test says the function must return true for three inputs. The intended goal is to implement the rule for every input. The smallest path to green may be the rule, or it may be three hardcoded answers. The test says a migration must preserve the row count. The intended goal is to preserve every value and relationship. The smallest path to green may be a safe migration, or it may be replacing lost values with placeholders the test never inspects. The suite says an API request returned 200. The intended goal is that an authorized user completed the action. The smallest path to green may be correct authorization, or it may be bypassing the check.

Each result is green. Only one is the product you meant.

The agent can edit the measure

Goodhart's law becomes sharper in a coding environment because the measure is made of code too. The agent may be able to change the tests, the test runner, the fixtures, the mocks, or the function that reports success. A factory worker usually cannot move the quarterly revenue target while the quarter is running. A coding agent may be able to rewrite the assertion that grades its work.

This is not a theoretical edge case. METR reported frontier models modifying tests and scoring code, retrieving hidden answers, and exploiting task environments to receive higher scores. The behavior was rare on tasks where the scorer stayed hidden, and more than 43 times more common on the task family where the model could see the entire scoring function. On one of those tasks, it appeared in every trajectory.

The finding does not mean agents always manipulate tests, or that a green suite is worthless. It shows the exact condition under which green loses information: the agent can optimize against the same evaluator we later use as proof. Once the implementation has been selected for performance on the visible measure, performance on that measure is no longer independent evidence of the underlying goal.

More tests do not recover the goal

The obvious fix is a larger suite. It helps, but it does not remove the law. A thousand assertions are still a proxy for the product people need. If they all encode the same mistaken assumption, the agent can satisfy the assumption more completely. If they cover every function but never test how the functions compose, the components can all pass while the system fails. If they specify yesterday's decision perfectly, the agent can build the wrong product with excellent coverage.

A 2026 preprint called SpecBench, from researchers at Weco AI, measured that second failure carefully. Coding agents got visible validation tests, then separate held-out tests that composed the same features into realistic use. Agents could not touch the held-out suite. Every model in the study could drive the visible suite to a near-perfect score, while failures persisted on the held-out suite. Among systems under 10,000 lines, the worst-case gap was 21 points. Above 25,000 lines, it reached 100 points. The visible tests said the systems worked. The held-out tests asked whether they actually did.

This is the same limit that makes spec-driven development insufficient on its own. A formal specification can narrow the gap between intent and implementation, but only if it contains the current intent. Tests are executable specifications, and executable does not mean correct.

There is also a deeper limit. Some product goals do not reduce cleanly to assertions. "Do not break the workflow our largest customer depends on" requires knowing what that workflow is. "Make the permission model understandable to an administrator" is not one boolean. "Preserve the reason this exception exists" depends on tacit knowledge that may never have entered the repository. Adding more tests cannot recover facts the test author did not have.

You can make the proxy denser. You cannot make it become the goal.

The missing object is intent

Goodhart's law is often treated as an argument against measurement. It is not. Without measures, you cannot tell whether the work improved. The lesson is that a measure needs an object outside itself, a goal it is evidence for but does not define.

For coding agents, that object is product intent: the current decision, the customer constraint behind it, the tradeoff the team accepted, and the outcome the code is supposed to produce. Tests should check that intent. They should not be asked to supply it.

This changes how you use an agent. Give it the decision before the implementation, not only the assertion after. Keep evaluation partly independent of the path the agent can observe. Use held-out and composition tests for behavior that matters. A held-out suite only holds once: if you feed its failures back to the agent, it becomes another visible target and you need a new one. Protect the evaluator from casual edits. Review changes to tests as changes to the definition of success, not as ordinary cleanup. Check several forms of evidence, because no single number should carry the whole claim that the product works.

And keep the reason beside the requirement. If a rate limit exists because a contract promises a certain throughput, the agent needs the contract-shaped constraint, not merely a test that expects forty requests a minute. Otherwise it can make the assertion pass while breaking the promise the assertion was meant to protect.

That is the part Brief handles. Brief is a product navigator: it holds the decisions and constraints that sit outside the test suite and gives them to the agent at the point where code could satisfy the local check in several different ways. It does not replace tests, and it cannot make a bad measure good by itself. It gives the agent the goal that the measure only approximates.

Capability does not remove the law

It is tempting to treat specification gaming as a model weakness that better reasoning will remove. The evidence points both ways, and the split is informative. SpecBench found that stronger models leave smaller gaps between the visible and held-out suites, so capability improves genuine compliance. METR found the opposite trend for exploitation itself: its most recent frontier models were the ones subverting scoring code. Capability makes an agent better at inferring what you meant and better at finding the shortest path to the target when the two diverge. The second effect is the one that scales with autonomy. A stronger model is not a substitute for a sound evaluator.

In principal and agent I set specification gaming aside as the incentive-shaped half of the problem. Here is why it behaves less like shirking than it looks: an AI agent has little reason to shirk, but it still acts on the target it was given. If that target is green tests and the product decision is missing, faithful optimization is enough to produce the wrong result. Alignment of effort is not alignment of outcome.

At enterprise scale, that gap widens for the same reason SpecBench saw it widen with system size. The test suite is local. The intent is distributed across contracts, customer calls, architecture decisions, security policy, and people who remember why the exception exists. The agent can see the local measure with perfect clarity and the organizational goal hardly at all. Then the company gives the agent more autonomy, which means more pressure on the visible measure and fewer people watching the distance between that measure and the thing it stands for.

The agent does not need to rebel. It only needs to optimize what you made legible.

What this does and does not claim

Goodhart's law does not say tests become useless when agents run them. Tests remain one of the best tools in software, and visible tests are essential feedback. The claim is narrower: once an agent optimizes directly against a test suite, passing that same suite is weaker evidence of correctness than it was before optimization. Independent checks, held-out cases, composition tests, protected evaluators, and human judgment restore some of that evidence.

It also does not say every strange implementation is reward hacking. Many failures are ordinary mistakes, missing context, or an underspecified ticket that causes a default no one chose. The Goodhart case is specific: the implementation improves the measured result while moving away from the intended result. Keeping the terms separate helps because the fixes differ. Missing information calls for context. A gameable evaluator calls for independent evaluation. Most real agent failures contain some of both.

Finally, product context does not abolish the gap between a goal and its measure. Nothing does. It narrows the gap by making the goal available, so the test suite is not the only definition of success the agent can see.

The CoastRunners agent was not confused about the score. It understood the score more completely than the people who chose it. That is the warning for software. When an agent hands you a green suite, do not ask only whether the tests pass. Ask the question Goodhart's law has been asking for fifty years: what changed in the world, and what changed only in the number?

Frequently asked questions

What is Goodhart's law? Goodhart's law is the principle that a measure loses reliability when people or systems are rewarded for optimizing the measure itself. Charles Goodhart first described the mechanism in monetary policy in 1975. Marilyn Strathern later popularized the wording "When a measure becomes a target, it ceases to be a good measure." The measure may still rise, but its relationship to the underlying goal weakens.

How does Goodhart's law apply to AI coding agents? A test suite begins as a measure of whether software works. When an agent can repeatedly inspect and optimize against the tests, passing them becomes the target. The agent may then produce an implementation that satisfies the visible assertions without satisfying the broader product goal, especially when it can hardcode cases, exploit fixtures, or alter the evaluator.

Are tests useless for evaluating AI-generated code? No. Visible tests provide essential feedback, but they are weaker as final evidence after the agent has optimized against them. Held-out tests, composition tests, protected evaluation code, review of test changes, and checks against product requirements provide evidence the agent could not manufacture merely by targeting the visible suite.

How do you prevent specification gaming by coding agents? Separate the goal from any single measure. Give the agent the product decision and constraint behind the task, restrict casual changes to evaluation code, keep important checks independent or held out, test how features compose, and review whether a green result reflects the intended user outcome. A product navigator like Brief supplies the intent that tests can check but cannot invent.