Drift

A crawler can tell you a page is different from last time it looked. Telling a redesign apart from a real strategy shift is a different problem, and it's the one that actually matters.

August 24, 2026 6 min read

A competitor's pricing page said "no setup fees" for two years. Three weeks ago, a $500 fee quietly appeared under implementation costs for new accounts. The headline claim, the layout, the testimonials, all of it stayed exactly the same. One clause, buried two sections down, changed. Nobody on the other side caught it, because nobody was still reading a page they had already filed as "the answer."

That gap has a name close enough to borrow: drift, the space between what is true now and what you last checked. The easy half of that problem is already solved. A crawler that fetches a page today and compares it to yesterday's fetch will find the fee clause without trying. The hard half is the one crawling doesn't touch: knowing which of the differences it finds are the one that matters, and which are a font change, a testimonial rotation, a redesign that moved the same sentence to a different part of the page.

Not every difference is a change

Machine learning has a formal name for a narrower version of this problem. Gerhard Widmer and Miroslav Kubat, in "Learning in the Presence of Concept Drift and Hidden Contexts" (Machine Learning, 1996), describe concept drift as what happens when the underlying relationship a learning system is trying to track shifts under a hidden context, so a rule that was accurate stops being accurate, and the system has to notice that the ground it was standing on moved. Their specific problem was online learning algorithms, not competitor pages, and I'm borrowing the shape of it more than the formal machinery. But the core structure carries over exactly: a system watching for change needs a way to tell a shift in the thing it actually cares about apart from noise in the surface it's reading. A crawler diffing two page fetches has no such distinction built in. Every character that changed counts the same as every other character that changed, whether it moved a paragraph three pixels to the left or quietly added a fee.

Why the diff itself isn't enough

Here's where it gets harder than a smarter diff would fix. Suppose the competitor rewrites the fee clause instead of adding it fresh, "no setup fees" becomes "no setup fees on annual plans." A raw diff catches the two added words without any trouble. A system that scores changes by how much text moved treats a two-word addition as minor and suppresses it as noise, exactly backward, since those two words reverse what a monthly customer should expect to pay. Or suppose they move from "no setup fees" to "implementation is included," different words entirely, same commitment, and now the size-based signal points the other way: a big textual change flagged as significant when nothing about the actual claim moved at all. Recognizing that two descriptions, phrased differently, are about the same underlying claim is the same problem a decision layer has to solve to know a new record and an old one are about the same thing. Drift detection needs that solved first. You cannot say a claim changed until you know which old claim it's being compared against, and phrasing is not a reliable enough signal to find it.

What this actually takes

Put the two pieces together and the real job comes into view. First, match the new page's claims to what you already had on record for that competitor, not by exact text but by what the claim is actually about, so a reworded sentence is still recognized as the same commitment or a different one. Second, once the match is made, ask whether the substance moved: not "did the words change" but "does this commit the competitor to something different than the version on file." Most differences a crawler finds fail that second test. Layout, tone, and phrasing move constantly without anything underneath them changing. The ones that pass it are rare, and they're the only ones worth a person's attention.

That's the job Brief's Competitor Agent is built around: it enriches the competitor set, crawls their pages, and flags when one of them drifts, not every time a page differs from its last fetch, but when what a competitor is actually claiming has moved.

What still needs a person

Not every borderline case resolves cleanly, and it's worth saying so plainly. Whether a reworded claim represents a real shift or the same commitment in new language is sometimes a judgment call even for someone who has followed a competitor for years. A system that quietly picks a side on those calls is worse than one that flags them and asks. The honest version of drift detection routes the ambiguous cases to a person instead of deciding on their behalf, the same way an agent should route a decision it can't resolve rather than default through it.

Most of what a crawler finds isn't drift at all, it's the normal churn of a page nobody is redesigning around your notice. Does what your team catches actually separate the two, or does everything that changed just look the same?

Frequently asked questions

What is "drift" in competitive monitoring? The gap between what a competitor's page or positioning currently says and what a team last checked, specifically the subset of changes that reflect an actual shift in what they're committing to, not surface changes like a redesign or reworded copy that leaves the underlying claim the same.

How is drift different from a crawler detecting that a page changed? A crawler compares two fetches and reports every difference with equal weight, a layout change and a pricing change look the same to it. Drift is the narrower, harder judgment underneath that: which of those differences reflect a real change in what the competitor is claiming, and which are noise in how the page happens to be written today.

Why does detecting drift require entity resolution first? Because you can't tell whether a claim changed until you know which earlier claim to compare it against, and competitors rarely repeat their exact wording. A reworded sentence has to be recognized as the same underlying commitment, or a different one, before "did it change" is even a question you can answer.

What happens when it's genuinely unclear whether something is real drift or just noise? It should go to a person rather than get silently decided. Some reworded claims are ambiguous even to someone who has tracked a competitor for years, and a system that guesses and moves on is worse than one that flags the ambiguous case and asks.