Benford's Law and SERP spam: what it catches, and what it only appears to
Benford’s Law says that in many naturally occurring collections of numbers, the leading digit is not evenly distributed. 1 turns up about 30% of the time and 9 under 5%. It is used in forensic accounting, election forensics and scientific-fraud detection, on the reasoning that people inventing numbers distribute first digits far more evenly than reality does.
It is periodically proposed as a way to detect SEO spam. The pitch is appealing: run a distribution test over backlink counts or rankings, find the profiles that fail, and you have caught something a search engine missed.
We have the data to test that pitch rather than argue about it. Everything below is computed from the published datasets across 34 indices when this page is built.
The expected distribution
Section titled “The expected distribution”| Leading digit | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|---|
| Benford expects | 30.1% | 17.6% | 12.5% | 9.7% | 7.9% | 6.7% | 5.8% | 5.1% | 4.6% |
For this to apply at all, a dataset has to meet conditions that are easy to state and easy to forget: values spanning several orders of magnitude, produced by a multiplicative or growth-like process, not bounded, not assigned, and numerous.
Most search metrics fail at least one of those.
What happened when we ran it
Section titled “What happened when we ran it”| Dataset | n | MAD | Standard verdict | Noise alone produces | Actually beyond noise? |
|---|---|---|---|---|---|
| Backlinks total inbound links per domain | 1153 | 0.0150 | marginal | 0.0104 | yes |
| Referring domains distinct sites linking in | 1153 | 0.0381 | nonconformity | 0.0104 | yes |
| Google reviews lifetime reviews per business | 839 | 0.0139 | marginal | 0.0120 | yes |
| Sitemap URLs pages a site lists in its own sitemap | 822 | 0.0101 | acceptable | 0.0122 | no |
| Homepage word count visible words on the front page | 1215 | 0.0146 | marginal | 0.0099 | yes |
The fourth column is the standard test. Nigrini’s mean absolute deviation thresholds — the ones every Benford tutorial quotes — call anything above 0.015 “nonconformity”, the polite phrase for look here for fraud.
The fifth column is the part those tutorials omit. It is the deviation you get from perfect Benford data at that sample size, from sampling noise alone: simulate a dataset that conforms exactly, draw n values, measure the MAD, repeat, take the 95th percentile.
For several of these datasets the noise floor is higher than the threshold. The standard test is declaring fraud-shaped deviation on data that is behaving impeccably. Nothing here is evidence of anything except that the thresholds were calibrated for accounting populations of many thousands of records, and a local business index does not have those.
That is the single most important thing to understand before using this technique on search data: at SEO sample sizes, the standard Benford test has a false-positive problem large enough to swamp its findings.
The counter-example
Section titled “The counter-example”Here is the same test on a measure that has no business being tested at all:
| Dataset | n | Observed leading digits |
|---|---|---|
| Lab LCP in milliseconds a bounded, clustered measure — the counter-example | 1055 | 31.8% · 10.3% · 10.7% · 10% · 10.8% · 9.8% · 7.8% · 4% · 4.7% |
Largest Contentful Paint in milliseconds is overwhelmingly a four-digit number between 1,000 and 9,999. Its leading-digit distribution is therefore shaped by the unit it happens to be reported in, not by any property of the websites. Report the same measurement in seconds and the distribution changes completely. A test that gives a different answer depending on whether you use milliseconds or seconds is not measuring the thing you care about.
Where it genuinely applies in search
Section titled “Where it genuinely applies in search”Three families of data plausibly satisfy the conditions:
- Backlink counts and referring domains across many domains. These span five or six orders of magnitude and grow by something close to a multiplicative process — sites that have links attract more. This is the best candidate in the field.
- Indexed page counts across a large corpus of sites.
- Traffic or impression estimates, where you have them at scale.
Note what these have in common: they are counts across many independent entities, not measurements of one site, and they need to be in the thousands before the test means much.
Where it cannot work, however appealing
Section titled “Where it cannot work, however appealing”Ranking positions. This is the application people reach for first and it is the clearest failure. Positions are bounded 1–100, they are assigned rather than accumulated, and they are ranks rather than magnitudes. Benford has nothing to bite on. Worse, rank data often looks roughly Benford-shaped, because the number of businesses ranking decays as you go deeper — a decaying distribution superficially resembles a logarithmic one. You get a plausible-looking result that means nothing, which is more dangerous than an obvious failure.
Anything bounded or capped. Ratings out of five. Scores out of 100. Percentages. Core Web Vitals thresholds. All structurally excluded.
Anything rounded or clustered. Metrics reported in round hundreds, or clustered in one band, carry the shape of their reporting convention.
AI answers: there is nothing to test
Section titled “AI answers: there is nothing to test”Applying leading-digit analysis to LLM search results is not a hard problem. It is a category error.
A Google results page is a ranked list of a hundred items with a rich set of numeric metrics behind it. An AI assistant’s answer names three or four businesses in a sentence. There is no numeric population, no magnitudes, and n is single digits per query. Benford needs thousands of values from a growth process; an answer engine gives you a handful of names.
If you want to detect manipulation in AI answers, the signals are entirely different in kind: which sources get cited and whether they exist, whether the same handful of domains appears across unrelated prompts, whether cited pages actually contain what the answer claims. Those are verification problems, not distributional ones.
The same applies to Bing, incidentally. The instinct to treat Google and Bing as different cases here is wrong — both return bounded ranked lists, and the structure of the data, not the identity of the engine, is what decides whether the test applies.
”Finding tricks search engines cannot”
Section titled “”Finding tricks search engines cannot””This premise deserves challenging directly, because it is the usual reason the technique gets proposed.
A search engine sees the crawl graph, the click stream, the historical record of every domain and the internal signals none of us can observe. The idea that a third party running a distribution test over public metrics will find manipulation Google has missed is, in almost every case, backwards. They have more data and more incentive.
There is a real use, but it points somewhere else. Benford is a tool for auditing reported numbers — the ones passed to you by someone with an interest in how they look:
- Agency reports. Traffic, impression or lead figures typed into a slide deck rather than exported from a system. This is exactly the fabricated-numbers case the technique was built for.
- Third-party tool data, where you want to know whether a vendor’s estimates are modelled or measured.
- Anything self-reported in a pitch.
That is a genuinely useful application, and it is about the people showing you numbers rather than about the SERP.
If you are going to use it anyway
Section titled “If you are going to use it anyway”Four rules, all of which our own results argue for:
- Simulate your null. Before treating a MAD as a finding, generate conforming data at your sample size and see what deviation noise alone produces. If your threshold is below your noise floor, you have no test.
- Check the conditions before running anything. Orders of magnitude, growth process, unbounded, unassigned, numerous. If the data fails one, the result is uninterpretable regardless of how it comes out.
- Treat a deviation as a question. It is a reason to look, never a finding in itself. Every dataset above “fails” the standard threshold and none of them is evidence of manipulation.
- Assume your adversary can read. Benford conformity is trivial to fake once you know it is being tested. Any detection method that is public and cheap to satisfy stops working on exactly the people you most want to catch.
The honest summary
Section titled “The honest summary”Benford’s Law is a real forensic instrument that works on large populations of unbounded counts generated by growth. A slice of that description fits backlink data at scale. Almost nothing else in search does, ranking positions least of all — and AI answers offer nothing to measure.
Our own data makes the practical point better than the theory does: run the standard test on real SEO datasets at real sample sizes and it flags everything, because the thresholds assume a sample size you do not have. A test that always fires is not a detector.