Everything we got wrong this quarter

We publish league tables scoring 1371 South West businesses out of 100. Businesses are ranked against their competitors on those numbers, and the numbers are meant to be checkable.

In the first full quarter of doing it, we changed the ruler six times. Four were corrections to figures already published. Two were changes to the method itself, forced by faults we found while looking at something else.

All six are dated in the changelog, which is the honest place for them and a place almost nobody reads. So here they are in one piece, with what each cost.

1. Three restaurants that were not in Gloucester

What was wrong. The Gloucester restaurant index contained three venues in Cheltenham. The seed sweep searches a radius around the town centre, and the radius reached into the next town: a hotel, the restaurant inside that same hotel listed separately, and a venue trading under a Cheltenham name.

What it cost. Three businesses were ranked in a league table for a town they do not trade in. The cohort dropped to 44 and its median moved from 47 to 46.

How it was caught. Reading the list. Not by any check in the pipeline — by looking at the names.

What stops it recurring. Hotel groups are now filtered like other national chains, and the review step checks whether a venue’s own name places it in a different town from the index it is being added to.

This is the correction that mattered most to the businesses involved and the least to the numbers. Being ranked in the wrong town’s table is a real harm, and it was our error rather than theirs.

2. A credentials check that only understood estate agents

What was wrong. The Content pillar checks whether a homepage links to professional credentials. That check matched Propertymark, NAEA, ARLA and RICS — property bodies — and was applied to every index. Construction, law, accountancy and dental cohorts were being tested for membership of organisations that have nothing to do with them.

What it cost. 41 businesses had the check change, and 43 of 274 ended up with a different score. Entire sectors had been scoring near zero on a signal many of them actually passed. Exeter solicitors moved from 69 to 72; Gloucester accountants from 48 to 49.

How it was caught. A sector-by-sector look at the credentials pass rate. Construction was 0 of 37. That is not a finding about builders, it is a broken detector.

What stops it recurring. The terms are now specific to each sector — ICAEW and ACCA for accountants, the SRA and Law Society for solicitors, FMB and Gas Safe for construction — with generic fallbacks so a firm that says “accredited” without naming a body still registers.

3. Review velocity, wrong for every business, all quarter

What was wrong. One line. Review timestamps arrive as 2024-03-15 10:23:45 +00:00 and were normalised with a replace that swaps only the first space. The result kept a space before the timezone offset, the date parser rejected it, every review was skipped as unreadable, and the count fell through to zero.

What it cost. That zero was not inert — it fed the Local presence pillar. Every published Local score carried a constant zero component for the whole quarter. A firm with 425 lifetime reviews was scored as having received none in ninety days. When it was corrected, 105 of 271 businesses had a different score, every one of them upward.

How it was caught. Trying to write a different article. The plan was a piece about review velocity; the data said zero for all 266 businesses with a Google profile, and zero reviews in ninety days across 266 local businesses including restaurants is not a finding, it is a bug.

What stops it recurring. The parser accepts every format the source returns, and — more importantly — when no timestamp in a response can be read, the function now returns unmeasured rather than zero. Absent data is excluded from scoring everywhere else in this pipeline. This was the one place it was silently counted against the business.

The lesson worth keeping is that the failure was invisible because it was uniform. Every business scored zero, so nothing looked anomalous. It did not look like a bug. It looked like a sector where nobody reviews anything.

4. Claiming four AI engines while querying one

What was wrong. The methodology page said AI search presence was measured across “AI Overviews, ChatGPT search, Perplexity and Gemini”. It never was. The pipeline has only ever queried Perplexity, on five prompts per index.

What it cost. No score changed — the code did what it did, and every published figure came from it. What was wrong was the claim about those figures, which was four times broader than the evidence behind it.

How it was caught. Checking the collector before writing an article about AI presence, rather than trusting the documentation.

What stops it recurring. The page was narrowed to the truth rather than the pipeline widened to the claim. Widening would have re-measured the pillar and made this quarter incomparable with the next. The narrower true statement stands until the broader one is actually implemented.

5. A pillar that was three boxes ticked

What was wrong. Content & trust read three boolean checks, so it could only ever return 0, 33, 67 or 100 — four values, for 10% of every published score. Two further signals the methodology advertised, indexed page count and content freshness, were null on all 271 records. The pillar had never measured them.

What it cost. The most crudely measured thing on the site was also the strongest predictor of whether a business can be found at all. Whole cohorts scored identically and a reader learned nothing from it.

What changed. Six signals from the same free homepage fetch, and 48 distinct values instead of 4. Indexed page count was removed rather than implemented: a real one needs Search Console access to each business’s own property, which we neither have nor should, and counting published pages is a different quantity that would have repeated the error under a more respectable name.

What we will not claim. The finer measurement barely improved the pillar’s predictive power — 0.519 to 0.526. The three crude booleans were already capturing most of the available signal. What is fixed is that the pillar now matches its published description.

6. Measuring the trade instead of the business

What was wrong. Review velocity — the same component as fault 3 — gave full marks at ten new reviews in ninety days, the same bar for everyone. Ten a quarter is routine for a restaurant and exceptional for a solicitor. The component was substantially measuring which trade a business was in.

What made it visible. Fixing fault 3 made it worse. The correction lifted restaurants and estate agents and left construction and accountancy untouched, widening the gap between restaurant and construction Local scores from 15 points to 31. A correct measurement of the wrong thing.

What changed. Velocity is now scored against a business’s own cohort. Where three quarters of a cohort receive no reviews at all — 83% of Cheltenham builders, 88% of Gloucester accountants — the component is excluded rather than normalised, because scaling to the cohort maximum would invent a hundred-point spread out of a nought-to-three range, and ranking by percentile would reward a business for having no reviews.

What it cost. 160 businesses moved up, 47 moved down and 64 were unchanged. This was the first change we published that moved any score down: a restaurant scoring well partly because its trade collects reviews now scores slightly lower.

What the pattern says

Reading the six together, they fall into two kinds, and only one of them is a bug.

Two were code faults — the timestamp parse and the wrong-town radius. Those are ordinary. Software has bugs.

Four were measurement faults: measuring something real that was not the thing we claimed. A credentials check that understood one sector. A pillar advertising signals it never read. A velocity component measuring trade culture. A methodology page describing four engines. None of those would have been caught by a test suite, because the code did exactly what it was written to do.

That is the uncomfortable part. The dangerous errors in a measurement project are not crashes. They are numbers that arrive on time, look plausible, and mean something other than what the label says. Every one of these was found by a person looking at output and asking whether it could be true — usually while trying to do something else.

Why publish it

Because the alternative is worse for us, not better.

This site’s entire claim is that you do not have to trust us: the data is open, the pipeline is public, the scoring is plain arithmetic. That claim is only worth anything if the corrections are public too. An index that has never announced an error has either never looked or is not telling you.

There is a practical version of the same argument. Superseded datasets are kept beside every correction, so a figure cited before a change remains checkable. If we quietly fixed numbers, every citation of this index would silently rot.

And it is the standard we would apply to anyone else. If an agency showed you a dashboard that had never once been wrong, the right question is not how good they are. It is how hard they are looking.

What to expect next

More of them. Q4 lands in September, and the weights audit already published says our own weighting is probably wrong — Speed is 20% of the score and correlates −0.13 with whether a business can be found. If that holds across two independent quarters, the weights change, announced in advance, with old and new scores side by side for a quarter.

That will be the seventh.