--- title: The Comparison Group Went Dark — and the Headline Number Doubled description: Our comparison sites stopped reporting mid-study and the treatment 'lift' inflated to +41% before anyone noticed. How tracking blackouts and ID migrations corrupt pre/post analyses, and the five integrity checks that catch them. url: https://preview.artificialpoets.com/blog/comparison-group-blackout/ site: Artificial Poets type: post date: 2026-08-10T03:32:09+00:00 modified: 2026-08-14T18:30:48+00:00 categories: - Measurement --- # The Comparison Group Went Dark — and the Headline Number Doubled ## What actually happened Three separate data-integrity failures stacked on top of each other, none of them visible from a dashboard. **The analytics IDs changed underneath the sites.** During a platform migration, several properties were re-registered in the analytics tool under new site identifiers. Our pipeline — like most pipelines — keyed every query on the site ID. To a query keyed that way, a migrated site looks like one site dying and an unrelated site being born. History splits across two identities, and both halves look wrong: the old ID shows collapse, the new one shows a site with no baseline. **Then the comparison group's tracking went out entirely.** Every comparison property stopped reporting in the same week — a coordinated cutover as sites moved infrastructure — and came back nine weeks later. The treatment sites, on different infrastructure, never broke. **And nothing alerted.** Aggregate dashboards summed whatever data existed; a group of sites reporting zero simply made the other group's share look magnificent. No per-site last-seen check existed, so the blackout surfaced only when someone asked a question the aggregate could not answer. ## What it did to the numbers The mechanics are brutal and completely silent. A treatment-versus-comparison lift divides treatment change by comparison change. When the comparison group's measured traffic slides toward zero inside the window — not because readers left, but because tracking did — the denominator collapses and the "lift" inflates without any reader anywhere doing anything different. Any measurement window ending inside the blackout produced a treatment advantage roughly double the real one. The **+41%** headline from that era was retracted; the rebuilt, blackout-excluded panel supported **+19.1%** on the same title and metric — a strong result, but the difference between the two is the difference between a number that survives a buyer's analyst and one that detonates in the meeting. The subtler damage came from the ID migrations. Sites that had "died" were quietly missing from the comparison set, which shrank the panel and let single-site noise dominate. And one comparison property's volume multiplied several-fold in a month on what turned out to be non-human traffic — inflating the comparison group in the *other* direction until it was screened out. ## Why nobody notices this Because every individual view of the data looks plausible. The dashboard renders. The trend lines move. The totals have the usual shape. Data-integrity failures do not look like errors; they look like results — and the failure that flatters you is the one nobody investigates. Ours flattered us, which is precisely why it survived as long as it did. The uncomfortable rule we took from it: **any surprising lift is a data-quality question before it is a success story.** The order of operations matters, because once a number has been in a deck, the organisation is negotiating with it. ## The five integrity checks These run before any pre/post comparison now. Each is one query. 1. **Last-seen date, per site.** Not aggregate volume — the most recent event date for every property in the panel, individually. A comparison site whose data ends inside your window disqualifies the window, not just the site. 2. **Identity continuity.** Key on a stable identifier — for web properties, the domain — rather than the analytics tool's internal site ID. Tool IDs change on migrations, re-registrations and re-tags; domains mostly don't. This single change is what made our rebuild possible. 3. **Balanced-panel requirement.** For every week in the analysis, require every panel site to be present, and drop weeks that fail rather than letting partial weeks average in. A lift computed on an unbalanced panel is comparing different groups at different times and calling it a trend. 4. **Volume-anomaly screen.** Flag any site whose week-over-week volume moves by multiples. The cause is occasionally news; it is usually bots, retags or migrations, and all three poison a comparison group. 5. **Cap the window at the data, not the calendar.** The analysis ends at the last date the *whole panel* is clean — even when the calendar says you have three more months. Three more months of corrupted comparison data is not three more months of evidence. - Every comparison site went dark the same week; the gap lasted **nine weeks** - Windows ending inside the gap roughly **doubled** the measured lift - Retracted headline: **+41%**. Rebuilt, blackout-excluded result: **+19.1%** - The fix that made the rebuild possible: keying on **domain, not tool ID** - Integrity checks now required before any comparison: **five, one query each** ## FAQ Per-site last-seen dates: weekly, automated. The full five checks: before any number leaves the analytics team, and again before it leaves the building. The cost is a morning; the retraction you avoid is priced in credibility. Then windows and seasonality own your result, and you should say so. A same-site design — measure, enable, and if the chance arises, measure an off-period — is weaker than a panel but far stronger than a bare before/after. What we learned from an unplanned four-week off-period ended up being the strongest evidence we hold. The mechanism is tool-agnostic. Property re-registrations, retags, consent-mode changes and CMS migrations all fracture identity or interrupt collection in any stack. The tool renders whatever arrives; continuity is your job. ## Ten diagnostics, one query each The Analytics Health Check packages these integrity checks with five more — pageleave coverage, bot-filter parity, window sensitivity — as a runnable worksheet for your analytics stack. ## See what your session depth looks like We read your analytics with you for twenty minutes and tell you what share of your sessions stop at the first page. You leave with the annotated read, whether or not we ever talk again. 20 minutes, your analytics, no deck. [Book my session-depth read](/request-a-demo/)