--- title: Seven Questions to Ask When a Vendor Shows You an Engagement Lift description: A chart that goes up answers nothing. Seven questions — window, comparison, off-switch, definition, segments, non-results, falsifiability — that take fifteen minutes and separate measured engagement lifts from manufactured ones. url: https://preview.artificialpoets.com/blog/engagement-lift-questions/ site: Artificial Poets type: post date: 2026-08-10T03:32:09+00:00 modified: 2026-08-14T18:30:48+00:00 categories: - Vendor Evaluation --- # Seven Questions to Ask When a Vendor Shows You an Engagement Lift ## 1 · "What window is that from — and what is it running at now?" Launch quarters are always the biggest: novelty, tuning and attention all peak early. A single percentage with no dates attached is a decision someone made for you. The honest answer has two numbers — peak and steady state — and the steady state is the one your business case should use. (Our own largest published lift: +19.1% in the first quarter, +4.0% at run rate. Both printed, because the difference is the point.) The pause to watch for: a vendor checking whether the run-rate number exists. ## 2 · "Compared to what?" A before/after on one site cannot separate the product from seasonality, news cycles or platform shifts. The minimum is comparable properties measured over identical weeks; the strong version is the *entire eligible set* of comparables, not a curated one. "Industry benchmarks" is not a comparison group — it is other people's unrelated sites at other times. The pause to watch for: "we typically see…" — typical of what denominator? ## 3 · "What happens when it's switched off?" The strongest evidence any vendor can hold: same site, same audience, same season, product removed — metric returns toward baseline, then recovers on resume. Almost nobody has it, because nobody switches off something that works on purpose. Ask anyway. A vendor who has an off-period and shows it is holding the rarest card in the category. A vendor who has never considered the question is telling you how their evidence was assembled. ## 4 · "Define the metric. What moves it that isn't a reader?" "Engagement" is not a metric, and on modern sites neither is a raw pageview count — auto-loading content, refreshes and instrumentation changes all inflate it with no reader involved. The definition should be precise enough for your analyst to recompute from your own data, and it should come with its failure modes attached. Watch especially for metrics that changed definition between the baseline and the result, and for composite "engagement scores" that cannot be decomposed. ## 5 · "Break it by device and channel — all of it." An aggregate lift can be one strong segment carrying six flat ones, or a mix-shift artifact with no behaviour change anywhere. The full cut, including whatever went the wrong way, is the fastest authenticity test in the meeting: **real interventions have boundaries.** In our own published segment cut, six of seven segments improved and the seventh — direct traffic — fell and is printed in red. If every single segment improved, someone chose the segments. ## 6 · "What did it not do?" Every genuine effect has an edge. Ours, on the record: traffic did not grow — sessions and users stayed flat; the product deepens visits, it does not acquire them. No search-ranking claims. No revenue claims, because we do not hold customers' revenue data. A vendor who cannot name their boundary either has not measured it or would rather you did not ask — and either way, you have learned the thing the meeting was for. ## 7 · "What result would you accept as proof this isn't working?" The falsifiability question, and the one that reorganises the room. A measurable product has a measurable failure condition: *"if multi-page share on enabled properties does not exceed the held-back group by X points after the launch quarter, it isn't working."* A vendor who can complete that sentence is selling a measurement. A vendor who answers with reasons the product cannot fail — learning periods without end dates, effects too diffuse to isolate, value that "shows up in brand" — is selling a belief system with a dashboard. This question also writes your pilot's exit clause for you. Agree the failure condition before signing and the end-of-pilot meeting becomes a reading, not a negotiation. - Time required: **~15 minutes** of any vendor meeting; no data team needed for five of the seven - The two-number answer to Question 1: e.g. **+19.1% launch → +4.0% run rate** — plan on the second - Question 5's authenticity test: our published cut is **six of seven** segments up, the seventh printed in red - Questions that end most meetings fastest, by our observation: **3 and 7** ## FAQ It is fifteen minutes of specificity in a meeting that was going to be an hour of adjectives. Good vendors visibly relax when the questions get concrete — measured products are easier to sell to people who ask measurable questions. Then the pilot creates the answers: staged rollout, frozen metrics, four-week baseline, pre-agreed failure condition. "No data yet" is fixable by design; "no interest in data" is not. Seven. The other six audit the past; the falsifiability question governs your future. Everything else in the relationship inherits its honesty from whether that sentence got completed. ## The printable version, for the meeting The Vendor Claim Audit Checklist expands these seven questions into fifteen checks — what to ask, the red flag, and what a pass looks like. ## See what your session depth looks like We read your analytics with you for twenty minutes and tell you what share of your sessions stop at the first page. You leave with the annotated read, whether or not we ever talk again. 20 minutes, your analytics, no deck. [Book my session-depth read](/request-a-demo/)