Marketing Measurement → How I Tested It
How I tried to make the finding disappear before I trusted it.
Four tough questions about the Marketing Measurement study, each with its answer attached.
I scrambled the timing, changed how credit was assigned, and hunted for false winners. The finding held, and its flaws stay on the record.
Four questions, with the answer already attached.
Could it just be luck? Not even close.
I shuffled the timing and re-ran the whole test, first 20 times under the written plan, then 999 more times to confirm. If the finding were luck, some shuffled runs would land near the real result.
See the exact counts
Open the question you want to inspect.
Each one holds the full answer and its chart, in normal text that stays readable on a phone.
Did changing who gets credit change the answer?
Yes: 45 campaign decisions changed. On the same corrected data, 100 campaigns passed when the final click got all the credit and 81 passed when credit was shared across several clicks.
Could testing many campaigns create false winners?
The public data has 675 campaigns. 59 didn’t have enough activity for this comparison, leaving 616. The first pass found 530 apparent winners; 480 remained after a stricter check for false winners.
For technical readers, the recorded cutoff was q = 0.1 and p < 0.002.
Did the direction hold week by week?
Of 652 campaigns with enough weekly activity, 473 stayed at or below the comparison line, 93 stayed above it, and 86 moved between the two.
The first week agreed with the full answer 88.4% of the time. My written prediction didn’t fully match what happened, so the mismatch stays in the record.
Does campaign size change the answer?
The campaigns that survived had 2.62× the ad views, 1.27× the click rate, and 0.16× the cost per customer of the ones that didn’t. By volume they ranked above about 77.3% of campaigns, so the correction depends on campaign size and shouldn’t be applied the same way everywhere.
Seven stages, each tested before a number is trusted.
The build: 21 scripts run against a public Criteo advertising dataset, with seven ordered stages from collecting the data through making the decision.
The checks: separate steps test false alarms, week-to-week stability, and the other questions shown above. Automated tests must pass before a number is trusted.
The record: the dataset, run identifier, and data-shape check are kept with the evidence. The analysis plan was written before the false-alarm results were known; when a final rerun exposed an ordering concern, every later output was regenerated without rewriting the plan.
The evidence is here when you need it.
You don’t need this page to understand the finding. It’s here so the claim can be checked when the decision matters. The same four checks work on any campaign report before a budget meeting.