Built in about 3 days, going by the project’s file dates: 21 scripts in seven ordered stages, with automated tests, on a public dataset anyone can repeat.
Marketing Measurement · public ad data
Most campaigns that looked worth scaling only looked good in hindsight.
Ad reports keep giving campaigns credit for weeks after their budget is set. I rebuilt 675 reports to show only what was known on decision day.
On decision day, only 39% of the eventual credit existed.
The result
167 of 285 campaigns that looked worth scaling only looked good thanks to late credit.
The 25 with the most late credit looked about 4× cheaper in hindsight. They were picked after the study, to show the problem’s size.
The problem
Credit counted today can belong to a budget decision made weeks ago.
Nothing is broken. Ad platforms usually file a result under the day someone clicked, so an old report keeps improving as late results trickle in. That helps you understand history and misleads you about a decision you already made.
This study measures that gap on a public ad dataset anyone can download. My part was choosing the cutoff and writing the rule down before seeing the answer.
1 hour later
31%
31%
85 campaigns look worth scaling
1 day laterBudget decided
39%
39%
118 campaigns look worth scaling
2 days later
44%
44%
142 campaigns look worth scaling
1 week later
64%
64%
208 campaigns look worth scaling
30 days later
100%
100%
285 campaigns look worth scaling
Will this page’s numbers change? No.
The study uses a closed public archive: a month of ad data that Criteo released for research, plus the month of results that followed, saved on August 20, 2026. Nothing new can arrive in it, so re-running the study gives the same numbers. A live account’s old reports keep changing.
How the correction works
Score every campaign twice: once the usual way, once with only what was known on decision day.
- 01 · The usual report285campaigns looked promising once all the late credit was counted
- 02 · Decision-day view118still looked promising using only credit that existed on decision day
- 03 · Worth another test33passed the remaining checks and earned another experiment
167 dropped outThey only looked good because of credit that arrived late. Another 85 did not clear the next-test bar.
The finding
The gap is real, it is large, and it is not luck.
To rule out luck, I shuffled the dates and re-ran the whole test 20 times, then 999 more. No shuffled run found more than 4 campaigns worth testing; the real dates found 33. See the shuffle chart →
Stopped393 of 675 393
Moved down152 of 675 152
Not enough evidence97 of 675 97
Earned another experiment33 of 675 33
What changes
A correction only matters if it changes the next budget decision.
So I tested both rules on campaigns held back from the first analysis. The usual rule picks 274 campaigns. The timing-corrected rule picks a different, smaller set of 99 with about half the cost per customer, a 50.8% improvement. That is a test on public data, not proof of results in a live account.
One campaign looked 6.1× cheaper than it really was because 431 late credits were filed back into it.
The usual rule, on campaigns set aside
274 campaigns picked
Picks campaigns that look good once late credit is counted
Timing-corrected rule, same campaigns
99 campaigns picked
About half the cost per customer of the usual picks
A question for your own reports
Does this happen in your account?
Probably, and it is not anyone’s fault. Many ad platforms backfill conversions into the day of the click, not the day the conversion occurred.
01
Pull the same report twice. Once today, once from a snapshot 30 days ago, like the readings above. If the older numbers have changed, you have the problem.
02
Check your click-to-conversion lag. Anything longer than your reporting cycle is the size of your blind spot.
03
Replay your last budget decision. Ask what it would have looked like using only the data that existed the morning you made it.
Evidence record
Every number on this page, with its source.
Open the full evidence record
Every value here comes from a dated record built from the public dataset. Nothing is hidden from this summary because the data is already public; the larger working record remains private.
Credit counts are grouped by campaign and day. They are not counts of unique customers or revenue.
The claim
What changed once late credit was removed, including the test on held-back campaigns.
285
campaigns that looked promising in the usual report
118
still promising with decision-day credit only
33
earned another experiment
39%
of the credit existed on decision day
61%
of the credit arrived after the decision
297%
middle cost-per-customer increase, 25 campaigns with the most late credit
393
campaigns stopped
152
campaigns moved down
97
campaigns without enough evidence
274
picked by the usual rule on held-back campaigns
99
picked by the corrected rule on held-back campaigns
50.8%
held-back improvement in cost per customer
0.01782
cost per customer, usual rule, held-back (dataset units)
0.00877
cost per customer, corrected rule, held-back (dataset units)
49.8%
first-analysis improvement in cost per customer
0.01671
cost per customer, usual rule, first analysis
0.00839
cost per customer, corrected rule, first analysis
60.3%
of test budget behind campaigns that failed the timing check (calculated after the planned study)
The checks
Whether luck, the credit rule, false winners, or week-to-week swings could explain the result.
20
planned timing shuffles
33
real result (campaigns worth another test)
0
middle result across planned shuffles
1
highest result across planned shuffles
0
planned shuffles at or above the real result
999
confirmation shuffles
0
middle result across confirmation shuffles
4
highest result across confirmation shuffles
0
confirmation shuffles at or above the real result
0.001
confirmation p-value
100
passed when the last click gets all the credit
81
passed when credit is shared across clicks
45
decisions changed by the credit rule
530
apparent winners before the false-winner check
480
apparent winners after the false-winner check
93
campaigns that stayed above the comparison line
473
campaigns that stayed at or below the comparison line
86
campaigns that moved between both
88.4%
first-week agreement with the full answer
0.00746
uncertainty floor, lowest-volume campaigns
0.00204
uncertainty floor, comparison group
The scale
How much data was involved, who survived the correction, and the one worked example.
9,979
campaign-day groups held back for a second test
8,730
held-back groups that received late credit
87.5%
of held-back groups received late credit
189,430
late credits filed back into earlier days (credits, not customers)
2.62
× the ad views of other campaigns (survivors)
1.27
× the click rate of other campaigns (survivors)
0.16
× the cost per customer of other campaigns (survivors)
77.3%
volume percentile of the survivors
2576437
worked-example campaign ID in the public dataset
0.00828
example cost per customer, usual report
0.05076
example cost per customer, decision-day view
6.1
× difference in the worked example
431
late credits in the worked example
What it took
The result only became useful after two uncomfortable checks.
Two bugs were caught before any result was trusted. Both were fixed, and everything was re-run without changing the plan.
How credit was assigned and how false winners were filtered out. See how I tested it →
Compare notes
Seen this in your own reports?
This is the closest match to my day job, and I’m always curious how other teams handle late credit. I’d like to hear what you’ve seen.