You're running retargeting: bringing back users who registered but didn't buy. You're spending money. And somewhere in the back of your mind there's a nagging question: would this person have come back on their own? Are we paying for something that would've happened anyway?
That's cannibalization. If it's been bothering you – here's a case study.
I worked with a mobile app – a social network with gaming mechanics, multiple geos: Africa, Europe, the US. Part of the budget went toward re-engaging users who had already installed the app. The question was direct: do they actually bring more value – or do they behave the same as users we never touched?
The ideal way to answer this is a controlled A/B test with proper group separation. But you can't control traffic distribution that precisely at the platform level. What we did have was rich historical data. Enough to work with.
We measured time in the app. Monetization was largely ad-based, so time translated almost linearly into revenue.
Sounds simple: compare re-engaged users against those we didn't touch. But there's a trap. Users who responded to an ad are already more engaged by definition. Selection bias. A direct comparison will always make the "re-engaged" group look better – not because of retargeting, but because of who they already were.
So we cut the samples more carefully. We built a matrix of segments by bucket: how many days a user was active, how long they were inactive before re-engagement. Plus geo, device, purchase behavior. And we compared within identical buckets: some re-engaged, some not.
The matrix revealed a clear pattern. Segments split into two types:
1. Worth re-engaging. The time-in-app uplift (i.e., revenue) exceeded the cost of re-engagement. Usually users who had invested something in the app and then drifted – they were worth bringing back.
2. Not worth re-engaging. The user left immediately after install, retargeting barely affects behavior – we're just burning budget here.
This let us direct retargeting spend only toward the segments that actually delivered results.
One caveat: this was a synthetic experiment on historical data – OLS, CausalImpact, and a couple of other models. Causal conclusions without a real experiment always carry noise. But in return, we didn't touch a single live campaign, lost zero budget, and got meaningful directional insight in a situation where a proper experiment wasn't feasible.
Want a stronger data analyst role or a raise? Grab the FREE Product Analyst Playbook and get the exact roadmap to your next offer.



