How to measure whether email and SMS really drive sales
By Toby Dames. Published 5 November 2026.
Why platform reports overstate
Most reports use attribution, which credits a channel when a customer clicked or opened before buying. But some of those customers were going to buy anyway. Attribution cannot tell the difference.
How to run a hold-out test
- Choose the audience, for example customers who are overdue for a reorder.
- Split it at random. Keep about 10% back as the hold-out.
- Send the campaign to the rest.
- After a fixed period, such as 30 days, compare orders per customer and revenue per customer in both groups.
Reading the result
If the sent group generated $4.20 per customer and the hold-out generated $3.10, the campaign added $1.10 per customer. Multiply by the number of recipients, subtract the cost of the offer and your margin, and you have the real return.
With small audiences the difference can be noise. Treat a small gap on a small list as a hint, not proof.
What to do with the answer
Use it to decide where to spend more effort. Flows with a clear incremental effect deserve investment. Flows where the groups look the same may be sending messages that do not change behaviour.
Questions
Does a hold-out cost me sales?
The cost is small, because a 10% hold-out only withholds messages from customers who might have ordered anyway. The value of knowing what works usually outweighs it.
Can I do this with Klaviyo?
Yes. Most email platforms let you create a random split and compare the groups. We set this up on every engagement where list size allows.
Want this worked out on your own data?
The free Profit Leak Detector takes a couple of minutes. Or book 30 minutes and we will look at your numbers together.