Every marketing team argues about the same question: do we write "20% off" or "save 100 EGP"? The truth is nobody knows until they test — and the gap between two phrasings can be double the response rate. A/B testing turns that argument into a number.
The idea, simply
You split your audience into two random groups, send version A to one and version B to the other, and compare. The winning group decides which version goes to everyone else.
Change exactly one thing
The single most important rule. If you change the offer, the wording and the timing together, you'll learn that one version won but never why — and that "why" is what accumulates into your team's expertise.
Things worth testing
- The first line: the most important element by far — it's what appears in the notification and decides whether the message gets opened at all.
- Offer framing: percentage versus absolute amount.
- The call to action: "Order now" versus "See the offer".
- With or without an image: images draw attention but slow loading.
- Length: a two-line message versus a full paragraph.
Sample size
Testing on 20 people is meaningless — any difference is chance, not a result. The practical rule: two groups of no fewer than 200–300 each before a difference means anything.
When is a difference real?
Not every gap in the numbers means one version is better. Practical rules that protect you from a wrong conclusion:
- A gap under 20% on a sample of 200: probably chance. Repeat before acting on it.
- A gap over 50%: a genuine result worth building on.
- Count conversions, not just percentages: 3 orders versus 2 is not a difference — the numbers are too small.
- Give it enough time: don't call it after two hours. Most replies arrive within 24 hours, but not within the first 20 minutes.
Read the right number
Delivery rate measures your gateway, not your message. What matters is response: replies, clicks, orders. A message delivered to 100% that nobody answers has failed; one delivered to 90% that produces orders has succeeded.
A testing plan for your first three months
Rather than testing at random, work from highest impact down:
- Month one — the first line: biggest effect and fastest result. Test four different openings.
- Month two — the offer itself: percentage versus amount, free shipping versus discount.
- Month three — format and timing: image or none, message length, send windows.
Then repeat — your audience shifts, and last year's winner isn't necessarily this year's.
Pair it with the send-time optimiser
Once you know the winning message, timing is the next lever. The send-time optimiser proposes the window your audience actually engages in — so you send the best message at the best moment.
Common mistakes
- Changing more than one element in the same test.
- Stopping the test early as soon as one version pulls ahead.
- Testing on a non-random split (say A to Cairo customers and B to Alexandria).
- Judging on delivery rate rather than response.
- Not logging the result — so you rerun the same test six months later.
Log your results
Keep a simple file: date, what you tested, the winner, by how much. After ten tests you'll have an internal playbook about your specific audience — worth more than any generic advice.
Frequently asked questions
Can I test more than two versions? Technically yes, but each extra version needs a larger sample. Start with two.
What split ratio? 50/50 for a clean test. On a large campaign, test on 20% and send the winner to the remaining 80%.
Does testing cost extra? No — the same number of messages, just divided across two versions.
