You can test a cold email with 50 prospects if you stay honest about what 50 can tell you. Change one thing between two versions, split the list at random into 25 and 25, send both on the same day, count replies (not opens), and decide your rule for a winner before you look at the results. Expect a nudge, not proof.

Wes sells mobile pressure-washing for commercial properties and has a list of 50 property managers. He wants to know whether naming a price in the first email helps or hurts. This is how he runs the test, and what he does with the answer.

Why small tests are still useful

The textbook objection is right: 25 emails per version is far too few to prove anything. Big companies test with thousands. But you aren't trying to publish a paper. You're trying to choose between two reasonable emails with a few hours of effort, and a small test, read carefully, beats a coin flip. It also forces you to write down what you're guessing, which is half the value.

The danger is over-reading. People see 5 replies against 3 and announce a winner. The point of the rest of this article is to avoid that mistake while still getting something out of the test. If you haven't yet got a baseline, start with what is a good cold email reply rate, which shows how to count.

Pick one thing to change

One change per test. If version B has a new subject line, a new first line, a different ask and a price, and it does better, you won't know why. And with 25 per group, you can only afford to learn one thing.

Things worth testing, roughly in order of what moves replies most: the audience (a different kind of buyer), the offer (what you're proposing), the first line, the ask, and last of all the subject line. Wes picks the offer: version A names a price ("from $180 per building"), version B says "I'd quote on request". Everything else, the subject, first line, length, signature and question, stays word for word the same.

If you want ideas for what to vary, cold email subject lines that get opened and cold email call to action examples give lists to draw from.

Split the list fairly

The two groups should be as alike as possible. If version A goes to your best-fit prospects and version B to the leftovers, B will lose for reasons that have nothing to do with the email.

The simplest fair split: put the list in a spreadsheet, add a column of random numbers (a RAND() formula), sort by that column, and give the top 25 to A and the bottom 25 to B. If you'd rather not use formulas, sort the list by company size and then alternate: A, B, B, A, A, B, B, A, and so on. Either way, check afterward that both halves have a similar mix of big and small companies and nearby and far ones.

Send both versions the same day

If A goes out Monday morning and B on Friday afternoon, you've changed two things: the email and the day. Send both on the same day, in the same window, in alternating order. If you spread sends across several days (see how many cold emails per day from Gmail for safe volume), do half of each group each day.

Both versions go from the same address and have the same signature and opt-out line. Use the same follow-up for both too, sent on the same schedule. Then wait. Replies come over a week or more, and counting after two days will undercount.

Count replies, not opens

Open counts depend on a hidden image loading, and many mail apps load it automatically, so a subject line can show a huge open rate that means nothing. The reliable thing to count is replies from real people, and the better version of it is positive replies, those that show interest. Why to stop measuring opens goes through the problem.

Wes's results after ten days:

  • Version A (price named): 25 sent, 1 bounced, 24 delivered, 5 replies, 3 positive.
  • Version B (quote on request): 25 sent, 0 bounced, 25 delivered, 3 replies, 2 positive.

So A got 5 of 24 (about 21 percent) and B got 3 of 25 (12 percent). That looks like a clear win for naming the price. It isn't, and that is the central point of this article.

Decide with a rule written in advance

A difference of two replies between groups this small can easily happen by chance. If both emails were identical in quality, you'd see a gap of two or more in a large share of tests, perhaps about half of them. So a gap of 2 means almost nothing.

The way to protect yourself is to write the rule before you send. Wes wrote this on the first page of his sheet: "B beats A only if it gets at least 4 more replies out of 25. Smaller gaps count as a tie. In a tie I keep the version that is shorter or simpler and test something else."

Four is a judgment call, not a statistical threshold; even a gap of four can occur by luck now and then. Treat a gap that big as a reason to lean toward the winner and to confirm it in the next round, not as a result you can announce. In Wes's case the gap was two. It's a tie, and he keeps version A because naming a price is simpler and saves a back-and-forth.

A tie is a real result. It tells you the price line wasn't the big lever, and he can spend the next test on something likelier to matter, such as who he's writing to.

Roll out the winner

When a version wins by your rule, or ties and you keep the simpler one, send it to the rest of your list. Then test again with a new challenger. That is how a small sender improves: not one big experiment, but a series of small ones, each starting from the best version so far.

Keep a running log: date, what changed, how many sent, replies, positives, and what you decided. After five rounds you'll have a record of what moved your numbers for your buyers, which is worth more than any general advice, including this article. For the bigger picture of testing when you haven't found a fit yet, read outbound before product-market fit.

Also, remember the human side. A reply with a question is worth more than a reply with a polite no, and a single conversation can teach you more than any percentage. Read the replies, not only the counts. Notes on making each email specific are in personalization at low volume.

Rocketship makes small tests easier because it runs the sending and the follow-up for you. Its AI worker sends from your own Gmail, follows up on schedule and reads the replies, so your log is the thread itself. See the AI workers page for how it runs.

Fifty prospects won't give you certainty, but they will tell you which direction to go next. Rocketship finds the buyers, emails them from your own Gmail, follows up, and books the meeting on its own. Plans that get you customers start at $24.99 a month (Launch), and every monthly plan starts with a 7-day free trial, card required.