The problem with the typical A/B test for a direct mail fundraising appeal or an email appeal is that it’s just too careful and conservative. That’s the point in this post from Seth’s blog. He says that we tend to test things that are too similar because, basically, we’re afraid to fail.
It’s true. When the question of testing comes up in a creative meeting for a fundraising appeal, lots of times the discussion will revolve around testing a closed face envelop versus a window envelope, or an appeal letter with a photo versus without the photo, or a handwritten margin note versus without the margin note, or an email appeal with a Give Now button versus a Donate Now button.
Tests like these are all but guaranteed to produce either a tie or a very, very slight win. In either case, we don’t learn much about the creative or the donors – which was the whole point of testing in the first place.
Does this mean you should always test some crazy new thing and swing for the fences? Not necessarily. Say you have a blockbuster control that’s blown everything else out of the water. Then it would probably make sense to test some minor things to generate incremental gains, provided you want to keep the control going instead of beating it.
Or say you want to see if you can reduce costs without hurting revenue. Then it would probably make sense to test the appeal, for example, with and without the insert. You may find it does just as well without the extra piece, which means cost goes down a little so overall revenue goes up a little.
But in a lot of cases, it’s more instructive to test, as Seth says, “radically different alternatives.” More panic-inducing too. But also more instructive.