This series on A/B testing web pages to increase online donations originally appeared on epolitics. See how you can raise more money by improving your donation pages. You don't need to attract more supporters to your site, you don't need to invest in more online ads; just by testing improvements to your donation pages, you can rake in more dollars online. Read more to learn how you can optimize your website or drop me a line and we can talk about how I can help you raise more online.
Small Change Can Yield Big Bucks: A/B Testing Your Fundraising Landing Pages
What would 150% more online donations mean to your organization? Would an increase like that pay for another staffer? Another round of direct mail? Maybe relieve some pressure on your budget? It might seem unlikely, but sometimes just a small change to the wording of your fundraising landing pages can make a big difference in the results. In the case of one New York City campaign, it boosted online donations by 158%!
Cathy Guerriero was a first-time candidate running for Public Advocate, and as with most first-time candidates, jump-starting her fundraising was difficult. But rather than throw our hands up in the air, we decided to try something different and see if changing the donation page could increase donations. These weren’t big changes, just a different headline and a couple of sentences about why donors mattered and how their donations would make a big impact for the campaign.
And guess what? It worked.
We tested the old version of the donation page against the new version in what’s called an A/B or split test. A line code split visitors to the page into two groups, with one group seeing the old version (variant A) and the other group seeing the new version (variant B). We quickly found that when someone saw the newer version, they were 158% more likely to donate than if they saw the older version. We didn’t change what they were donating to, how they got to the page, or even what the donation form looked like. We just changed the first few words that people would see!
Not only did the campaign bring in thousands of dollars more in donations during the test, but the boost wasn’t temporary, as we found once we made the new donation page the standard. Each and every fundraising email we sent from that point on suddenly became significantly more effective, and at a crucial point in the election. Emboldened by that success, we did other tests and found more page-tweaks to help convert more and more website visitors into donors.
Testing can also improve other parts of your site, too: volunteer signups are another big opportunity. For that test, we added a photo of Cathy Guerriero with some young volunteers to a new variant of our main signup page. It wasn’t even the best photo — it had been taken on a cellphone camera and the light was off — but the kids looked cute. So we went for it…and didn’t just see a small increase in volunteer signups. They literally tripled compared to the old volunteer form! And we’re talking about a tiny change that took maybe 5 minutes to set up, but the results were incredible.
A/B testing isn’t some black magic, and you don’t need a PhD to make these changes to your website. Next, we’ll talk about how you can get more donations or volunteers by testing small, strategic improvements to your website.
How Can I Get More Donations With Just a Small Change? A/B Testing Your Fundraising Landing Pages
I explained how even a small, underfunded campaign can easily use A/B testing to bring in a lot more donations without much work. But how do you put those lessons to work for you?
Whenever I work on a website, I generally have a long list of changes I want to make, and if you’re running a site, it shouldn’t take you long to come up with a similarly long list of things that you don’t like about the site. Maybe we’re talking about elements of the site that might not work well, or perhaps it’s pages that you’re curious if you could improve. For your first tests, look for the big things that everyone will see: don’t worry about changing a few words in a paragraph somewhere on one page which might be only seen by a tiny portion of people. High-visibility means page elements like headlines, images (adding a photo or an image of any kind usually improves the results) and calls-to-action like “Donate Now.” To get rolling, start with just one thing that you think could result in a big change. For instance, with Cathy Guerriero’s campaign, we changed the headline on the donation page.
No matter what system you use, you’ll have to also set the success page, the page you want visitors to reach. That page is probably a thank-you page after the donation or volunteer signup, and anyone who reaches that page was a successful conversion. If you can, also tag those people in the backend of your CRM or donation processing system (so that you know who came from where) and double-check that everything is working properly (i.e., making sure that people are actually going into your system from both page variants, etc.).
The next step is the most frustrating: waiting for results to come in. You’ll want to wait for what’s called “statistical significance,” a fancy term that means that there’s a 95% chance that the results are representative of what would happen if you let the test go on forever. One version should have a 95% likelihood of being the best. How long that takes depends on the difference between the variants. If 2% of people donate when they see page A but 10% of people donate when they reach page B, you would only need 100 visitors (not donors) to be 96% certain that version B is better. (Don’t worry about the math; whatever tool you use should let you know whether the results are statistically significant). But if the difference is only a couple percentage points, it’ll take a while longer and you’ll need more visitors to be certain which version is better.
Then, once the results are in, set the winner to be the standard page (make sure the URL stays the same so that visitors end up at the right page), and start thinking about your next A/B test.
How Do I Get Started with Website A/B Testing? Questions and Potential Pitfalls
So far, we’ve talked about how one campaign managed to increase their online donations by 158% with A/B tests. We went over how A/B tests work. Now, lets make sure you’re ready to avoid problems.
A/B testing seems complicated. Why can’t I just make a change to my website and then compare the new results to the old results, without bothering with an A/B test?
Once you have a system up and running, an A/B test can take only a few minutes to set up. And if you don’t set up an A/B test, you could end up misguided because of external events and actually hurt your website. What if you make the change just before a big event (in the campaign world, perhaps a debate) and then all of the sudden, you get more donations? How do you know whether the increase was a result of the debate or because your website is better? You don’t. With an A/B test, you know that any changes are solely because of the website change, not because of anything else, because you’d see the effects of anything else on both variants, not just one. The upshot: A/B tests help you isolate variables.
How many things can I test at once?
You can run an unlimited number of tests with an infinite number of variants; the only limitations are your time and the number of visitors you can test website changes on. However, if at all possible, only test one aspect at a time. Maybe you want to test this headline or that image — in that case, split them up into separate tests so that you can identify whether the changes are because of the headline or the image, and then run those tests sequentially.
The only time you want to make several changes for one test is if those changes only make sense together. For instance, if the donation page currently asks for contributions to support a local soup kitchen but you think it might be better to emphasize the kids fed by the organization, changing just the headline wouldn’t make much sense. Instead, it’s probably better to just test the new page against the old page and see which does better. Keep in mind, though, that if you’ve worked on one page much more than another, you might be comparing a Picasso to a school-kid’s drawing: they might be superficially similar — both are paintings on paper or both are donation pages — but they are very different in quality.
Where can I learn more about A/B testing?
- The founder of Visual Website Optimizer had a great article in Smashing Magazine about A/B testing with several case studies that might inspire you. While it’s a few years old, it’s still a great resource.
- KissMetrics has a great list of resources specifically for Google’s Content Experiments (previously called Website Optimizer). Their blog is also a great source of advice about online marketing in general.
- How to create an A/B testing plan.
I’m also happy to talk to anyone about how to do A/B testing and increasing donations, volunteer signups, or otherwise making your website better. Reach out!