If you’ve read a few articles on conversion optimization, you’ve read them all. That’s the way it seems, anyway.
While the advice is usually on point (have a process, statistical significance does not equal validity, prioritize your tests so you don’t waste time/traffic on silly things), these things are common knowledge now.
We recently put together an email course that featured some of the most intelligent people I know in the experimentation and measurement spaces. They, of course, wrote things that were not often talked about, which inspired me to put together a little round-up.
Here are the 4 ideas from that course that I found most refreshing and the least utilized by the CRO world.
Note: unless otherwise noted, quotes in the piece come from CXL’s email course on optimization in the enterprise.
1. It’s Not (all) About Conversion
This sounds weird on the surface. How could conversion rate optimization not be about the conversion? Well, for many reasons, most of which are due to the short sighted nature of optimizing for conversion rate.
First, conversion rate is not usually a good northstar metric to follow. If you’re tracking micro-conversions and increase that metric, it might still end up being worse off for the macro-conversions. For instance, a “free beer and bacon” sign will make a lot of people walk into a restaurant, but many more will walk out disappointed if you don’t actually offer it .
And if you do offer “free bacon and beer,” or on the web, that would be the equivalent of mass discounts, you’re cutting into your profit, which is a real macro-conversion.
Beyond the business-practical reasons for this tip, the bigger benefit of experimentation is a culture change. In a culture of experimentation, here’s the ideal shift in thinking:
- Instead of “What should I test?” ask, “What problems are my users having?”
- Instead of “What are my competitors doing?” ask, “What makes my favorite online experiences so effective?”
- Instead of “What’s the best color for this button?” ask, “Does this button even matter? Is there a more radical way to approach this landing page?”
This tip came from Adam Avramescu from Optimizely. He summed it up really well:
“It’s about changing your culture to one that answers validated user problems and moves key business metrics (not vanity metrics) by finding key insights and exploring radical ways to use those insights.”
2. Prioritize Human Intelligence
There’s the famous business joke of investing in employees:
CFO asks CEO: “What happens if we invest in developing our people and they leave us?”
CEO: “What happens if we don’t, and they stay?”
We talk a lot about artificial intelligence in optimization, but we can’t stray too far from focusing on human intelligence. In other words, the strength of your optimization program relies on the strength of your team.
CRO teams can generally be structured in two ways: centralized or decentralized.
How do you build a conversion optimization team, particularly a centralized team? You start with a conversion optimization champion, someone with a mind for strategy and a deep understanding of the practice of CRO.
Merrit Aho from Dun & Bradstreet put it this way:
“It’s worth taking the time to hire someone with 3-5 years of CRO experience to be a founding member of the team, although it could take the better part of a year to find such a person.
Alternatively, organizations could start with an agency, putting someone more green in charge of the engagement to learn from the pros.
Either way, this individual needs to establish some healthy, sustainable practices, get some early wins, and demonstrate the high ROI of the function. He or she should be able to partner up with other internal resources such as designers, developers, and analysts when necessary.
After that, your biggest bottleneck will be development. So hire a developer with an inclination for experimentation. This is harder to find than you think, as most developers aren’t used to creating “throwaway” variations, but it’s important to find someone with that mindset.
From there you’ll pick and choose specialists, likely an analyst, a designer, a copywriter, in that order.
3. Defend Yourself from Skeptics
If you’ve been in the CRO space for more than a few weeks, you know that often the biggest hurdle is not technical, but rather organizational. People are skeptical, and sometimes they have vested interests. Sometimes those vested interests are in direct opposition to what your data says. No bueno for you.
Sometimes, you’ll present the results of an A/B test and hear something like this….
“Alex, is it possible that you screwed up?”
In larger organizations, you’re likely to get some version – usually slightly more polite – of this question.
When you hear a question like that, you can assume the person has a vested interest in making sure that the test shows a certain answer. It’s important to not be defensive, and not take it personally.
As it turns out, the best way to deal with this is to be pre-emptive.
When Ryan Farley, co-founder of LawnStarter, was an analyst at Capital One, the business problems were very complex, with lots of variables. To make sure the business was profitable, it was the job of the higher-ups to question every datapoint.
Before Ryan would present to stakeholders, he would run through the results with his manager. She would point out things that may be counterintuitive, and he would then dive into them in order to gain comfort around their accuracy.
Eventually, he would start to recognize these ‘red flags’, and put slides into an appendix explaining what the differences might be. This resulted in fewer meetings and faster decisions, benefitting everyone.
Basically, he pre-empted the presentations by finding potential holes in his story and bulletproofing the presentation with explanations and ready defenses. Being prepared is certainly the best strategy here.
4. Stop Talking About Right and Wrong
So much of optimization seems to be about validating ideas. You do research, come up with a hypothesis, and whether you admit it or not, you’re really rooting for your idea to win.
In actuality, the best thing that could happen is if you’re wrong. It sounds contrarian, but hear me out.
Two contributors to the email course made this point, actually: Erik Johnson and Andrew Anderson.
In the context of selling optimization to your boss/organization, Erik Johnson says a persuasive argument is that testing is a process to finding answers that no one knows for sure.
“Most people have been trained to think they must have answers, have been rewarded when they were right, and punished when they were wrong,” Johnson says. “Sell testing as a relief to this stress: none of us have the answers, but all of us have great ideas.”
Andrew Anderson’s point is also brilliant. Essentially, you should celebrate being “wrong.” It’s the best thing that could happen:
“Talk about how great it is that you found something that worked even better. It was never about a single idea – it is about the constant search for a better outcome. It is about that marginal gain and what the best option was from the tested group, not any single outcome or single idea.
While that person is caught up on why their idea didn’t work, you should be moving them towards the fact that something outperformed what they thought would happen. Not only does this mean the company will now make more money, but it means that you learned something new or re-evaluated false knowledge. That isn’t a bad thing. That’s the best thing that could happen.”
A corollary here is that when you disregard right and wrong answers and celebrate discovery, you embrace a culture of experimentation that can’t be drowned.
No matter the particulars of a given test in this culture, you’ll continue on and pass the competition with insights, learning, discovery, and innovation. It’s hard to beat a company with those core values.
Conclusion
Keep reading good conversion optimization content, but keep an eye out for the rare gems that appear to be uncommon advice. I don’t think any of the above is controversial, it’s just that you don’t hear it discussed as often.
That’s because most people are still trying to introduce a rigorous and profitable CRO program at their company. When you’re still trying to set things up, it’s hard to think about a culture of experimentation, defending test results or the absolution being “wrong.”
If you’re hungry for more uncommon advice, find out more about the email course here.
