October 18, 2024
Select 1 or 2 players. It’s the CRO Roundtable Roundup!
Thanks to Matt Beischel, Iqbal Ali, Craig Sullivan, and Tim Mehta for joining us. Want to get in on the action and talk with cool CRO people like this? Then join us this Friday, September 25.
One Sentence Takeaway
Trustworthy presentation of data is critical in A/B testing, requiring transparency and realistic context to prevent misleading conclusions.
This Time We Discussed:
- The validity of A/B test results
- Being suspicious of perfect, rounded test figures
- The importance of transparency in presenting data results
- Challenges with predictive engines in CRO
- Predictive models often lack context to provide accurate results
- Prioritization tools can assist but may not always be reliable
- Using AI in qualitative analysis
- AI tools can help with busy work in analyzing survey data
- GPT for simplifying technical documents
- The role of context in conversion rate optimization
- External factors and business context are crucial in interpreting test results
- Test novelty effects diminish over time
- Experiment prioritization strategies
- Quick and low-cost tests are favored, even with minimal data backing
- Some tests are beneficial despite a lack of prior research or evidence
Quotes of the Week
Standout quips from this week’s roundtable
- “High on my own supply.” – Iqbal Ali
- “This sounds insightful, but I’m really doubting that it’s based on anything in reality.” – Tim Mehta
- “Our job isn’t to come up with the ideas. Our job is to run the experiments and tell you it didn’t work.” – Craig Sullivan
CRO Link Digest
Links to interesting, amusing, thought-provoking, or downright silly content shared at this week’s roundtable
- Product Idea Confidence Meter – Downloadable spreadsheet
- How can you better predict the potential outcome of an AB test idea? – LinkedIn post by Tim Mehta
- How would you react if you saw these experiment results? – LinkedIn post by Manuel Da Costa
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies. This week we had some sidebars about:
- Jet lag recovery tips
- Using melatonin
- Craig’s garden
- Making beetroot curry
- Travel planning to Paris and London
Off-Topic Shareables
- Secret London – Travel & tourism recommendations for London