September 13, 2025
Am I mistaken in thinking that this is the best idea ever? It’s the CRO Roundtable Roundup!
Thanks to Matt Beischel, Iqbal Ali, Craig Sullivan, Jorden Lentze, and Marcella Sullivan for joining us. Want to get in on the action and talk with cool CRO people like this? Then join us next Friday.
One Sentence Takeaway
Experimentation is not a linear, factory-like process but requires flexibility, adaptability, and the ability to learn from each result to guide future tests.
This Week We Discussed
- Experimentation as a flexible process
- Experimenting isn’t like an assembly line
- Test results should guide future actions
- New tools in experimentation
- Varify and Intelligems as emerging tools
- Simplified tools can mislead non-technical users
- Planning and flexibility in testing
- Planning must accommodate change during execution
- Long-term test roadmaps rarely work as expected
- Organizational dynamics in experimentation
- Tension between UX teams and experimentation teams
- Successful experimentation requires cross-team alignment
- A/B testing tool comparisons
- Dynamic Yield’s flawed test calculator criticized
- Booking.com’s calculator remains a reliable option
- Learning from failures and counterintuitive results
- Counterintuitive test results provide the greatest insights
- Teams often discard confusing results, missing key learnings
Quotes of the Week
Standout quips from this week’s roundtable
- “Experimentation can’t be applied as an assembly line in a factory.” — Matt Beischel
- “Nobody’s really happy with their tool, right?” — Craig Sullivan
- “Plans will definitely change right the moment you start testing.” — Jorden Lentze
CRO Link Digest
Links to interesting, amusing, thought-provoking, or downright silly content shared at this week’s roundtable
- Varify – Fixed-Price A/B Testing Software
- Intelligems – A/B Testing and Analytics Software
- A/B Testing Tools Comparison Chart – compiled by Speero
- This is the Worst Sample Size Calculator – LinkedIn post by Ryan Thomas
- A-B Test Guide – A/B Test Sample Size and Significance Calculator
- A/B Test Calculator – by Speero
- Sample Size Calculator – by Evan Miller
- Sample Size Calculator – by Optimizely
- Power Calculator – by Booking.com
- Gigacalculator – power & sample size calculator by Georgi Georgiev
- An Unexpected Result when Testing Social Proof – LinkedIn post by Frazer Mawson
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies. This week we had some sidebars about:
- The impact of Elon Musk’s management style on product teams
- Salsa verde made from home-grown tomatillos
- The FAA’s role in space regulation and its effects on innovation
- Classic versus modern education methods
- Featuring dogs on team pages and what it implies about corporate culture
Off-Topic Shareables
- The Voight-Kampff Test – YouTube Video
- A chart comparing the improvements of Chat GPT 4o and o1 – X post by Rohan Paul