February 14, 2025
Love, death, and robots. It’s the CRO Talks Roundtable Roundup!
Thanks to Matt Beischel, Iqbal Ali, Shiva Manjunath, Madalin Chirila, Craig Sullivan, and Surjit Panda for joining us. Want to get in on the action and talk with cool CRO people like this? Then join us this Friday, February 21.
One Sentence Takeaway
This Time We Discussed:
- Accessibility audits and design strategies
- Visual simulations convince stakeholders
- WCAG compliance benefits both UX and SEO
- Cost challenges hinder full accessibility adoption
- Not all disabilities are permanent; situational and temporary disabilities highlight the importance of accessibility
- A/B testing and statistical significance pitfalls
- Tools may falsely indicate significance early, don’t blindly trust it
- Real-time data can be misleading without proper context
- Frequentist versus Bayesian analysis
- The importance of statistical pre-planning
- The UX of visual comparisons
- Side-by-side videos outperform raw data graphs
- Organizational challenges in experimentation
- Leadership turnover disrupts institutional memory
- Comparing center-of-excellence and decentralized organizational structures
- Building a test repository is crucial to combatting staff turnover
- User education and support for A/B testing tools
- Tools need robust beginner support; it’s very easy to run tests incorrectly
- Video onboarding boosts technical proficiency
- Simplified interfaces demystify statistics, but don’t eliminate advanced/power-user features just for simplicity
- Approaches to translation and localization
- Literal translations miss idiomatic nuances
- Local experts ensure cultural adaptation
- Community collaboration on open-source QA
- Cross-platform integration unifies processes
- Peer review builds a community of practice
Quotes of the Week
Standout quips from this week’s roundtable
- “It’s a money on the table test.” — Craig Sullivan
- “Accessibility really benefits everyone.” — Matt Beischel
- “The tool tells me it’s almost green, which I hate.” — Shiva Manjunath
CRO Book Club
Design for Impact: Your Guide to Designing Effective Product Experiments
by Erin Weigel
Design for Impact is a down–to–earth A/B testing guide. It features the Conversion Design process to operationalize effective experimentation in your company. In it, Erin Weigel gives you practical tips and tools to design better experiments at scale. She does this with self–deprecating humor that will leave you smiling—if not laughing aloud. As a bonus, The Good Experimental Design toolkit presents everything you learn into step-by-step process for you to use each day. – Amazon Link (non-referral)
CRO Link Digest
Links to interesting, amusing, thought-provoking, or downright silly content shared at this week’s roundtable
- WCAG 2.2 guidelines – recommendations for making web content more accessible by the W3C
- Web Disability Simulator – Chrome browser plugin
- Web Accessibility Evaluation Tools – by Wave
- How A Screen Reader User Surfs The Web – YouTube Video by Smashing Magazine
- The Dance of the p Values – YouTube Video by Geoff Cumming
- The debate between Bayesian and Frequentist approaches isn’t really relevant for most A/B testing cases – LinkedIn post by Yuzheng Sun
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies. This week we had some sidebars about:
- Marathon running and early morning races
- Travel-induced illnesses impacting performance
- Being sick on flights
- Undercover Boss analogies
Off-Topic Shareables
- No off-topic shareables this week, sorry!