CRO Roundtable 238

February 14, 2025

Love, death, and robots. It’s the CRO Talks Roundtable Roundup!

Thanks to Matt Beischel, Iqbal Ali, Shiva Manjunath, Madalin Chirila, Craig Sullivan, and Surjit Panda for joining us. Want to get in on the action and talk with cool CRO people like this? Then join us this Friday, February 21.

One Sentence Takeaway

Accessibility and user inclusivity are not just ethical obligations but strategic advantages that can significantly increase reach and usability for everyone.

This Time We Discussed:

  • Accessibility audits and design strategies
    • Visual simulations convince stakeholders
    • WCAG compliance benefits both UX and SEO
    • Cost challenges hinder full accessibility adoption
    • Not all disabilities are permanent; situational and temporary disabilities highlight the importance of accessibility
  • A/B testing and statistical significance pitfalls
    • Tools may falsely indicate significance early, don’t blindly trust it
    • Real-time data can be misleading without proper context
    • Frequentist versus Bayesian analysis
    • The importance of statistical pre-planning
  • The UX of visual comparisons
    • Side-by-side videos outperform raw data graphs
  • Organizational challenges in experimentation
    • Leadership turnover disrupts institutional memory
    • Comparing center-of-excellence and decentralized organizational structures
    • Building a test repository is crucial to combatting staff turnover
  • User education and support for A/B testing tools
    • Tools need robust beginner support; it’s very easy to run tests incorrectly
    • Video onboarding boosts technical proficiency
    • Simplified interfaces demystify statistics, but don’t eliminate advanced/power-user features just for simplicity
  • Approaches to translation and localization
    • Literal translations miss idiomatic nuances
    • Local experts ensure cultural adaptation
  • Community collaboration on open-source QA
    • Cross-platform integration unifies processes
    • Peer review builds a community of practice

Quotes of the Week

Standout quips from this week’s roundtable

  • “It’s a money on the table test.” — Craig Sullivan
  • “Accessibility really benefits everyone.” — Matt Beischel
  • “The tool tells me it’s almost green, which I hate.” — Shiva Manjunath

CRO Book Club

Design for Impact: Your Guide to Designing Effective Product Experiments

by Erin Weigel

Design for Impact is a down–to–earth A/B testing guide. It features the Conversion Design process to operationalize effective experimentation in your company. In it, Erin Weigel gives you practical tips and tools to design better experiments at scale. She does this with self–deprecating humor that will leave you smiling—if not laughing aloud. As a bonus, The Good Experimental Design toolkit presents everything you learn into step-by-step process for you to use each day. – Amazon Link (non-referral)

CRO Link Digest

Links to interesting, amusing, thought-provoking, or downright silly content shared at this week’s roundtable

Off-Topic Sidebars

Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies. This week we had some sidebars about:

  • Marathon running and early morning races
  • Travel-induced illnesses impacting performance
    • Being sick on flights
  • Undercover Boss analogies

Off-Topic Shareables

  • No off-topic shareables this week, sorry!