February 21, 2025
Join for free at the front desk. It’s the CRO Roundtable Roundup!
Thanks to Matt Beischel, Craig Sullivan, Shiva Manjunath, and Surjit Panda for joining us. Want to get in on the action and talk with cool CRO people like this? Then join us this Friday, February 28, 2025.
One Sentence Takeaway
This Time We Discussed:
Use of AI at Different Business Scales in Experimentation
- Small agencies use AI for content and research automation
- Larger agencies invest in workflow automation and AI-driven QA
A/B Testing & Experimentation Best Practices
- Test duration should align with real-world conversion cycles
- Avoid premature stopping of tests without statistical justification
- Stopping tests early—valid in rare cases but often leads to poor decisions
- Using whittling to refine multivariate tests
- Adding a tail-end to an experiment in order to capture delayed conversions
- Multivariate and A/B/n testing can provide better learning if you have an adequate sample size
Statistical Challenges & Tool Limitations
- Some tools falsely indicate statistical significance too early
- Session-based models led to inaccurate test conclusions
- Why pre-planning sample sizes before running tests is important
- Tool responsibility vs. user responsibility in misinterpretations
- Role of guardrail metrics in preventing negative business impact from tests
Applying AI to Productivity & Automation
- AI can assist in structuring research and summarizing insights
- The problem with synthetic users vs. real user testing
- How to generate hypotheses based on research data
- Cleaning and visualizing survey data
- AI as an ideation assistant, supplementing human-generated test ideas
- Time and motion studies to determine where AI adds real value
- Getting off the hype train to find AI’s real potential
- Evaluating tools based on real business needs rather than vendor marketing
- Comparing several different meeting transcription services
Quotes of the Week
Standout quips from this week’s roundtable
- “Propagating poor practice isn’t helping decision-making.” – Matt Beischel
- “I got addicted to Python over the weekend.” – Craig Sullivan
- “Is it the CRO’s fault or the tool’s fault when a bad stat-sig call is made?” – Shiva Manjunath
- “Replacing users with machines in research? No. Using AI to enhance user research? Yes.” – Craig Sullivan
CRO Link Digest
Links to interesting, amusing, thought-provoking, or downright silly content shared at this week’s roundtable
- Nielsen Norman Group – UX design training, consulting, and thought leadership
- The principle of “affordance” in UX design – article from the Interaction Design Foundation
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies. This week we had some sidebars about:
- Travel logistics for conferences
- Work-life balance and long-term career stability
- AI hype vs. actual usefulness
Off-Topic Shareables
- No off-topic shareables this week, sorry!