August 16, 2024
Stay a while and listen. It’s the CRO Roundtable Roundup!
Thanks to Matt Beischel, Iqbal Ali, Shiva Manjunath, Craig Sullivan, Nils Koppelmann, and Slobodan Manić for joining us. Want to get in on the action and talk with cool CRO people like this? Then join us next Friday.
One Sentence Takeaway
Integrating AI into hypothesis creation and prioritization can significantly enhance the efficiency and effectiveness of experimentation processes.
This Week We Discussed:
- How AI can assist in writing hypotheses and experiment design
- improving the quality of hypotheses
- augmenting human creativity in problem-solving
- potential of AI to understand and predict experiment outcomes
- Benefits of integrating customer research into hypothesis creation
- Growth of CRO in the Latin American market
- When to decide what metrics should be used in an experiment
- problem-first approaches
- Use of metrics in prioritization
- shifting from subjective to objective-based metrics
- inefficiencies of current prioritization frameworks
- Frequentist vs. Bayesian
- both have their place in experimentation
- understanding the null hypothesis
- Frequentist methods rely on disproving the null hypothesis
Bayesian methods update assumptions based on prior data
- Evaluating the efficiency of experimentation processes
- measuring time-to-decision
- visualization techniques to aid in decision-making from experiment results
- automating the decision-making process
- Frameworks are a guide, not a set of rules
- applying the context of your use-case is necessary
- The impact of statistical illiteracy on experiment outcomes
- statistical literacy is crucial for meaningful experimentation but often lacking
- Why using A/B testing tools as deployment tools is a bad idea
- The role of organizational culture in the success of experimentation
- automating mundane tasks can free up analysts for more critical work
- having to work with multiple tool sets
Quotes of the Week
Standout quips from this week’s roundtable
“It’s like getting on a bike after a long time, not riding” – Craig Sullivan
CRO Link Digest
Links to interesting, amusing, thought-provoking, or downright silly content shared at this week’s roundtable
- When do you first document metrics, on idea or experiment level? – LinkedIn post by Nils Koppelmann
- Hypothesis Kit V4 – Medium article by Craig Sullivan
- The Lever Framework – Blog article by Conversion
- 7 Levels of Conversion Framework – PDF by André Morys
CRO Book Club
Naked Statistics
by Daniel Priestly
Recommended by Iqbal Ali
Once considered tedious, the field of statistics is rapidly evolving into a discipline Hal Varian, chief economist at Google, has actually called “sexy.” From batting averages and political polls to game shows and medical research, the real-world application of statistics continues to grow by leaps and bounds. How can we catch schools that cheat on standardized tests? How does Netflix know which movies you’ll like? What is causing the rising incidence of autism? As best-selling author Charles Wheelan shows us in Naked Statistics, the right data and a few well-chosen statistical tools can help us answer these questions and more. – Amazon Link (non-referral)
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies. This week we had some sidebars about:
- Sci-fi integration of the human body with technology
- body modification
- installing microchips into your brain
- instant language translation
Off-Topic Shareables
- Neuromancer – Award-winning Scient Fiction Novel by William Gibson