Customization available upon request. It’s the CRO Roundtable Roundup!
Thanks to Matt Beischel, Iqbal Ali, Craig Sullivan, and Jorden Lentze for joining us.
Want to get in on the action and talk with cool CRO people like this?
One Sentence Takeaway
I’m a busy person; give me the TL:DR
Multi-armed bandits (MABs) can optimize short-term decisions when type 1 error control isn’t essential, but AB testing remains vital for learning and confidence in results.
The Notes
What we discussed this week
- Multi-Armed Bandits (MAB) vs AB Testing
- AB testing controls type 1 error, MABs don’t
- Use MABs when random selection is acceptable
- Bandits are more efficient for perishable contexts
- When to Use Bandits
- Suitable for headline tests or short-term campaigns
- Ineffective if you need error guarantees
- High return user scenarios can bias outcomes
- Types of Bandit Algorithms
- Epsilon Greedy: explores randomly 20%, exploits 80%
- UCB (Upper Confidence Bound): selects based on upper confidence intervals
- Thompson Sampling: draws from posterior distributions to determine selections
- Boltzmann: uses temperature parameter to adjust conservatism or creativity
- Tool Limitations and Trade-offs
- Tools may promote MABs for speed or novelty, not appropriateness
- “Always-on” testing claims can mislead due to lack of statistical guarantees
- Bandits are vulnerable to randomness if no real difference exists
- Contextual Bandits
- Combine bandit selection with predictive models
- Effective when user context reliably predicts performance
- Not useful if no structure exists in user-treatment interaction
- Random vs Adaptive Assignment
- Updating only from random policy avoids bias
- Propensity score weighting can address complexity from adaptive selection
- Challenges with Stopping Criteria
- Many bandit approaches lack clear endpoints
- Epsilon-first allows for fixed-horizon stopping
- Ad hoc stopping rules add complexity and risk
- Algorithm Robustness over Result Optimization
- Simple methods like epsilon greedy are favored for reliability
- Sophisticated algorithms may not yield meaningful improvements
- Understand problem context over tool sophistication
- Evaluating Bandits Against AB Testing
- You can AB test your bandit program
- Compare random policy vs. bandit-driven selection to assess value
- Determine if added complexity is justified
Hey Experimenter!
We’ll be kicking off the summer 2025 Experimentation Elite conference with our first-ever LIVE CRO Talks Roundtable! We want to see you there; come say hi!
- When: June 4th & 5th
- Where: Birmingham, UK
Use discount code CROTALKS10 for 10% off.
The Quotes
Standout quips from this week
- “The tool itself makes it easy to execute, but the design around makes it too easy to make mistakes.” — Matt Beischel
- “Solutions live in problems. And that’s why it’s important to understand the problem.” — Matt Gershoff
Book Club
Relevant reads recommended this week
No books this week, sorry!
CRO Link Digest
Useful and thought-provoking content shared this week
- Adjusted Power Multi-Armed Bandit – Conductrics blog article by Matt Gershoff
- Do No Harm or AB Testing without P-Values – Conductrics blog article by Matt Gershoff
- n8n – Flexible AI workflow automation software
- Zapier – No-code tool for automating app workflows
- Webflow Optimize – AI-driven platform for dynamic website personalization testing
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies.
- Misplacing laptops
- Farming rare drops in video games
Sidebar Shareables
Amusing sidebar content shared this week
- Secret London – ultra-shareable online guide to news, events, and things to do in London
- Barts Pathology Museum – medical museum housing one of the largest collections of human pathological specimens in the UK
- Can’t Get You Out of My Head – part one of a six-part documentary series investigating the history of modern political populism by Adam Curtis