Flashing energizers score 50 points. It’s the CRO Roundtable Roundup!
Thanks to Matt Beischel, Iqbal Ali, Craig Sullivan, Bhavik Patel, and Slobodan Manić for joining us.
Want to get in on the action and talk with cool CRO people like this?
One Sentence Takeaway
I’m a busy person; give me the TL:DR
Experimentation measurement is evolving toward sophisticated quasi-experimental models as traditional A/B testing becomes less feasible in real-world environments.
The Notes
What we discussed this week
- Automation and AI for CRO workflows
- Using n8n to automate logging, alerting, and stat processing
- AI agents route data to specific tools for interpretation
- Slack integrations for automated notifications and reporting
- Blending no-code automation solutions with custom scripting
- Challenges with tool vendors and interfaces
- User frustration with rigid, legacy UI decisions
- Agencies often build custom dashboards over vendor backends
- Tool vendors need flexible APIs and agent-based approaches
- Shifts in experimentation tooling
- Some orgs run 80% of releases as A/B tests
- Others treat experimentation as phased rollouts with localized monitoring
- Lacking pre-test planning undermines result validity
- Strategic value of in-house roles vs consulting
- In-house allows deeper ownership and strategic input
- Agencies focus on deliverables, not outcomes
- Freelance work is unstable but offers independence
- Measurement complexity in omnichannel environments
- In-store data limits traditional ROI tracking
- Media mix modeling (MMM) used for broad brand impact over D2C-only metrics
- Need to evolve past strict digital attribution models
- AI overuse in automation contexts
- Mistaking automation problems for AI problems
- Scripts often more effective than generative models for stats
- AI more useful for analysis presentation, not computation
- Human role in AI workflows
- AI helpful in drafting, editing, and accelerating—not replacing
- Human judgment is still critical for data interpretation
- Must train models with rich metadata and context
Hey Experimenter!
We’ll be kicking off the summer 2025 Experimentation Elite conference with our first-ever LIVE CRO Talks Roundtable! We want to see you there; come say hi!
- When: June 4th & 5th
- Where: Birmingham, UK
Use discount code CROTALKS10 for 10% off.
The Quotes
Standout quips from this week
- “This is not an AI problem, this is an automation problem.” – Iqbal Ali
- “You can do more, faster and better—but still get paid the same.” – Matt Beischel
- “If you’re behind the scenes, most of what vendors have is UI.” – Craig Sullivan
- “The value I’m going to bring is critical thinking around how we measure.” – Bhavik Patel
- “Not because AI is that advanced—it’s because the human workforce is not that advanced.” – Slobodan Manic
Book Club
Relevant reads recommended this week
- No books this week, sorry!
CRO Link Digest
Useful and thought-provoking content shared this week
- What is MMM in Marketing? – blog article from Adjust
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies.
No off-topic sidebars this week, sorry!
Sidebar Shareables
Amusing sidebar content shared this week
No sidebar shareables this week, sorry!