Net weight 82 oz. It’s the CRO Roundtable Roundup!
Thanks to Matt Beischel, Iqbal Ali, Slobodan Manić, Stu Bowker, Craig Sullivan, and Marcella Sullivan for joining us.
Want to get in on the action and talk with cool CRO people like this?
One Sentence Takeaway
I’m a busy person; give me the TL:DR
Poorly managed client onboarding and AI overhype underscore the critical importance of context, communication, and realism in tools and services.
The Notes
What we discussed this week
- Impact of software vendor acquisitions
Client migration fears
Price increases
Service degradation
Share your QA failures
A/B test once broke add-to-cart for an hour, costing ~$30k
GA data showed hidden promo fields unexpectedly increased conversion
- Your A/B Testing Software is not a CMS
Don’t use 100% traffic allocation as a workaround for slow dev cycles
You’ll have to pay that tech debt at some point
- Band-aids on top of band-aids
Over-hyping AI tools; downplaying their limitations
App builders fail to deliver usable output
Over-reliance on AI for automation vs. structured scripting
Scoring interview transcripts with AI lacks consistency and context
“Black box” AI tools not viable for production or support
Many AI agents are just glorified if-else statements
Environmental impacts of AI on sustainability
Placing hosting in cold climates and edge computing for efficiency
DeepSeek and other models showing power-efficient AI potential
AI as time-saver vs. energy-consumer tradeoff
Your experience with AI-based CRO tools
Tools promising automatic test generation are often hollow
Prompt-based test idea tools can be effective when combined with deep research
Scoring and context inconsistencies persist
Hey Experimenter!
We’ll be kicking off the summer 2025 Experimentation Elite conference with our first-ever LIVE CRO Talks Roundtable! We want to see you there; come say hi!
- When: June 4th & 5th
- Where: Birmingham, UK
Use discount code CROTALKS10 for 10% off.
The Quotes
Standout quips from this week
“The reward for my effort was punishment.” — Matt Beischel
“You don’t buy Legos in a black box.” — Slobodan Manić
“What I wanted was to not fail, not to fail gracefully.” — Iqbal Ali
Book Club
Relevant reads recommended this week
No books this week, sorry!
CRO Link Digest
Useful and thought-provoking content shared this week
- Lovable – AI-powered software engineer
- Why I’m not worried about LLMs long context problem – Medium article by Social Scholarly
Off-Topic Sidebars
Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies.
Dead Internet theory
Potentially 60% of web traffic is from bots
AI firms crawling excessively without a clear purpose
The movie “Demolition Man”
All restaurants are now Taco Bell
…or Pizza Hut depending on the region where you saw the movie
Sidebar Shareables
Amusing sidebar content shared this week
- Atomfall: How a forgotten nuclear disaster inspired a video game – Article by Peter Gillibrand & Tom Richardson for the BBC
- Unsolved Mysteries: The disappearance of Dave Bocks – Unsolved Mysteries wiki article