CRO Roundtable 193

February 16, 2024

I hope this message finds you well. It’s the CRO Roundtable Roundup!

Thanks to Matt Beischel, Iqbal Ali, Craig Sullivan, and Jorden Lentze for joining us. Want to get in on the action and talk with cool CRO people like this? Then join us this Friday, February 23.

This Time We Discussed:

• Handling mistakes in experiments
  • Human factor – no one is infallible
  • Preemtive measures
    • Having a standardized process/system
    • robust, multi-layered QA
  • Reactive measures
    • Conduct a post-mortem
    • Not attempting to salvage garbage data
    • Fix and rerun
• Aligning expectations
• How good are you at guessing test results?
  • Probably not as good as you think
• Is it OK to stop tests early?
  • Yes, if you understand the stats, context, and consequences
  • Only looking at relative change is a trap
  • Check power, duration, user sample, conversion sample, absolute difference
• One size does NOT fit all in CRO
  • Experimentation programs at large businesses don’t map to small businesses
    • Discrepency between access to capital, resources, momentum, etc.
• Experiment QA processes
  • Quantity of QA depends on quality of tooling
    • Familiarity with software
    • Level of developer experience
  • Things to check:
    • Targeting & audiences
    • Variations
    • Goals
    • Traffic settings
  • Balancing robustness & speed
    • Checking outlier segments
    • Introducing automation – expensive/advanced
    • Experience allows for shortcuts
  • Conduct QA after build AND after live

Quotes of the Week

Standout quips from this week’s roundtable

“If your head has been shoved in a toilet bowl, then everything looks like a toilet” – Craig Sullivan

CRO Link Digest

Links to interesting, amusing, thought-provoking, or downright silly content shared at this week’s roundtable

• No links this week, sorry!

Off-Topic Sidebars

Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies. This week we had some sidebars about:

• The difference between comics and cartoons
  • Comics are static
  • Cartoons are animated

Off-Topic Shareables

• How was your Valentine’s Day?
  • Maintaining relationships over long distances
• Dune 2
• How ‘far’ is far away?