CRO Roundtable 219

September 20, 2024

I hope this message finds you well. It’s the CRO Roundtable Roundup!

Thanks to Matt Beischel, Iqbal Ali, Jakub Linowski, and Jorden Lentze for joining us. Want to get in on the action and talk with cool CRO people like this? Then join us this Friday, September 27.

One Sentence Takeaway

Targeting and triggering in server-side testing can create significant complexities, particularly with enrollment, segmentation, and scroll-depth tracking.

This Week We Discussed:

  • Test Targeting and Triggering
    • Server-side vs. client-side complexities
    • Scroll-depth tracking as a key trigger for tests
    • Over-targeting leading to noise in test data
  • Challenges with Server-Side Testing
    • Lack of enrollment awareness among developers
    • Issues with feature flagging in server-side experiments
    • Manual enrollment triggers to avoid selection bias
  • Sequential Assignment in Testing
    • Assignment and enrollment as distinct phases
    • Measuring the impact of test assignment separately from test activation
  • Scroll-Depth and Triggering in Tests
    • Dynamic triggers to prevent flashing or loading delays
    • The need for both manual and automated enrollments
  • Single-Page Applications (SPA) and Testing
    • Increased complexity due to SPA architecture
    • Metrics that account for continuous scrolling and page load
  • Importance of Quality in Experimentation
    • Over-reliance on win rates without considering the quality of tests
    • Methods to measure test quality, including post-test actions
  • Role of Interleaving in Learning
    • Alternating between learning tasks to improve overall learning speed
  • The Role of AI in Mitigating Solution Bias
    • Use of AI to generate unbiased ideas for experiments
    • Challenges in overcoming solution bias in experimentation
  • Pressures in Agency CRO Work
    • The risk of inflating win rates due to client pressures
    • Agencies adopting conservative testing strategies to guarantee results
  • Multi-Page Testing and Variations
    • Managing multi-page tests with discrete variations on multiple pages

Quotes of the Week

Standout quips from this week’s roundtable

  • “The more you know, the easier it is to know more.” – Iqbal Ali
  • “Chasing revenue leads to perverse incentivization.” – Matt Beischel
  • “Win rate without a guardrail is not a good idea.” – Iqbal Ali

CRO Link Digest

Links to interesting, amusing, thought-provoking, or downright silly content shared at this week’s roundtable

Off-Topic Sidebars

Experimentation isn’t the only thing we talk about at the CRO Roundtable. There’s often a healthy dose of discussion on shared interests, personal passions, and hobbies. This week we had some sidebars about:

  • Attending Comic Cons
    • Meeting Mike Carey, author of “The Girl with All the Gifts”
  • Can QR codes expire?
  • Biking trips in Quebec during the summer
  • Music and Sequencing as a hobby

Off-Topic Shareables