Matt and Iqbal explain how to select primary metrics for experiments and A/B tests. They explore the definition of a primary metric as the key metric used to evaluate the success or impact of changes being tested. They emphasize that while conversion rate is often considered a primary metric, it is not always the case. Focusing solely on conversion rate can be misleading. A primary metric should be relevant to the specific goal of the experiment and that multiple metrics, including secondary and guardrail metrics, should be considered in decision-making.
Experimentation should be viewed as a research tool for learning and validating hypotheses rather than merely increasing conversion rates or revenue. There should be a balance between user behavior metrics and business benefit metrics, highlighting that changes should ultimately benefit the business. They caution against performance-based payment models in CRO, as they can incentivize negative behaviors like gaming the system.
The episode concludes with a discussion on running experiments on low-traffic sites and the importance of aligning user needs with business goals.