In this episode, Matt Beischel and Iqbal Ali discuss the process for determining the right metrics to track when running tests and performing research. The topic originated from a LinkedIn post by Nils Koppelman.
Key points discussed include:
- Importance of Metrics: Metrics should be in a balanced “Goldilocks zone”—neither too few (leading to directionless testing) nor too many (causing decision paralysis).
- Common Problems: Problems arise from setting too many or too few metrics, picking the wrong metrics, and not setting them at the right time.
- Classification of Metrics:
- Primary Metrics: Crucial for decision-making in a specific experiment.
- Guardrail Metrics: Ensure that primary metrics do not adversely impact other important business metrics.
- Secondary Metrics: Investigative and often specific to the experiment.
- Timing of Metric Setting: Opinions vary on when to set metrics—whether at the conceptualization of an idea, before starting the experiment, or as a mix of both. Both hosts agree that metrics should ideally be set after designing the test but before it goes live, to avoid skewing data.
- Challenges and Best Practices:
- Metrics should be relevant and specific to the changes being tested.
- Avoid adding new metrics mid-test as it can disrupt the experiment’s integrity.
- Ensure metrics are reflective and comparable across control and variant groups.