A/B Testing

This skill teaches experimental design fundamentals for product teams and growth marketers. It guides you through structured hypothesis development, sample size calculations, metric selection (primary, secondary, and guardrail metrics), and result interpretation with statistical rigor. The skill covers design choices: A/B variants, multivariate tests, and split-URL approaches. You'll learn how to avoid common pitfalls like peeking at results before reaching significance or changing test conditions mid-run.

Highlights

• Build testable hypotheses using structured frameworks
• Calculate required sample sizes for statistical validity
• Design variations that isolate single meaningful changes
• Select metrics tied to business impact
• Interpret results using confidence intervals and p-values
• Create a reusable experiment playbook

Beyond single tests, the skill helps establish a continuous experimentation program. It covers hypothesis generation from multiple sources (analytics, customer research, competitor analysis), prioritization using ICE scoring, and tracking experiment velocity. Teams building systematic experimentation programs compound learnings over time, turning isolated wins into proven patterns applicable across their product. The skill includes templates, checklists, and practical implementation guidance for both client-side and server-side testing solutions.