As discussed in our standup today, here's a proposal for what the goal for Experimentation can be for this sprint. Please refer to the main RFC for details & context, https://github.com/PostHog/product-internal/blob/main/requests-for-comments/2021-10-22-collaboration.md

Goal

Moonshot: A/B testing suite MVP.

Problems to tackle

These are partial problems from the original document, scoped for a 2-week MVP.

• P1. Defining experiments with a clear goal. Answering what are the results of the experiment.
• P2. Experiments should run quickly. Answering how long should the experiment run for, given parameters.
• P3. Experiments must be statistically significant. Answering whether the experiment produced statistically significant results.

Scope

• Basic experiment planning
  • Users can create an experiment with a target metric (only simple trends metric or conversion rates for now) and assign participants, understand the timescale required (P1).
  • We can predict the approximate user base size & time required to reach conclusive and statistically significant results (P2).
• Measure experiment results
  • Users can automatically track in experiment dashboard a single target number for experiment and control group (as well as deltas between the two) (P1).
  • Normalize experiment results (i.e. if control/experiment groups have different # of users and we're measuring a volume metric, we account for the size of each group) (P1).
• Statistical significance
  • Users can see a binary validation that experiment is statistically significant based on a heuristic (P3).

Additional considerations

• Support only A/B testing for now (i.e. no multivariate).
• Problems P1-P2 can be treated as roofshot goals & P3, moonshot.
• Full roadmap for what this milestone could look like after this coming up.