1. How big is the effect?
Each row estimates the same thing from a different angle. Intervals are 95%.
2. Is the experiment trustworthy?
3. Who actually saw the ad?
4. Which users respond?
5. Can a model find the responders?
6. What is targeting worth? (scenario)
Value and cost are assumptions you set. The profit curve is computed from the held-out Qini curve of the uplift model.
7. How big must the next test be?
8. What would watching the test every day have cost?
The control group was split at random into two fake arms of 50,000 users, 5,000 times, so every rejection is false. Each rule looks at the data a set number of times.