Title: Modern Statistical Challenges in A/B Tests and Recent Work in Metric Decomposition
Presenter: Nathaniel Stevens
Time: June 13, 2024, 17:00-18:00 CEST / 11:00 AM EST / 8:00 AM PST
Abstract: The rise of internet-based services and products has brought about an unprecedented opportunity for online businesses to engage in large scale data-driven decision making. Over the past two decades, tech organizations have invested tremendous resources in online controlled experiments (colloquially referred to as A/B tests) to assess the impact of innovation on their customers and businesses. Running these experiments at scale has presented a host of novel statistical challenges and hence research opportunities. In this talk we review some of these challenges and present new work on one problem in particular, arising from an academia-industry collaboration with Airbnb. In particular, we’ll look at a new direction for sensitivity improvement whereby a target metric of interest is decomposed into components with high signal-to-noise disparity. Through both frequentist and Bayesian theory as well as real world applications, we’ll demonstrate the agility metric decomposition yields relative to an un-decomposed analysis.
Bio:
Nathaniel Stevens is an Assistant Professor in the Department of Statistics and Actuarial Science at the University of Waterloo. Prior to this Nathaniel held a faculty position at the University of San Francisco in the Department of Mathematics and Statistics. He is and has been Program Director of both university’s undergraduate data science programs. Having overseen 30+ data science internships at 20+ companies, Nathaniel is interested in using statistics to solve practical problems, and he has a passion for inspiring and training students to do the same. His research interests lie at the intersection of data science and industrial statistics; his publications span topics including experimental design and A/B testing, social network modeling and monitoring, survival and reliability analysis, measurement system analysis, and the design and analysis of estimation-based alternatives to traditional hypothesis testing.
Collaborators:
· Luke Hagar (University of Waterloo)
· Alex Deng (Airbnb)
· Tatiana Xifara (Airbnb)
· Amit Gandhi (Airbnb)
.png)
