Many experiments ("A/B tests") in online platforms exhibit interference, where an intervention applied to one market participant influences the behavior of another participant. This interference can lead to biased estimates of the treatment effect of the intervention.
In this talk we first focus on such experiments in two-sided platforms where "customers" book "listings". We develop a stochastic market model and associated mean field limit to capture dynamics in such experiments, and use our model to investigate how the performance of different designs and estimators is affected by marketplace interference effects. Platforms typically use two common experimental designs: demand-side ("customer") randomization (CR) and supply-side ("listing") randomization (LR), along with their associated estimators. We show that good experimental design depends on market balance: in highly demand-constrained markets, CR is unbiased, while LR is biased; conversely, in highly supply-constrained markets, LR is unbiased, while CR is biased. We also introduce and study a novel experimental design based on two-sided randomization (TSR) where both customers and listings are randomized to treatment and control. We show that appropriate choices of TSR designs can be unbiased in both extremes of market balance, while yielding relatively low bias in intermediate regimes of market balance. (This is based on joint work with Hannah Li, Inessa Liskovich, and Gabriel Weintraub.)
Time permitting, we will conclude the talk with some discussion of other experimental designs used by online platforms to address interference, including adaptive designs such as switchback experiments, and clustered experimental designs. The goal is to provide an overview of some of the open challenges that arise in this domain. (This part of the talk is based in part on joint work with Peter Glynn and Mohammad Rasouli.)
The ISL Colloquium meets weekly during the academic year. Seminars are each Thursday at 4:30pm PT, unless indicated otherwise.
Until further notice, the ISL Colloquium convenes exclusively via Zoom (on Thursdays at 4:30pm PT) due to the ongoing pandemic. To avoid "Zoom-bombing", we ask attendees to input their email address here https://stanford.zoom.us/meeting/register/tJckfuCurzkvEtKKOBvDCrPv3McapgP6HygJ to receive the Zoom meeting details via email.