Abstract

Abstract: At Spotify, we have hundreds of experiments, so-called A/B tests, running at any given point in time. A/B testing is a form of online controlled experimentation that builds on the foundational ideas of randomized controlled trials (RCTs). What makes A/B tests different from RCTs is the environment in which they are used and the requirements they face. A/B tests are often cheap, they can easily involve hundreds of millions of units, they can often have effects that are heterogeneous across units, and they can be risky but only in terms of degrading the user experience. In this talk, I will give a high-level overview of how we do online experimentation at Spotify and present two challenges we currently have: sequential inference when outcomes are measured over open-ended windows of time, and finding groups of users with heterogeneous treatment effects.