Abstract

At Spotify, we have hundreds of experiments, so-called A/B tests, running at any given point in time. A/B testing is a form of online controlled experimentation that builds on the foundational ideas of randomized controlled trials (RCTs). What makes A/B tests different from RCTs is the environment in which they are used and the requirements they face. A/B tests are often cheap, they can easily involve hundreds of millions of units, they can often have effects that are heterogeneous across units, and they can be risky but only in terms of degrading the user experience. In this talk, I will give a high-level overview of how we do online experimentation at Spotify and then focus on one challenge we are currently occupied with: sequential inference when outcomes are measured over open-ended windows of time.