Parallel Experimentation in a Competitive Advertising Marketplace

When multiple firms are simultaneously running experiments on a platform, the treatment effects for one firm may depend on the experimentation policies of others. This paper presents a set of causal estimands that are relevant to such an environment. We also present an experimental design that is suitable for facilitating experimentation across multiple competitors in such an environment. Together, these can be used by a platform to run experiments “as a service,” on behalf of its participating firms. We show that the causal estimands we develop are identified nonparametrically by the variation induced by the design, and present two scalable estimators that help measure them in typical high-dimensional situations. We implement the design on the advertising platform of JD.com, an eCommerce company, which is also a publisher of digital ads in China. We discuss how the design is engineered within the platform’s auction-driven ad-allocation system, which is typical of modern, digital advertising marketplaces. Finally, we present results from a parallel experiment involving 16 advertisers and millions of JD.com users. These results showcase the importance of accommodating a role for interactions across experimenters and demonstrates the viability of the framework.