A/B Testing for Recommender Systems in a Two-sided Marketplace
Dez 6, 2021
Sprecher:innen
Preetam Nandy
Speaker · 0 followers
Divya Venugopalan
Speaker · 0 followers
Shaunak Chatterjee
Speaker · 0 followers
Über
Two-sided marketplaces are standard business models of many online platforms (e.g., Amazon, Facebook, LinkedIn), wherein the platforms have consumers, buyers or content viewers on one side and producers, sellers or content-creators on the other. Consumer side measurement of the impact of a treatment variant can be done via simple online A/B testing. Producer side measurement is more challenging because the producer experience depends on the treatment assignment of the consumers. Existing approaches for producer side measurement are either based on graph cluster-based randomization or on certain treatment propagation assumptions. The former approach results in low-powered experiments as the producer-consumer network density increases and the latter approach lacks a strict notion of error control. In this paper, we propose (i) a quantification of the quality of a producer side experiment, and (ii) a new experiment design mechanism that generates high quality experiments based on this quantification. Our approach, called UniCoRn (Unifying Counterfactual Rankings), provides explicit control over the quality of the experiment and its computation cost. Further, we prove that our experiment design is optimal. Our approach is agnostic to the density of the producer-consumer network and does not rely on any treatment propagation assumption. Moreover, unlike the existing approaches, we do not need to know the underlying network in advance, making this widely applicable to the industrial setting where the underlying network is unknown and challenging to predict a priori due to its dynamic nature. We use simulations to thoroughly validate our approach and compare it against existing methods. We also implement UniCoRn in an edge recommendation application that serves tens of millions of members and billions of edge recommendations daily.Two-sided marketplaces are standard business models of many online platforms (e.g., Amazon, Facebook, LinkedIn), wherein the platforms have consumers, buyers or content viewers on one side and producers, sellers or content-creators on the other. Consumer side measurement of the impact of a treatment variant can be done via simple online A/B testing. Producer side measurement is more challenging because the producer experience depends on the treatment assignment of the consumers. Existing approac…
Organisator
NeurIPS 2021
Account · 1.9k followers
Über NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Gefällt euch das Format? Vertraut auf SlidesLive, um euer nächstes Event festzuhalten!
Professionelle Aufzeichnung und Livestreaming – weltweit.
Freigeben
Empfohlene Videos
Präsentationen, deren Thema, Kategorie oder Sprecher:in ähnlich sind
SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning
Watch later
Jongjin Park, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Understanding and Knowledge Extraction from Mathematical and Scientific Text
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
TenSet: A Large-scale Program Performance Dataset for Learned Tensor Compilers
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Enhancing Robustness against Model Poisoning Attacks in Federated Learning from a Client Perspective
Watch later
Jingwei Sun, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
DLO@Scale: A Large-Scale Meta Dataset for Learning Non-Rigid Object Pushing Dynamics
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%