HAPI: A Large-scale Longitudinal Dataset of Commercial ML API Predictions
Nov 28, 2022
Speakers
Lingjiao Chen
Speaker · 0 followers
Zhihua Jin
Speaker · 0 followers
Sabri Eyuboglu
Speaker · 0 followers
Matei Zaharia
Speaker · 0 followers
Christopher Ré
Speaker · 5 followers
About
Commercial ML APIs offered by providers such as Google, Amazon and Microsoft have dramatically simplified ML adoptions in many applications. Numerous companies and academics pay to use ML APIs for tasks such as object detection, OCR and sentiment analysis. Different ML APIs tackling the same task can have very heterogeneous performances. Moreover, the ML models underlying the APIs also evolve over time. As ML APIs rapidly become a valuable marketplace and an integral part of analytics, it is critical to systematically study and compare different APIs with each other and to characterize how individual APIs change over time. However, this practically important topic is currently underexplored due to the lack of data. In this paper, we present HAPI (History of APIs), a longitudinal dataset of 1,761,417 instances of commercial ML API applications (involving APIs from Amazon, Google, IBM, Microsoft and other providers) across diverse tasks including image tagging, speech recognition, and text mining from 2020 to 2022. Each instance consists of a query input for an API (e.g., an image or text) along with the API’s output prediction/annotation and confidence scores. HAPI is the first large-scale dataset of ML API usages and is a unique resource for studying ML as-a-service (MLaaS). As examples of the types of analyses that HAPI enables, we show that ML APIs’ performance changes substantially over time—several APIs’ accuracies dropped on specific benchmark datasets. Even when the API’s aggregate performance stays steady, its error modes can shift across different subtypes of data between 2020 and 2022. Such changes can substantially impact the entire analytics pipelines that use some ML API as a component. We further use HAPI to study commercial APIs’ performance disparities across demographic subgroups over time. HAPI can stimulate more research in the growing field of MLaaS.Commercial ML APIs offered by providers such as Google, Amazon and Microsoft have dramatically simplified ML adoptions in many applications. Numerous companies and academics pay to use ML APIs for tasks such as object detection, OCR and sentiment analysis. Different ML APIs tackling the same task can have very heterogeneous performances. Moreover, the ML models underlying the APIs also evolve over time. As ML APIs rapidly become a valuable marketplace and an integral part of analytics, it is cri…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Gradient Methods Provably Converge to Non-Robust Networks
Watch later
Gal Vardi, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The World is not Uniformly Distributed: Challenges and opportunities for Deep RL
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement
Watch later
Erik Wijmans, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Re-Analyze Gauss: Bounds for Private Matrix Approximation via Dyson Brownian Motion
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Learning to Reconstruct Missing Data from Spatiotemporal Graphs with Sparse Observations
Watch later
Ivan Marisca, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Generative Collage and its Sticky Questions on Human-AI Co-Creativity
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%