Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time
Nov 28, 2022
Sprecher:innen
Huaxiu Yao
Sprecher:in · 0 Follower:innen
Caroline Choi
Sprecher:in · 0 Follower:innen
Yoonho Lee
Sprecher:in · 0 Follower:innen
Pang Wei Koh
Sprecher:in · 0 Follower:innen
Chelsea Finn
Sprecher:in · 57 Follower:innen
Über
Distribution shifts occur when the test distribution differs from the training distribution, and can considerably degrade performance of machine learning models deployed in the real world. While recent works have studied robustness to distribution shifts, distribution shifts arising from the passage of time have the additional structure of timestamp metadata. Real-world examples of such shifts are underexplored, and it is unclear whether existing models can leverage trends in past distribution shifts to reliably extrapolate into the future. To address this gap, we curate Wild-Time, a benchmark of 7 datasets that reflect temporal distribution shifts arising in a variety of real-world applications, including drug discovery, patient prognosis, and news classification. On these datasets, we systematically benchmark 13 approaches with various inductive biases. We evaluate methods in domain-generalization, continual learning, self-supervised learning, and ensemble learning, which leverage timestamps to extract the common structure of the distribution shifts. We extend several domain-generalization methods to the temporal distribution shift setting by treating windows of time as different domains. Finally, we propose two evaluation strategies to evaluate model performance under temporal distribution shifts—evaluation with a fixed time split (Eval-Fix) and evaluation with a data stream (Eval-Stream). Eval-Fix, our primary evaluation strategy, aims to provide a simple evaluation protocol for the broader machine learning community, while Eval-Stream serves as a complementary benchmark for continual learning approaches. Our experiments demonstrate that existing methods are limited in tackling temporal distribution shift: across all settings, we observe an average performance drop of 20Distribution shifts occur when the test distribution differs from the training distribution, and can considerably degrade performance of machine learning models deployed in the real world. While recent works have studied robustness to distribution shifts, distribution shifts arising from the passage of time have the additional structure of timestamp metadata. Real-world examples of such shifts are underexplored, and it is unclear whether existing models can leverage trends in past distribution s…
Organisator
NeurIPS 2022
Konto · 963 Follower:innen
Gefällt euch das Format? Vertraut auf SlidesLive, um euer nächstes Event festzuhalten!
Professionelle Aufzeichnung und Livestreaming – weltweit.
Freigeben
Empfohlene Videos
Präsentationen, deren Thema, Kategorie oder Sprecher:in ähnlich sind
Calibrated Perception Uncertainty Across Objects and Regions in Bird's-Eye-View
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Riemannian Score-Based Generative Modeling
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Causality for Temporal Unfairness Evaluation and Mitigation
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
LBD: Decouple Relevance and Observation for Individual-Level Unbiased Learning to Rank
Später ansehen
Ewigspeicher-Fortschrittswert: 0 = 0.0%
Contextual Transformer for Offline Meta Offline Reinforcement Learning
Später ansehen
Runji Lin, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%
AntiFaceGAN: Animatable 3D-Aware Face Image Generation for Realistic Video Avatars
Später ansehen
Yue Wu, …
Ewigspeicher-Fortschrittswert: 0 = 0.0%