Quantifying the Knowledge in GNNs for Reliable Distillation into MLPs
Jul 24, 2023
Sprecher:innen
Haitao Lin
Řečník · 0 sledujících
Yufei Huang
Řečník · 0 sledujících
Stan Z. Li
Řečník · 0 sledujících
Über
To bridge the gaps between topology-aware Graph Neural Networks (GNNs) and inference-efficient Multi-Layer Perceptron (MLPs), GLNN proposes to distill knowledge from a well-trained teacher GNN into a student MLP. Despite their great progress, comparatively little work has been done to explore the reliability of different knowledge points (nodes) in GNNs, especially their roles played during distillation. In this paper, we first quantify the knowledge reliability in GNN by measuring the invariance of their information entropy to noise perturbations, from which we observe that different knowledge points (1) show different distillation speeds (temporally); (2) are differentially distributed in the graph (spatially). To achieve reliable distillation, we propose an effective approach, namely Knowledge-inspired Reliable Distillation (KRD), that models the probability of each node being an informative and reliable knowledge point, based on which we sample a set of additional reliable knowledge points as supervision for training student MLPs. Extensive experiments show that KRD improves over the vanilla MLPs by 12.62To bridge the gaps between topology-aware Graph Neural Networks (GNNs) and inference-efficient Multi-Layer Perceptron (MLPs), GLNN proposes to distill knowledge from a well-trained teacher GNN into a student MLP. Despite their great progress, comparatively little work has been done to explore the reliability of different knowledge points (nodes) in GNNs, especially their roles played during distillation. In this paper, we first quantify the knowledge reliability in GNN by measuring the invarianc…
Gefällt euch das Format? Vertraut auf SlidesLive, um euer nächstes Event festzuhalten!
Professionelle Aufzeichnung und Livestreaming – weltweit.
Freigeben
Empfohlene Videos
Präsentationen, deren Thema, Kategorie oder Sprecher:in ähnlich sind
On the Global Convergence of Risk-Averse Policy Gradient Methods with Expected Conditional Risk Measures
Zhlédnout později
Xian Yu, …
Panel Discussion: “Challenges and lessons learned in deploying Generative AI”
Zhlédnout později
Deep Ganguli, …
Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learning
Zhlédnout později
Yulai Zhao, …
Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods
Zhlédnout později
