Exphormer: Sparse Transformers for Graphs
Jul 24, 2023
Speakers
Hamed Shirzad
Speaker · 0 followers
Ameya Velingker
Speaker · 1 follower
Balaji Venkatachalam
Speaker · 0 followers
Danica Sutherland
Speaker · 0 followers
About
Graph transformers have emerged as a promising architecture for a variety of graph learning and representation tasks. Despite their successes, though, it remains challenging to scale graph transformers to large graphs while maintaining accuracy competitive with message-passing networks. In this paper, we introduce Exphormer, a framework for building powerful and scalable graph transformers. Exphormer consists of a sparse attention mechanism based on two mechanisms: virtual global nodes and expander graphs, whose mathematical characteristics, such as spectral expansion, pseduorandomness, and sparsity, yield graph transformers with complexity only linear in the size of the graph, while allowing us to prove desirable theoretical properties of the resulting transformer models. We show that incorporating Exphormer into the recently-proposed GraphGPS framework produces models with competitive empirical results on a wide variety of graph datasets, including state-of-the-art results on three datasets. We also show that Exphormer can scale to datasets on larger graphs than shown in previous graph transformer architectures.Graph transformers have emerged as a promising architecture for a variety of graph learning and representation tasks. Despite their successes, though, it remains challenging to scale graph transformers to large graphs while maintaining accuracy competitive with message-passing networks. In this paper, we introduce Exphormer, a framework for building powerful and scalable graph transformers. Exphormer consists of a sparse attention mechanism based on two mechanisms: virtual global nodes and expan…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
An Adaptive Entropy-Regularization Framework for Multi-Agent Reinforcement Learning
Watch later
Woojun Kim, …
Provable Dynamic Fusion for Low-Quality Multimodal Data
Watch later
Can Forward Gradient Match Backpropagation?
Watch later
Toward Large Kernel Models
Watch later
Localized Learning: Past, Present and Future
Watch later
Jack Kendall, …
