Learnable Fourier Features for Multi-dimensional Spatial Positional Encoding
Dec 6, 2021
Speakers
Cho-Jui Hsieh
Speaker · 1 follower
Samy Bengio
Speaker · 1 follower
About
Attentional mechanisms are order-invariant. Positional encoding is a crucial component to allow attention-based deep model architectures such as Transformer to address sequences or images where the position of information matters. In this paper, we propose a novel positional encoding method based on learnable Fourier features. Instead of hard-coding each position as a token or a vector, we represent each position, which can be multi-dimensional, as a trainable encoding based on learnable Fourier feature mapping, modulated with a multi-layer perceptron. The representation is particularly advantageous for a spatial multi-dimensional position, e.g., pixel positions on an image, where L2 distances or more complex positional relationships need to be captured. Our experiments based on several public benchmark tasks show that our learnable Fourier feature representation for multi-dimensional positional encoding outperforms existing methods by both improving the accuracy and allowing faster convergence.Attentional mechanisms are order-invariant. Positional encoding is a crucial component to allow attention-based deep model architectures such as Transformer to address sequences or images where the position of information matters. In this paper, we propose a novel positional encoding method based on learnable Fourier features. Instead of hard-coding each position as a token or a vector, we represent each position, which can be multi-dimensional, as a trainable encoding based on learnable Fourier…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
On Blame Attribution for Accountable Multi-Agent Sequential Decision Making
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On Calibration and Out-of-domain Generalization
Watch later
Yoav Wald, …
Total of 1 viewers voted for saving the presentation to eternal vault which is 0.1%
Learning latent causal graphs via mixture oracles
Watch later
Bohdan Kivva, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Meaning in Context: Pragmatic Communication in Humans and Machines
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
From ImageNet to Image Classification
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
On Component Interactions in Two-Stage Recommender Systems
Watch later
Jiri Hron, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%