The content discusses a novel approach to incorporating graph structural information into Transformer architectures for graph representation learning. The key contributions are:
The authors propose a spectrum-aware attention (SAA) mechanism that factorizes the attention matrix into fixed spectral similarities and learned frequency importances. This allows the attention mechanism to capture important graph structural information from the Laplacian spectrum, without the need for explicit positional encodings.
The SAA mechanism is shown to be able to approximate shortest path distances between nodes, as well as capture higher-order neighborhood information, providing strong graph inductive biases.
The proposed Eigenformer architecture, which uses the SAA mechanism, is empirically evaluated on several standard GNN benchmarks and is found to perform competitively with or better than state-of-the-art Graph Transformer models that use various positional encoding schemes.
The simpler attention mechanism in Eigenformer allows training of wider and deeper models for a given parameter budget, compared to other Graph Transformer architectures.
Visualizations of the learned attention weights and the distribution of node similarities at different frequencies provide insights into how the SAA mechanism captures the graph structure.
Overall, the work presents a novel and effective approach to incorporating structural inductive biases into Transformer-based models for graph representation learning, without relying on explicit positional encodings.
In eine andere Sprache
aus dem Quellinhalt
arxiv.org
Tiefere Fragen