핵심 개념
The Graph Spectral Token is a novel approach to directly encode graph spectral information into the transformer architecture, capturing the global structure of the graph and enhancing the expressive power of graph transformers.
초록
The report proposes the Graph Spectral Token, a method to incorporate graph spectral information into the design of graph transformers. The key ideas are:
Graph spectral information is processed with an auxiliary network and assigned to the [CLS] token, while ordinary node features are processed through conventional tokens with node degree and optional Laplacian eigenvectors information.
The improved graph transformers, SubFormer-Spec and GraphTrans-Spec, are extensively benchmarked on multiple molecular modeling datasets. The results demonstrate that incorporating spectral information can significantly boost the performance, especially on large graph datasets where the eigen spectrum becomes a powerful discriminative feature.
On the ZINC dataset, SubFormer-Spec achieves comparable performance to state-of-the-art methods. On long-range graph benchmarks like Peptides-Struct and Peptides-Func, SubFormer-Spec outperforms the original SubFormer model.
Across various MoleculeNet datasets, SubFormer-Spec consistently outperforms the original SubFormer, showcasing the effectiveness of the spectral token.
On the OPDA dataset, which involves long-range charge-transfer phenomena, SubFormer-Spec significantly outperforms other baselines, highlighting the ability of graph transformers with spectral information to capture long-range interactions in large graphs.
The report suggests that the Graph Spectral Token is a promising approach to efficiently inject global structural information into graph transformers, leading to improved performance on a wide range of graph learning tasks.
통계
The ZINC dataset has an average of 23.2 nodes and 24.9 edges per graph.
The Peptides-Struct and Peptides-Func datasets have an average of 150.9 nodes and 307.3 edges per graph.
The OPDA dataset has an average of 55.8 nodes and 63.9 edges per graph.
인용구
"Incorporating graph inductive bias into transformer architectures remains a significant challenge."
"By parameterizing the auxiliary [CLS] token and leaving other tokens representing graph nodes, our method seamlessly integrates spectral information into the learning process."
"The improved GraphTrans, dubbed GraphTrans-Spec, achieves over 10% improvements on large graph benchmark datasets while maintaining efficiency comparable to MP-GNNs."