Kernekoncepter
Neural graph features (GRAF) provide fast and interpretable performance prediction that outperforms zero-cost proxies and other common encodings. The combination of GRAF and zero-cost proxies achieves the best performance at a fraction of the cost.
Resumé
The paper introduces neural graph features (GRAF) as a simple-to-compute set of properties of architectural graphs that can be used for efficient performance prediction in neural architecture search (NAS).
The authors first examine the limitations of existing zero-cost proxies, showing that many of them directly depend on the number of convolutions in the network rather than capturing more complex structural properties. Inspired by this, the authors propose GRAF, which includes features like operation counts, path lengths, and node degrees.
When used as input to a random forest predictor, GRAF outperforms zero-cost proxies and other common encodings like one-hot representations, especially on smaller training sets. The combination of GRAF and zero-cost proxies achieves the best overall performance, outperforming most existing predictors at a fraction of the computational cost.
The interpretability of GRAF also allows the authors to analyze which network properties are important for different tasks. For example, skip connections and convolution path lengths are crucial for image classification tasks, while node degree features are more important for other domains like autoencoding.
The authors further evaluate GRAF on a variety of tasks beyond just validation accuracy prediction, including hardware metrics and robustness. GRAF demonstrates strong performance across these diverse settings as well. Finally, they show that GRAF can also improve the performance of more complex predictors like BRP-NAS when used as additional input features.
Statistik
The number of convolutions in an architecture is a key driver of the performance of many zero-cost proxies.
GRAF features like skip connection path lengths and node degrees are important predictors of network performance across different tasks.
Combining GRAF and zero-cost proxies outperforms most existing performance predictors at a fraction of the computational cost.
Citater
"Inspired by the drawbacks of zero-cost proxies, we propose neural graph features (GRAF), simple to compute properties of architectural graphs."
"GRAF offers fast and interpretable performance prediction while outperforming zero-cost proxies and other common encodings."
"Using GRAF's interpretability, we demonstrate that different tasks favor diverse network properties."