Core Concepts
The author proposes a novel NAS algorithm that efficiently profiles the Pareto front across multiple devices with a single search run, addressing hardware constraints and diverse objectives. By leveraging hypernetworks and multiple gradient descent, the method outperforms existing MOO NAS methods.
Abstract
The content discusses a novel approach to multi-objective neural architecture search (NAS) that balances performance and hardware metrics efficiently. The proposed method, MODNAS, showcases scalability and effectiveness across various search spaces, datasets, and hardware devices. By integrating user preferences for trade-offs between objectives, MODNAS optimizes architectures without additional search costs.
The paper introduces the concept of Pareto front profiling in multi-objective optimization (MOO) using a novel NAS algorithm called MODNAS. This algorithm efficiently balances performance and hardware metrics across multiple devices in just one search run. By parameterizing architectural distribution via a hypernetwork conditioned on hardware features and preference vectors, MODNAS enables zero-shot transferability to new devices.
MODNAS is compared against baselines like Random Search (RS), Random MetaHypernetwork (RHPN), and other state-of-the-art methods like LEMONADE and MetaD2A + HELP. Results demonstrate that MODNAS outperforms these baselines in terms of hypervolume across different devices and objectives.
The experiments conducted on various search spaces such as NAS-Bench-201, HAT space for Transformers, and OFA space for ImageNet-1k showcase the scalability and efficiency of MODNAS in profiling the Pareto front. The method proves to be effective in optimizing architectures across diverse objectives while considering hardware constraints.
Stats
Extensive experiments with up to 19 hardware devices
Up to 3 objectives showcased effectiveness
Zero-shot transferability to new devices
Outperforms existing MOO NAS methods
Quotes
"Our contributions can be summarized as follows: We present a principled and robust approach for Multi-objective Differentiable NAS."
"This work is the first to provide a global view of the Pareto solutions with just a single model."
"Extensive evaluation of our method across 3 different search spaces show both improved efficiency."