Efficient Multi-Objective Hardware-Aware Neural Architecture Search for Edge Devices

Core Concepts

A multi-objective method for hardware-aware neural architecture search that efficiently identifies a set of high-performing architectures with diverse hardware costs in a single run.

Abstract

The paper proposes a multi-objective hardware-aware neural architecture search (MO-HDNAS) method to address the challenges in existing approaches. The key contributions are: Generalization of the single-objective hardware-aware NAS framework to a multi-objective one to address the increased search cost when multiple hardware cost constraints are present. Introduction of a hardware cost diversity objective to encourage the exploration of architectures with diverse hardware costs, allowing the discovery of high-performing architectures with higher hardware cost requirements. Demonstration of the robustness of the proposed method on six different edge devices for image classification tasks. The method employs a multi-objective evolutionary algorithm (NSGA-II) to optimize three objectives: (1) maximizing the representation similarity metric to a reference model, (2) minimizing hardware cost, and (3) maximizing hardware cost diversity. The hardware cost diversity objective facilitates a better exploration of the architecture search space, leading to the identification of a set of high-performing architectures with diverse hardware costs in a single run. Experimental results on the NAS-Bench-201 benchmark show that the proposed MO-HDNAS method outperforms the single-objective approach (HW-EvRSNAS) in terms of both the quality of the discovered architectures and the search cost. MO-HDNAS is able to find a more diverse set of high-performing architectures, ranging from low to high hardware costs, while reducing the search cost by 32 times compared to the single-objective method.

Stats

The best architecture found within the search space has a test accuracy of 94.37% on CIFAR-10, 73.49% on CIFAR-100, and 46.77% on ImageNet16-120. The search cost of the proposed MO-HDNAS method is 0.65 GPU hours, while the single-objective HW-EvRSNAS method requires 20.87 GPU hours.

Quotes

"To alleviate this problem, a recent direction in the literature has employed representation similarity metric for efficiently evaluating architecture performance." "Nonetheless, since it is inherently a single objective method, it requires multiple runs to identify the optimal architecture set satisfying the diverse hardware cost constraints, thereby increasing the search cost." "Furthermore, simply converting the single objective into a multi-objective approach results in an under-explored architectural search space."

Key Insights Distilled From

Multi-Objective Hardware Aware Neural Architecture Search using Hardware Cost Diversity

by Nilotpal Sin... at arxiv.org 04-22-2024

https://arxiv.org/pdf/2404.12403.pdf

Deeper Inquiries

How can the proposed MO-HDNAS method be extended to handle more complex hardware constraints, such as energy consumption or memory usage, in addition to latency

The MO-HDNAS method can be extended to handle more complex hardware constraints by incorporating additional objectives that specifically target energy consumption and memory usage. To include energy consumption, a new objective can be introduced to minimize the energy required for inference or training of the neural architectures. This objective can be formulated to penalize architectures that consume excessive energy during computation. Similarly, for memory usage, an objective can be added to optimize the memory footprint of the architectures, ensuring efficient utilization of available memory resources. By integrating these objectives into the MO-HDNAS framework, the search algorithm can simultaneously consider multiple hardware constraints, including latency, energy consumption, and memory usage, to identify architectures that meet a broader range of hardware requirements.

What are the potential limitations of the hardware cost diversity objective, and how can it be further improved to ensure a more comprehensive exploration of the architecture search space

The hardware cost diversity objective in MO-HDNAS may have limitations in certain scenarios, such as when the diversity metric does not adequately capture the variations in hardware costs across architectures. To address this, the hardware cost diversity objective can be further improved by incorporating a weighted diversity measure that assigns different weights to architectures based on their hardware costs. This weighted approach can ensure that architectures with significantly different hardware costs contribute more to the diversity metric, leading to a more comprehensive exploration of the architecture search space. Additionally, introducing a regularization term that encourages exploration of underrepresented regions of the hardware cost spectrum can help mitigate biases towards architectures with similar hardware costs, enhancing the overall diversity of the discovered architectures.

Given the diverse set of architectures discovered by MO-HDNAS, how can the final architecture selection be optimized for specific deployment scenarios, such as edge devices with varying hardware capabilities

To optimize the final architecture selection for specific deployment scenarios, such as edge devices with varying hardware capabilities, a post-processing step can be introduced after the MO-HDNAS search process. This step involves evaluating the discovered architectures on a validation set representative of the target deployment environment. Performance metrics, including accuracy, latency, energy consumption, and memory usage, can be measured for each architecture on the validation set. Based on these metrics and the specific hardware constraints of the deployment scenario, a selection criterion can be defined to choose the most suitable architecture. This criterion can prioritize architectures that strike a balance between performance and hardware constraints, ensuring optimal deployment on edge devices with diverse hardware capabilities. Additionally, techniques like transfer learning or fine-tuning can be applied to further optimize the selected architecture for the specific deployment scenario, adapting it to the unique characteristics of the target edge device.