Efficient Neural Architecture Search with Differentiable Architecture Estimation on Hierarchical Search Spaces

Core Concepts

FaDE, a method that leverages differentiable architecture search to aggregate path decisions from a fixed hierarchical hyper-architecture into point estimates for an open-ended search, can guide an outer search in a pseudo-gradient descent manner.

Abstract

The paper presents FaDE, a method for efficient neural architecture search (NAS) on hierarchical search spaces. The key ideas are:

Constructing a chained hierarchical search space for neural architectures, where an architecture is composed of several structurally identical sub-modules (cells).

Using differentiable architecture search (DARTS) to train a hyper-architecture that contains multiple cell architectures per layer. This allows for fast evaluation of sub-modules as a surrogate for full architecture evaluation.

Deriving FaDE-ranks, which are relative performance predictions on finite regions of the hierarchical search space, by factorizing the trained architecture parameters along the corresponding paths of the hyper-architecture.

Guiding an outer NAS optimization using the FaDE-ranks in a pseudo-gradient descent manner, without the need for a proxy search space. The outer optimization discovers new cell architectures per layer in an iterative fashion.

The authors show that the FaDE-ranks correlate well with the actual performance of the corresponding architectures. They also demonstrate that the FaDE-guided search can improve the evaluation performance of the discovered architectures over iterations, though not significantly.

Stats

The search space consists of neural network cell architectures, where each cell is a directed acyclic graph (DAG) with up to 6 nodes.
The overall neural architecture has 4 chained cells.
The hyper-architecture used in the first experiment has 5 manually selected DAGs as parallel computation paths per cell, resulting in 625 possible architectures.
The feature space F used in the iterative FaDE-guided search is 3-dimensional, based on the normed eccentricity variance, degree variance, and number of vertices of the DAGs.

Quotes

"FaDE is especially suited on deep hierarchical, respectively multi-cell search spaces, which it can explore by linear instead of exponential cost and therefore eliminates the need for a proxy search space."
"The relative nature of these ranks calls for a memory-less, batch-wise outer search algorithm for which we use an evolutionary algorithm with pseudo-gradient descent."

Key Insights Distilled From

Efficient NAS with FaDE on Hierarchical Spaces

by Simon Neumey... at arxiv.org 04-26-2024

https://arxiv.org/pdf/2404.16218.pdf

Deeper Inquiries

How can the FaDE-guided search be further improved to yield more significant performance gains over iterations

To further enhance the performance gains of the FaDE-guided search over iterations, several strategies can be implemented:

Dynamic Hyper-Parameter Tuning: Introduce a mechanism to dynamically adjust hyper-parameters such as learning rates, regularization factors, and step sizes during the search process. This adaptive tuning can help optimize the search trajectory based on the current state of the search space.

Ensemble of Models: Utilize an ensemble approach where multiple models with different architectures are trained simultaneously. By combining the predictions of these models, a more robust and accurate estimation of architecture performance can be obtained, leading to better-informed decisions during the search.

Exploration-Exploitation Balance: Implement a strategy that balances exploration (searching for new architectures) and exploitation (focusing on promising architectures). This balance can prevent the search from getting stuck in local optima and ensure a more thorough exploration of the search space.

Transfer Learning: Incorporate transfer learning techniques to leverage knowledge gained from previous search iterations. By transferring insights and weights from well-performing architectures to new iterations, the search process can benefit from past learnings and accelerate the discovery of optimal architectures.

Diversification of Search Space: Introduce mechanisms to diversify the search space exploration, such as introducing randomness in architecture selection or prioritizing under-explored regions of the search space. This diversification can help discover novel architectures that may lead to significant performance improvements.

What other types of hierarchical search spaces could the FaDE method be generalized to, and how would the implementation differ

The FaDE method can be generalized to various types of hierarchical search spaces beyond the specific chained hierarchical architectures discussed in the context. Some examples of hierarchical search spaces that FaDE could be applied to include:

Nested Hierarchies: Implementing FaDE on nested hierarchies where architectures are organized in multiple levels of abstraction, allowing for more complex and diverse architecture structures.

Graph-based Hierarchies: Extending FaDE to hierarchical search spaces represented as graphs, where nodes represent architectural components and edges define relationships between them. This approach can capture intricate dependencies and interactions within architectures.

Recursive Hierarchies: Adapting FaDE to recursive hierarchies where architectures are recursively defined based on sub-architectures, enabling the exploration of hierarchical structures with varying levels of recursion.

The implementation of FaDE in these generalized hierarchical search spaces would involve modifying the architecture representation, training procedures, and rank estimation methods to accommodate the specific characteristics and complexities of each hierarchy type.

Can the FaDE-ranks be used to guide the search in a more global manner, beyond the local pseudo-gradient descent approach

FaDE-ranks can indeed be utilized to guide the search in a more global manner, extending beyond the local pseudo-gradient descent approach. Some strategies to leverage FaDE-ranks for global search guidance include:

Multi-Objective Optimization: Incorporating multiple objectives such as model accuracy, complexity, and resource efficiency into the FaDE-rank estimation. By optimizing across these objectives simultaneously, a more holistic view of architecture performance can be obtained, leading to better-informed global search decisions.

Population-Based Methods: Implementing population-based search algorithms that utilize FaDE-ranks to guide the selection, mutation, and crossover of architectures within a population. This approach can facilitate a more comprehensive exploration of the search space and enhance the diversity of architectures considered.

Reinforcement Learning: Integrating FaDE-ranks into a reinforcement learning framework where the ranks serve as rewards for guiding the learning agent to discover optimal architectures. By training the agent to make sequential decisions based on FaDE-ranks, the search process can be optimized towards global performance improvements.