Read the original document by opening this link in a new tab.
Table of Contents
1 Introduction
2 Related Work
3 Approach
Summary
Automatic neural architecture design has shown its potential in discovering powerful neural network architectures. Existing methods conduct architecture search in a discrete space, which is highly inefficient. In this paper, we propose a simple and efficient method for automatic neural architecture design based on continuous optimization, named Neural Architecture Optimization (NAO). The core of NAO consists of an encoder, a predictor, and a decoder. The encoder maps neural network architectures into a continuous space, the predictor estimates accuracy based on the continuous representation, and the decoder maps the representation back to the architecture. Experiments show competitive results for image classification and language modeling tasks. NAO achieves improved efficiency in discovering architectures with limited computational resources. The approach involves training the encoder, predictor, and decoder jointly, optimizing structure reconstruction and performance prediction losses. In the inference process, better architectures are obtained by iteratively optimizing the continuous representation and decoding new architectures.