Neural Architecture Search
lastmod 2026-04-10created 2026-04-101 edits
Introduction
Deep neural networks have shown that they can successfully perform many tasks that could not be solved with the techniques of the pre-deep-learning era. However, the architectures of the models that currently achieve state-of-the-art performance in many problem domains, such as image recognition, are mostly the result of hand design by experts in the field. As deep learning researchers know well, deciding which architecture to use for a given task and dataset is truly tiresome work. For example, when building a convolutional network, the model's implementer has to decide things such as how many hidden layers to use, how many filters per layer, how to set the filter size, padding, and stride, and when to pool; the problem is that you don't know whether these design decisions are good or bad until you train the model. On top of that, it would be nice if there were a single optimal network architecture, but the optimal architecture differs depending on the task and dataset, so a model that does well on one dataset is not guaranteed to perform well on another with the same architecture. And (reasonably deep) artificial neural networks are notorious for slow training even among machine learning models. In the end, designing the optimal network architecture is hard work that wastes a ton of time. Neural Architecture Search (NAS) is the field that studies methodologies for automating this hard work, to search conveniently and quickly for the network architecture best suited to a given task.