Alibaba Open-Sources ‘KNAS’: An Automated Machine Learning Algorithm That Evaluates Given Architectures Without Training 

Architecture evaluation is a systematic approach for identifying flaws and dangers in architectural designs. The evaluation process is ideally performed before they are implemented.

Typically, neural architecture search (NAS) systems are used for architectural evaluation. Neural architecture search (NAS) is an AutoML branch that aims to find the best deep-learning model architecture for a task. The systems achieve this by finding an architecture that will achieve the best performance metric on the given task dataset and search space of possible architectures. However, this usually necessitates training each proposed model completely on the dataset, which takes a long time.

Prior studies have focused on improving search algorithms so that fewer models need to be trained. However, a team of researchers at Alibaba Group and Peking University have conducted a study to investigate a green NAS solution that evaluates architectures without training. To this end, they propose an efficient automatic machine learning (AutoML) system called Kernel Neural Architecture Search (KNAS).

According to researchers, gradients can be utilized as a coarse-grained proxy of downstream training to evaluate randomly-initialized architectures. They undertook a theoretical analysis and discovered that a realistic gradient kernel (the mean of the Gram matrix (MGM) of gradients) has a substantial link with the accuracy of a model.

KNAS uses a gradient kernel as a proxy for model quality and consumes an order of magnitude less computing resources than standard techniques. The KNAS algorithm then calculates the MGM for each suggested model architecture, selects the best few, calculates model accuracy for those candidates, and chooses the model with the highest accuracy as the final result.

The team tested their proposed approach on the NAS-Bench-201 benchmark against various other NAS algorithms such as random search, reinforcement learning (RL) search, evolutionary search, hyper-parameter search, differentiable algorithms, and two additional “training-free” methods. The results show that KNAS achieves considerable speedups with competitive accuracies. Except for the two training-free algorithms, the team discovered that KNAS was faster than all other approaches.

KNAS also generalizes well and can look for better topologies for text classification problems. Its wide range of applications is enabled by its incredibly low search cost. NAS methods that require little to no model training have also been researched by a number of other research institutions.

Paper: http://proceedings.mlr.press/v139/xu21m/xu21m.pdf

Github: https://github.com/Jingjing-NLP/KNAS

Reference: https://www.infoq.com/news/2022/02/alibaba-automl-knas/

Suggested