DeepAI AI Chat
Log In Sign Up

AutoML-Zero: Evolving Machine Learning Algorithms From Scratch

by   Esteban Real, et al.

Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks—or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.


page 1

page 2

page 3

page 4


Evolving Real-Time Heuristics Search Algorithms with Building Blocks

The research area of real-time heuristics search has produced quite many...

Effective Building Block Design for Deep Convolutional Neural Networks using Search

Deep learning has shown promising results on many machine learning tasks...

Improving exploration in policy gradient search: Application to symbolic optimization

Many machine learning strategies designed to automate mathematical tasks...

Hyperbox based machine learning algorithms: A comprehensive survey

With the rapid development of digital information, the data volume gener...

An Introduction to Modern Statistical Learning

This work in progress aims to provide a unified introduction to statisti...

A natural approach to studying schema processing

The Building Block Hypothesis (BBH) states that adaptive systems combine...

Large-Scale Evolution of Image Classifiers

Neural networks have proven effective at solving difficult problems but ...

Code Repositories


Curated Great Tips, Mindsets and Practices for Coders 😎

view repo


Webservice for extracting text from pdfs on the web

view repo