Training Stronger Baselines for Learning to Optimize

by   Tianlong Chen, et al.

Learning to optimize (L2O) has gained increasing attention since classical optimizers require laborious problem-specific design and hyperparameter tuning. However, there is a gap between the practical demand and the achievable performance of existing L2O models. Specifically, those learned optimizers are applicable to only a limited class of problems, and often exhibit instability. With many efforts devoted to designing more sophisticated L2O models, we argue for another orthogonal, under-explored theme: the training techniques for those L2O models. We show that even the simplest L2O model could have been trained much better. We first present a progressive training scheme to gradually increase the optimizer unroll length, to mitigate a well-known L2O dilemma of truncation bias (shorter unrolling) versus gradient explosion (longer unrolling). We further leverage off-policy imitation learning to guide the L2O learning, by taking reference to the behavior of analytical optimizers. Our improved training techniques are plugged into a variety of state-of-the-art L2O models, and immediately boost their performance, without making any change to their model structures. Especially, by our proposed techniques, an earliest and simplest L2O model can be trained to outperform the latest complicated L2O models on a number of tasks. Our results demonstrate a greater potential of L2O yet to be unleashed, and urge to rethink the recent progress. Our codes are publicly available at:



page 1

page 2

page 3

page 4


Optimizer Amalgamation

Selecting an appropriate optimizer for a given problem is of major inter...

Scaling Imitation Learning in Minecraft

Imitation learning is a powerful family of techniques for learning senso...

Training Adaptive Computation for Open-Domain Question Answering with Computational Constraints

Adaptive Computation (AC) has been shown to be effective in improving th...

Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation

Current language generation models suffer from issues such as repetition...

Knowledge Infused Decoding

Pre-trained language models (LMs) have been shown to memorize a substant...

Automated Progressive Learning for Efficient Training of Vision Transformers

Recent advances in vision Transformers (ViTs) have come with a voracious...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.