Learning to Optimize with Dynamic Mode Decomposition

11/29/2022
by   Petr Šimánek, et al.
0

Designing faster optimization algorithms is of ever-growing interest. In recent years, learning to learn methods that learn how to optimize demonstrated very encouraging results. Current approaches usually do not effectively include the dynamics of the optimization process during training. They either omit it entirely or only implicitly assume the dynamics of an isolated parameter. In this paper, we show how to utilize the dynamic mode decomposition method for extracting informative features about optimization dynamics. By employing those features, we show that our learned optimizer generalizes much better to unseen optimization problems in short. The improved generalization is illustrated on multiple tasks where training the optimizer on one neural network generalizes to different architectures and distinct datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/03/2023

Learning to Optimize for Reinforcement Learning

In recent years, by leveraging more data, computation, and diverse tasks...
research
09/22/2022

A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases

Learned optimizers – neural networks that are trained to act as optimize...
research
03/14/2017

Learned Optimizers that Scale and Generalize

Learning to learn has emerged as an important direction for achieving ar...
research
03/01/2017

Learning to Optimize Neural Nets

Learning to Optimize is a recently proposed framework for learning optim...
research
11/09/2019

Learning to Optimize in Swarms

Learning to optimize has emerged as a powerful framework for various opt...
research
10/08/2019

Dynamic Mode Decomposition based feature for Image Classification

Irrespective of the fact that Machine learning has produced groundbreaki...
research
11/22/2018

HyperAdam: A Learnable Task-Adaptive Adam for Network Training

Deep neural networks are traditionally trained using human-designed stoc...

Please sign up or login with your details

Forgot password? Click here to reset