Optimization-Derived Learning with Essential Convergence Analysis of Training and Hyper-training

06/16/2022
by   Risheng Liu, et al.
0

Recently, Optimization-Derived Learning (ODL) has attracted attention from learning and vision areas, which designs learning models from the perspective of optimization. However, previous ODL approaches regard the training and hyper-training procedures as two separated stages, meaning that the hyper-training variables have to be fixed during the training process, and thus it is also impossible to simultaneously obtain the convergence of training and hyper-training variables. In this work, we design a Generalized Krasnoselskii-Mann (GKM) scheme based on fixed-point iterations as our fundamental ODL module, which unifies existing ODL methods as special cases. Under the GKM scheme, a Bilevel Meta Optimization (BMO) algorithmic framework is constructed to solve the optimal training and hyper-training variables together. We rigorously prove the essential joint convergence of the fixed-point iteration for training and the process of optimizing hyper-parameters for hyper-training, both on the approximation quality, and on the stationary analysis. Experiments demonstrate the efficiency of BMO with competitive performance on sparse coding and real-world applications such as image deconvolution and rain streak removal.

READ FULL TEXT

page 8

page 9

research
02/26/2022

Fixed Point Iterations for SURE-based PSF Estimation for Image Deconvolution

Stein's unbiased risk estimator (SURE) has been shown to be an effective...
research
01/27/2021

Convergence Analysis of Fixed Point Chance Constrained Optimal Power Flow Problems

For optimal power flow problems with chance constraints, a particularly ...
research
08/05/2021

Proof of convergence of LoRaWAN model

In this document, we prove the convergence of the model proposed in [1],...
research
11/24/2019

Stage-based Hyper-parameter Optimization for Deep Learning

As deep learning techniques advance more than ever, hyper-parameter opti...
research
11/20/2017

Hyper Converged Infrastructures: Beyond virtualization

Hyper Convergence has brought virtualization and IT strategies to a new ...
research
06/22/2020

Hippo: Taming Hyper-parameter Optimization of Deep Learning with Stage Trees

Hyper-parameter optimization is crucial for pushing the accuracy of a de...
research
06/08/2015

ARock: an Algorithmic Framework for Asynchronous Parallel Coordinate Updates

Finding a fixed point to a nonexpansive operator, i.e., x^*=Tx^*, abstra...

Please sign up or login with your details

Forgot password? Click here to reset