Cogradient Descent for Bilinear Optimization

06/16/2020
by   Li'an Zhuo, et al.
0

Conventional learning methods simplify the bilinear model by regarding two intrinsically coupled factors independently, which degrades the optimization procedure. One reason lies in the insufficient training due to the asynchronous gradient descent, which results in vanishing gradients for the coupled variables. In this paper, we introduce a Cogradient Descent algorithm (CoGD) to address the bilinear problem, based on a theoretical framework to coordinate the gradient of hidden variables via a projection function. We solve one variable by considering its coupling relationship with the other, leading to a synchronous gradient descent to facilitate the optimization procedure. Our algorithm is applied to solve problems with one variable under the sparsity constraint, which is widely used in the learning paradigm. We validate our CoGD considering an extensive set of applications including image reconstruction, inpainting, and network pruning. Experiments show that it improves the state-of-the-art by a significant margin.

READ FULL TEXT

page 5

page 6

research
06/20/2021

Cogradient Descent for Dependable Learning

Conventional gradient descent methods compute the gradients for multiple...
research
06/20/2020

Blind Descent: A Prequel to Gradient Descent

We describe an alternative to gradient descent for backpropogation throu...
research
05/14/2019

Task-Driven Data Verification via Gradient Descent

We introduce a novel algorithm for the detection of possible sample corr...
research
08/15/2019

Convergence Behaviour of Some Gradient-Based Methods on Bilinear Games

Min-max optimization has attracted much attention in the machine learnin...
research
03/14/2023

Low-Complexity Iterative Methods for Complex-Variable Matrix Optimization Problems in Frobenius Norm

Complex-variable matrix optimization problems (CMOPs) in Frobenius norm ...
research
05/27/2023

Faster Margin Maximization Rates for Generic Optimization Methods

First-order optimization methods tend to inherently favor certain soluti...
research
07/25/2013

A gradient descent technique coupled with a dynamic simulation to determine the near optimum orientation of floor plan designs

A prototype tool to assist architects during the early design stage of f...

Please sign up or login with your details

Forgot password? Click here to reset