DeepAI
Log In Sign Up

2nd-order Updates with 1st-order Complexity

05/24/2021
by   Michael F. Zimmer, et al.
0

It has long been a goal to efficiently compute and use second order information on a function (f) to assist in numerical approximations. Here it is shown how, using only basic physics and a numerical approximation, such information can be accurately obtained at a cost of O(N) complexity, where N is the dimensionality of the parameter space of f. In this paper, an algorithm (VA-Flow) is developed to exploit this second order information, and pseudocode is presented. It is applied to two classes of problems, that of inverse kinematics (IK) and gradient descent (GD). In the IK application, the algorithm is fast and robust, and is shown to lead to smooth behavior even near singularities. For GD the algorithm also works very well, provided the cost function is locally well-described by a polynomial.

READ FULL TEXT

page 1

page 2

page 3

page 4

01/28/2022

Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization

Second-order optimizers are thought to hold the potential to speed up ne...
07/22/2021

Structured second-order methods via natural gradient descent

In this paper, we propose new structured second-order methods and struct...
02/03/2021

Numerical Differentiation using local Chebyshev-Approximation

In applied mathematics, especially in optimization, functions are often ...
09/08/2021

An efficient jet marcher for computing the quasipotential for 2D SDEs

We present a new algorithm, the efficient jet marching method (EJM), for...
06/08/2018

A Stein variational Newton method

Stein variational gradient descent (SVGD) was recently proposed as a gen...
05/10/2020

A Robust Matching Pursuit Algorithm Using Information Theoretic Learning

Current orthogonal matching pursuit (OMP) algorithms calculate the corre...
06/03/2017

Optimal Envelope Approximation in Fourier Basis with Applications in TV White Space

Lowpass envelope approximation of smooth continuous-variable signals are...