Log In Sign Up

2nd-order Updates with 1st-order Complexity

by   Michael F. Zimmer, et al.

It has long been a goal to efficiently compute and use second order information on a function (f) to assist in numerical approximations. Here it is shown how, using only basic physics and a numerical approximation, such information can be accurately obtained at a cost of O(N) complexity, where N is the dimensionality of the parameter space of f. In this paper, an algorithm (VA-Flow) is developed to exploit this second order information, and pseudocode is presented. It is applied to two classes of problems, that of inverse kinematics (IK) and gradient descent (GD). In the IK application, the algorithm is fast and robust, and is shown to lead to smooth behavior even near singularities. For GD the algorithm also works very well, provided the cost function is locally well-described by a polynomial.


page 1

page 2

page 3

page 4


Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization

Second-order optimizers are thought to hold the potential to speed up ne...

Structured second-order methods via natural gradient descent

In this paper, we propose new structured second-order methods and struct...

Numerical Differentiation using local Chebyshev-Approximation

In applied mathematics, especially in optimization, functions are often ...

An efficient jet marcher for computing the quasipotential for 2D SDEs

We present a new algorithm, the efficient jet marching method (EJM), for...

A Stein variational Newton method

Stein variational gradient descent (SVGD) was recently proposed as a gen...

A Robust Matching Pursuit Algorithm Using Information Theoretic Learning

Current orthogonal matching pursuit (OMP) algorithms calculate the corre...

Optimal Envelope Approximation in Fourier Basis with Applications in TV White Space

Lowpass envelope approximation of smooth continuous-variable signals are...