Neurons learn slower than they think

04/02/2021
by   Ilona Kulikovskikh, et al.
0

Recent studies revealed complex convergence dynamics in gradient-based methods, which has been little understood so far. Changing the step size to balance between high convergence rate and small generalization error may not be sufficient: maximizing the test accuracy usually requires a larger learning rate than minimizing the training loss. To explore the dynamic bounds of convergence rate, this study introduces differential capability into an optimization process, which measures whether the test accuracy increases as fast as a model approaches the decision boundary in a classification problem. The convergence analysis showed that: 1) a higher convergence rate leads to slower capability growth; 2) a lower convergence rate results in faster capability growth and decay; 3) regulating a convergence rate in either direction reduces differential capability.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/31/2023

On the Linear Convergence of Policy Gradient under Hadamard Parameterization

The convergence of deterministic policy gradient under the Hadamard para...
research
02/22/2021

Super-Convergence with an Unstable Learning Rate

Conventional wisdom dictates that learning rate should be in the stable ...
research
10/31/2021

Fast Global Convergence of Policy Optimization for Constrained MDPs

We address the issue of safety in reinforcement learning. We pose the pr...
research
06/06/2022

Essential convergence rate of ordinary differential equations appearing in optimization

Some continuous optimization methods can be connected to ordinary differ...
research
04/01/2017

Faster Subgradient Methods for Functions with Hölderian Growth

The purpose of this manuscript is to derive new convergence results for ...
research
04/18/2017

Accelerated Distributed Dual Averaging over Evolving Networks of Growing Connectivity

We consider the problem of accelerating distributed optimization in mult...
research
10/09/2020

Bioinspired Bipedal Locomotion Control for Humanoid Robotics Based on EACO

To construct a robot that can walk as efficiently and steadily as humans...

Please sign up or login with your details

Forgot password? Click here to reset