AdaSwarm: A Novel PSO optimization Method for the Mathematical Equivalence of Error Gradients

05/19/2020
by   Rohan Mohapatra, et al.
0

This paper tackles the age-old question of derivative free optimization in neural networks. This paper introduces AdaSwarm, a novel derivative-free optimizer to have similar or better performance to Adam but without "gradients". To support the AdaSwarm, a novel Particle Swarm Optimization Exponentially weighted Momentum PSO (EM-PSO), a derivative-free optimizer, is also proposed which tackles constrained and unconstrained single objective optimization problems and looks at applying the proposed momentum particle swarm optimization on benchmark test functions, engineering optimization problems and habitability scores for exoplanets which show speed and convergence of the technique. The EM-PSO is extended by approximating the gradient of a function at any point using the parameters of the particle swarm optimization. This is a novel technique to simulate gradient descent, an extremely popular method in the back-propagation algorithm, using the approximated gradients from the particle swarm optimization parameters. Mathematical proofs of gradient approximation by EM-PSO, thereby bypassing the gradient computation, are presented. The AdaSwarm is compared with various optimizers and the theory and algorithmic performance are supported by promising results.

READ FULL TEXT
research
04/10/2021

A Swarm Variant for the Schrödinger Solver

This paper introduces application of the Exponentially Averaged Momentum...
research
04/10/2021

Fairly Constricted Particle Swarm Optimization

We have adapted the use of exponentially averaged momentum in PSO to mul...
research
02/24/2014

A hybrid swarm-based algorithm for single-objective optimization problems involving high-cost analyses

In many technical fields, single-objective optimization procedures in co...
research
05/08/2022

Hamiltonian Monte Carlo Particle Swarm Optimizer

We introduce the Hamiltonian Monte Carlo Particle Swarm Optimizer (HMC-P...
research
04/03/2018

A Bi-population Particle Swarm Optimizer for Learning Automata based Slow Intelligent System

Particle Swarm Optimization (PSO) is an Evolutionary Algorithm (EA) that...
research
04/28/2020

Design of multifunctional metamaterials using optimization

This paper explores the use of optimization to design multifunctional me...
research
07/23/2014

Quadratically constrained quadratic programming for classification using particle swarms and applications

Particle swarm optimization is used in several combinatorial optimizatio...

Please sign up or login with your details

Forgot password? Click here to reset