SSFN: Self Size-estimating Feed-forward Network and Low Complexity Design

05/17/2019
by   Saikat Chatterjee, et al.
0

We design a self size-estimating feed-forward network (SSFN) using a joint optimization approach for estimation of number of layers, number of nodes and learning of weight matrices at a low computational complexity. In the proposed approach, SSFN grows from a small-size network to a large-size network. The increase in size from small-size to large-size guarantees a monotonically decreasing cost with addition of nodes and layers. The optimization approach uses a sequence of layer-wise target-seeking non-convex optimization problems. Using `lossless flow property' of some activation functions, such as rectified linear unit (ReLU), we analytically find regularization parameters in the layer-wise non-convex optimization problems. Closed-form analytic expressions of regularization parameters allow to avoid tedious cross-validations. The layer-wise non-convex optimization problems are further relaxed to convex optimization problems for ease of implementation and analytical tractability. The convex relaxation helps to design a low-complexity algorithm for construction of the SSFN. We experiment with eight popular benchmark datasets for sound and image classification tasks. Using extensive experiments we show that the SSFN can self-estimate its size using the low-complexity algorithm. The size of SSFN varies significantly across the eight datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
10/23/2017

Progressive Learning for Systematic Design of Large Neural Networks

We develop an algorithm for systematic design of a large artificial neur...
research
10/15/2021

Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalization

Many supervised machine learning methods are naturally cast as optimizat...
research
10/06/2021

Use of Deterministic Transforms to Design Weight Matrices of a Neural Network

Self size-estimating feedforward network (SSFN) is a feedforward multila...
research
03/23/2018

Lifting Layers: Analysis and Applications

The great advances of learning-based approaches in image processing and ...
research
02/02/2022

Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositions

We develop fast algorithms and robust software for convex optimization o...
research
11/06/2021

AGGLIO: Global Optimization for Locally Convex Functions

This paper presents AGGLIO (Accelerated Graduated Generalized LInear-mod...
research
01/12/2020

Channel Assignment in Uplink Wireless Communication using Machine Learning Approach

This letter investigates a channel assignment problem in uplink wireless...

Please sign up or login with your details

Forgot password? Click here to reset