Tensor-based framework for training flexible neural networks

06/25/2021
by   Yassine Zniyed, et al.
0

Activation functions (AFs) are an important part of the design of neural networks (NNs), and their choice plays a predominant role in the performance of a NN. In this work, we are particularly interested in the estimation of flexible activation functions using tensor-based solutions, where the AFs are expressed as a weighted sum of predefined basis functions. To do so, we propose a new learning algorithm which solves a constrained coupled matrix-tensor factorization (CMTF) problem. This technique fuses the first and zeroth order information of the NN, where the first-order information is contained in a Jacobian tensor, following a constrained canonical polyadic decomposition (CPD). The proposed algorithm can handle different decomposition bases. The goal of this method is to compress large pretrained NN models, by replacing subnetworks, i.e., one or multiple layers of the original network, by a new flexible layer. The approach is applied to a pretrained convolutional neural network (CNN) used for character classification.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 17

08/23/2016

Neural Networks with Smooth Adaptive Activation Functions for Regression

In Neural Networks (NN), Adaptive Activation Functions (AAF) have parame...
02/24/2017

Activation Ensembles for Deep Neural Networks

Many activation functions have been proposed in the past, but selecting ...
06/20/2016

A New Training Method for Feedforward Neural Networks Based on Geometric Contraction Property of Activation Functions

We propose a new training method for a feedforward neural network having...
11/02/2020

Reducing Neural Network Parameter Initialization Into an SMT Problem

Training a neural network (NN) depends on multiple factors, including bu...
06/10/2020

Scalable Partial Explainability in Neural Networks via Flexible Activation Functions

Achieving transparency in black-box deep learning algorithms is still an...
05/21/2018

Faster Neural Network Training with Approximate Tensor Operations

We propose a novel technique for faster Neural Network (NN) training by ...
12/22/2020

Residual Matrix Product State for Machine Learning

Tensor network (TN), which originates from quantum physics, shows broad ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.