DeepAI AI Chat
Log In Sign Up

A Kernel Perspective of Skip Connections in Convolutional Networks

by   Daniel Barzilai, et al.

Over-parameterized residual networks (ResNets) are amongst the most successful convolutional neural architectures for image processing. Here we study their properties through their Gaussian Process and Neural Tangent kernels. We derive explicit formulas for these kernels, analyze their spectra, and provide bounds on their implied condition numbers. Our results indicate that (1) with ReLU activation, the eigenvalues of these residual kernels decay polynomially at a similar rate compared to the same kernels when skip connections are not used, thus maintaining a similar frequency bias; (2) however, residual kernels are more locally biased. Our analysis further shows that the matrices obtained by these residual kernels yield favorable condition numbers at finite depths than those obtained without the skip connections, enabling therefore faster convergence of training with gradient descent.


On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels

We study the properties of various over-parametrized convolutional neura...

Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks

Deep residual network architectures have been shown to achieve superior ...

Residual Tangent Kernels

A recent body of work has focused on the theoretical study of neural net...

New Insights into Graph Convolutional Networks using Neural Tangent Kernels

Graph Convolutional Networks (GCNs) have emerged as powerful tools for l...

Visualizing Residual Networks

Residual networks are the current state of the art on ImageNet. Similar ...

Training Quantized Network with Auxiliary Gradient Module

In this paper, we seek to tackle two challenges in training low-precisio...

Universal flow approximation with deep residual networks

Residual networks (ResNets) are a deep learning architecture with the re...