Learning Connectivity of Neural Networks from a Topological Perspective

08/19/2020
by   Kun Yuan, et al.
39

Seeking effective neural networks is a critical and practical field in deep learning. Besides designing the depth, type of convolution, normalization, and nonlinearities, the topological connectivity of neural networks is also important. Previous principles of rule-based modular design simplify the difficulty of building an effective architecture, but constrain the possible topologies in limited spaces. In this paper, we attempt to optimize the connectivity in neural networks. We propose a topological perspective to represent a network into a complete graph for analysis, where nodes carry out aggregation and transformation of features, and edges determine the flow of information. By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner. We further attach auxiliary sparsity constraint to the distribution of connectedness, which promotes the learned topology focus on critical connections. This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks. Quantitative results of experiments reflect the learned connectivity is superior to traditional rule-based ones, such as random, residual, and complete. In addition, it obtains significant improvements in image classification and object detection without introducing excessive computation burden.

READ FULL TEXT
05/11/2021

Component Edge Connectivity of Hypercube-like Networks

As a generalization of the traditional connectivity, the g-component edg...
06/24/2020

Topological Insights in Sparse Neural Networks

Sparse neural networks are effective approaches to reduce the resource r...
10/02/2020

Dynamic Graph: Learning Instance-aware Connectivity for Neural Networks

One practice of employing deep neural networks is to apply the same arch...
04/16/2019

SparseMask: Differentiable Connectivity Learning for Dense Image Prediction

In this paper, we aim at automatically searching an efficient network ar...
04/19/2022

Topology and geometry of data manifold in deep learning

Despite significant advances in the field of deep learning in applicatio...
06/03/2019

Discovering Neural Wirings

The success of neural networks has driven a shift in focus from feature ...
04/20/2011

Emergent Criticality Through Adaptive Information Processing in Boolean Networks

We study information processing in populations of Boolean networks with ...