Learning Connectivity of Neural Networks from a Topological Perspective

by   Kun Yuan, et al.

Seeking effective neural networks is a critical and practical field in deep learning. Besides designing the depth, type of convolution, normalization, and nonlinearities, the topological connectivity of neural networks is also important. Previous principles of rule-based modular design simplify the difficulty of building an effective architecture, but constrain the possible topologies in limited spaces. In this paper, we attempt to optimize the connectivity in neural networks. We propose a topological perspective to represent a network into a complete graph for analysis, where nodes carry out aggregation and transformation of features, and edges determine the flow of information. By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner. We further attach auxiliary sparsity constraint to the distribution of connectedness, which promotes the learned topology focus on critical connections. This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks. Quantitative results of experiments reflect the learned connectivity is superior to traditional rule-based ones, such as random, residual, and complete. In addition, it obtains significant improvements in image classification and object detection without introducing excessive computation burden.


Component Edge Connectivity of Hypercube-like Networks

As a generalization of the traditional connectivity, the g-component edg...

Topological Insights in Sparse Neural Networks

Sparse neural networks are effective approaches to reduce the resource r...

Dynamic Graph: Learning Instance-aware Connectivity for Neural Networks

One practice of employing deep neural networks is to apply the same arch...

SparseMask: Differentiable Connectivity Learning for Dense Image Prediction

In this paper, we aim at automatically searching an efficient network ar...

Topology and geometry of data manifold in deep learning

Despite significant advances in the field of deep learning in applicatio...

Discovering Neural Wirings

The success of neural networks has driven a shift in focus from feature ...

Emergent Criticality Through Adaptive Information Processing in Boolean Networks

We study information processing in populations of Boolean networks with ...