Do ideas have shape? Plato's theory of forms as the continuous limit of artificial neural networks
share
We show that ResNets converge, in the infinite depth limit, to a generalization of computational anatomy/image registration algorithms. In this generalization (idea registration), images are replaced by abstractions (ideas) living in high dimensional RKHS spaces, and material points are replaced by data points. Whereas computational anatomy compares images by creating alignments via deformations of their coordinate systems (the material space), idea registration compares ideas by creating alignments via transformations of their (abstract RKHS) feature spaces. This identification of ResNets as idea registration algorithms has several remarkable consequences. The search for good architectures can be reduced to that of good kernels, and we show that the composition of idea registration blocks (idea formation) with reduced equivariant multi-channel kernels (introduced here) recovers and generalizes CNNs to arbitrary spaces and groups of transformations. Minimizers of L_2 regularized ResNets satisfy a discrete least action principle implying the near preservation of the norm of weights and biases across layers. The parameters of trained ResNets can be identified as solutions of an autonomous Hamiltonian system defined by the activation function and the architecture of the ANN. Momenta variables provide a sparse representation of the parameters of a ResNet. Minimizers of the L_2 regularized ResNets and ANNs (1) exist (2) are unique up to the value of the initial momentum, and (3) converge to minimizers of continuous idea formation variational problems. The registration regularization strategy provides a principled alternative to Dropout for ANNs. Pointwise RKHS error estimates lead to deterministic error estimates for ANNs, and the identification of ResNets as MAP estimators of deep residual Gaussian processes (introduced here) provides probabilistic error estimates.
READ FULL TEXT