On the emergence of tetrahedral symmetry in the final and penultimate layers of neural network classifiers
share
A recent numerical study observed that neural network classifiers enjoy a large degree of symmetry in the penultimate layer. Namely, if h(x) = Af(x) +b where A is a linear map and f is the output of the penultimate layer of the network (after activation), then all data points x_i, 1, …, x_i, N_i in a class C_i are mapped to a single point y_i by f and the points y_i are located at the vertices of a regular k-1-dimensional tetrahedron in a high-dimensional Euclidean space. We explain this observation analytically in toy models for highly expressive deep neural networks. In complementary examples, we demonstrate rigorously that even the final output of the classifier h is not uniform over data samples from a class C_i if h is a shallow network (or if the deeper layers do not bring the data samples into a convenient geometric configuration).
READ FULL TEXT