Get the Best of Both Worlds: Improving Accuracy and Transferability by Grassmann Class Representation
share
We generalize the class vectors found in neural networks to linear subspaces (i.e. points in the Grassmann manifold) and show that the Grassmann Class Representation (GCR) enables the simultaneous improvement in accuracy and feature transferability. In GCR, each class is a subspace and the logit is defined as the norm of the projection of a feature onto the class subspace. We integrate Riemannian SGD into deep learning frameworks such that class subspaces in a Grassmannian are jointly optimized with the rest model parameters. Compared to the vector form, the representative capability of subspaces is more powerful. We show that on ImageNet-1K, the top-1 error of ResNet50-D, ResNeXt50, Swin-T and Deit3-S are reduced by 5.6 3.5 observed that the intra-class feature variability grows when the subspace dimension increases. Consequently, we found the quality of GCR features is better for downstream tasks. For ResNet50-D, the average linear transfer accuracy across 6 datasets improves from 77.98 strong baseline of vanilla softmax. For Swin-T, it improves from 81.5 and for Deit3, it improves from 73.8 we believe that more applications could benefit from the Grassmann class representation. Code is released at https://github.com/innerlee/GCR.
READ FULL TEXT