A neural anisotropic view of underspecification in deep learning

The underspecification of most machine learning pipelines means that we cannot rely solely on validation performance to assess the robustness of deep learning systems to naturally occurring distribution shifts. Instead, making sure that a neural network can generalize across a large number of different situations requires to understand the specific way in which it solves a task. In this work, we propose to study this problem from a geometric perspective with the aim to understand two key characteristics of neural network solutions in underspecified settings: how is the geometry of the learned function related to the data representation? And, are deep networks always biased towards simpler solutions, as conjectured in recent literature? We show that the way neural networks handle the underspecification of these problems is highly dependent on the data representation, affecting both the geometry and the complexity of the learned predictors. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.


page 1

page 2

page 3

page 4

page 5

page 6

page 7

page 9


Geometrization of deep networks for the interpretability of deep learning systems

How to understand deep learning systems remains an open problem. In this...

How deep learning works --The geometry of deep learning

Why and how that deep learning works well on different tasks remains a m...

Deep network as memory space: complexity, generalization, disentangled representation and interpretability

By bridging deep networks and physics, the programme of geometrization o...

Understanding over-parameterized deep networks by geometrization

A complete understanding of the widely used over-parameterized deep netw...

Robustness and invariance properties of image classifiers

Deep neural networks have achieved impressive results in many image clas...

Optimism in the Face of Adversity: Understanding and Improving Deep Learning through Adversarial Robustness

Driven by massive amounts of data and important advances in computationa...

Robust Generalization of Quadratic Neural Networks via Function Identification

A key challenge facing deep learning is that neural networks are often n...

Please sign up or login with your details

Forgot password? Click here to reset