On Linear Identifiability of Learned Representations

07/01/2020
by   Geoffrey Roeder, et al.
3

Identifiability is a desirable property of a statistical model: it implies that the true model parameters may be estimated to any desired precision, given sufficient computational resources and data. We study identifiability in the context of representation learning: discovering nonlinear data representations that are optimal with respect to some downstream task. When parameterized as deep neural networks, such representation functions typically lack identifiability in parameter space, because they are overparameterized by design. In this paper, building on recent advances in nonlinear ICA, we aim to rehabilitate identifiability by showing that a large family of discriminative models are in fact identifiable in function space, up to a linear indeterminacy. Many models for representation learning in a wide variety of domains have been identifiable in this sense, including text, images and audio, state-of-the-art at time of publication. We derive sufficient conditions for linear identifiability and provide empirical support for the result on both simulated and real-world data.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/08/2017

Global optimality conditions for deep neural networks

We study the error landscape of deep linear and nonlinear neural network...
research
03/23/2018

SEGEN: Sample-Ensemble Genetic Evolutional Network Model

Deep learning, a rebranding of deep neural network research works, has a...
research
07/12/2022

Learning Bellman Complete Representations for Offline Policy Evaluation

We study representation learning for Offline Reinforcement Learning (RL)...
research
02/14/2022

On Pitfalls of Identifiability in Unsupervised Learning. A Note on: "Desiderata for Representation Learning: A Causal Perspective"

Model identifiability is a desirable property in the context of unsuperv...
research
04/19/2023

Denoising Cosine Similarity: A Theory-Driven Approach for Efficient Representation Learning

Representation learning has been increasing its impact on the research a...
research
07/14/2020

Robust Identifiability in Linear Structural Equation Models of Causal Inference

In this work, we consider the problem of robust parameter estimation fro...
research
02/02/2022

AtmoDist: Self-supervised Representation Learning for Atmospheric Dynamics

Representation learning has proven to be a powerful methodology in a wid...

Please sign up or login with your details

Forgot password? Click here to reset