Evaluating representations by the complexity of learning low-loss predictors

09/15/2020
by   William F. Whitney, et al.
7

We consider the problem of evaluating representations of data for use in solving a downstream task. We propose to measure the quality of a representation by the complexity of learning a predictor on top of the representation that achieves low loss on a task of interest, and introduce two methods, surplus description length (SDL) and ε sample complexity (εSC). In contrast to prior methods, which measure the amount of information about the optimal predictor that is present in a specific amount of data, our methods measure the amount of information needed from the data to recover an approximation of the optimal predictor up to a specified tolerance. We present a framework to compare these methods based on plotting the validation loss versus training set size (the "loss-data" curve). Existing measures, such as mutual information and minimum description length probes, correspond to slices and integrals along the data-axis of the loss-data curve, while ours correspond to slices and integrals along the loss-axis. We provide experiments on real data to compare the behavior of each of these methods over datasets of varying size along with a high performance open source library for representation evaluation at https://github.com/willwhitney/reprieve.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
11/28/2022

On the Sample Complexity of Representation Learning in Multi-task Bandits with Global and Local structure

We investigate the sample complexity of learning the optimal arm for mul...
research
09/20/2020

Latent Representation Prediction Networks

Deeply-learned planning methods are often based on learning representati...
research
08/03/2020

Low-loss connection of weight vectors: distribution-based approaches

Recent research shows that sublevel sets of the loss surfaces of overpar...
research
12/01/2022

Mutual Information-based Generalized Category Discovery

We introduce an information-maximization approach for the Generalized Ca...
research
03/27/2020

Information-Theoretic Probing with Minimum Description Length

To measure how well pretrained representations encode some linguistic pr...
research
02/19/2023

Evaluating Representations with Readout Model Switching

Although much of the success of Deep Learning builds on learning good re...
research
07/10/2021

Prediction of concept lengths for fast concept learning in description logics

Concept learning approaches based on refinement operators explore partia...

Please sign up or login with your details

Forgot password? Click here to reset