Learning Unsupervised Learning Rules

03/31/2018
by   Luke Metz, et al.
0

A major goal of unsupervised learning is to discover data representations that are useful for subsequent tasks, without access to supervised labels during training. Typically, this goal is approached by minimizing a surrogate objective, such as the negative log likelihood of a generative model, with the hope that representations useful for subsequent tasks will arise as a side effect. In this work, we propose instead to directly target a later desired task by meta-learning an unsupervised learning rule, which leads to representations useful for that task. Here, our desired task (meta-objective) is the performance of the representation on semi-supervised classification, and we meta-learn an algorithm -- an unsupervised weight update rule -- that produces representations that perform well under this meta-objective. Additionally, we constrain our unsupervised update rule to a be a biologically-motivated, neuron-local function, which enables it to generalize to novel neural network architectures. We show that the meta-learned update rule produces useful features and sometimes outperforms existing unsupervised learning techniques. We show that the meta-learned unsupervised update rule generalizes to train networks with different widths, depths, and nonlinearities. It also generalizes to train on data with randomly permuted input dimensions and even generalizes from image datasets to a text task.

READ FULL TEXT

page 7

page 8

page 12

page 13

research
03/17/2021

Augmenting Supervised Learning by Meta-learning Unsupervised Local Rules

The brain performs unsupervised learning and (perhaps) simultaneous supe...
research
04/12/2020

Pre-training Text Representations as Meta Learning

Pre-training text representations has recently been shown to significant...
research
10/04/2018

Unsupervised Learning via Meta-Learning

A central goal of unsupervised learning is to acquire representations fr...
research
03/06/2020

Finding online neural update rules by learning to remember

We investigate learning of the online local update rules for neural acti...
research
01/08/2023

Unsupervised Learning for Combinatorial Optimization Needs Meta-Learning

A general framework of unsupervised learning for combinatorial optimizat...
research
06/06/2022

Stacked unsupervised learning with a network architecture found by supervised meta-learning

Stacked unsupervised learning (SUL) seems more biologically plausible th...
research
02/21/2017

Online Representation Learning with Multi-layer Hebbian Networks for Image Classification Tasks

Unsupervised learning permits the development of algorithms that are abl...

Please sign up or login with your details

Forgot password? Click here to reset