Meta Dropout: Learning to Perturb Features for Generalization

05/30/2019
by   Hae Beom Lee, et al.
0

A machine learning model that generalizes well should obtain low errors on the unseen test examples. Test examples could be understood as perturbations of training examples, which means that if we know how to optimally perturb training examples to simulate test examples, we could achieve better generalization at test time. However, obtaining such perturbation is not possible in standard machine learning frameworks as the distribution of the test data is unknown. To tackle this challenge, we propose a meta-learning framework that learns to perturb the latent features of training examples for generalization. Specifically, we meta-learn a noise generator that will output the optimal noise distribution for latent features across all network layers to obtain low error on the test instances, in an input-dependent manner. Then, the learned noise generator will perturb the training examples of unseen tasks at the meta-test time. We show that our method, Meta-dropout, could be also understood as meta-learning of the variational inference framework for a specific graphical model, and describe its connection to existing regularizers. Finally, we validate Meta-dropout on multiple benchmark datasets for few-shot classification, whose results show that it not only significantly improves the generalization performance of meta-learners but also allows them to obtain fast converegence.

READ FULL TEXT

page 8

page 15

research
01/30/2023

Robust Meta Learning for Image based tasks

A machine learning model that generalizes well should obtain low errors ...
research
07/26/2018

Meta-learning autoencoders for few-shot prediction

Compared to humans, machine learning models generally require significan...
research
05/30/2019

Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Tasks

While tasks could come with varying number of instances in realistic set...
research
12/13/2017

Deep Prior

The recent literature on deep learning offers new tools to learn a rich ...
research
11/11/2019

Evaluating Combinatorial Generalization in Variational Autoencoders

We evaluate the ability of variational autoencoders to generalize to uns...
research
03/09/2021

More data or more parameters? Investigating the effect of data structure on generalization

One of the central features of deep learning is the generalization abili...
research
02/20/2023

Meta-World Conditional Neural Processes

We propose Meta-World Conditional Neural Processes (MW-CNP), a condition...

Please sign up or login with your details

Forgot password? Click here to reset