On the Importance of Attention in Meta-Learning for Few-Shot Text Classification

06/03/2018
by   Xiang Jiang, et al.
0

Current deep learning based text classification methods are limited by their ability to achieve fast learning and generalization when the data is scarce. We address this problem by integrating a meta-learning procedure that uses the knowledge learned across many tasks as an inductive bias towards better natural language understanding. Based on the Model-Agnostic Meta-Learning framework (MAML), we introduce the Attentive Task-Agnostic Meta-Learning (ATAML) algorithm for text classification. The essential difference between MAML and ATAML is in the separation of task-agnostic representation learning and task-specific attentive adaptation. The proposed ATAML is designed to encourage task-agnostic representation learning by way of task-agnostic parameterization and facilitate task-specific adaptation via attention mechanisms. We provide evidence to show that the attention mechanism in ATAML has a synergistic effect on learning performance. In comparisons with models trained from random initialization, pretrained models and meta trained MAML, our proposed ATAML method generalizes better on single-label and multi-label classification tasks in miniRCV1 and miniReuters-21578 datasets.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/24/2020

When does MAML Work the Best? An Empirical Study on Model-Agnostic Meta-Learning in NLP Applications

Model-Agnostic Meta-Learning (MAML), a model-agnostic meta-learning meth...
research
01/27/2021

Learning task-agnostic representation via toddler-inspired learning

One of the inherent limitations of current AI systems, stemming from the...
research
11/03/2020

Meta-Learning for Natural Language Understanding under Continual Learning Framework

Neural network has been recognized with its accomplishments on tackling ...
research
10/28/2019

HIDRA: Head Initialization across Dynamic targets for Robust Architectures

The performance of gradient-based optimization strategies depends heavil...
research
11/27/2020

Connecting Context-specific Adaptation in Humans to Meta-learning

Cognitive control, the ability of a system to adapt to the demands of a ...
research
03/18/2020

Few-Shot Graph Classification with Model Agnostic Meta-Learning

Graph classification aims to perform accurate information extraction and...
research
03/22/2022

Improving Meta-learning for Low-resource Text Classification and Generation via Memory Imitation

Building models of natural language processing (NLP) is challenging in l...

Please sign up or login with your details

Forgot password? Click here to reset