Hyper-X: A Unified Hypernetwork for Multi-Task Multilingual Transfer

05/24/2022
by   Ahmet Üstün, et al.
0

Massively multilingual models are promising for transfer learning across tasks and languages. However, existing methods are unable to fully leverage training data when it is available in different task-language combinations. To exploit such heterogeneous supervision we propose Hyper-X, a unified hypernetwork that generates weights for parameter-efficient adapter modules conditioned on both tasks and language embeddings. By learning to combine task and language-specific knowledge our model enables zero-shot transfer for unseen languages and task-language combinations. Our experiments on a diverse set of languages demonstrate that Hyper-X achieves the best gain when a mixture of multiple resources is available while performing on par with strong baselines in the standard scenario. Finally, Hyper-X consistently produces strong results in few-shot scenarios for new languages and tasks showing the effectiveness of our approach beyond zero-shot transfer.

READ FULL TEXT

page 1

page 3

research
05/12/2022

Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models

Massively Multilingual Transformer based Language Models have been obser...
research
05/22/2022

Multilingual Machine Translation with Hyper-Adapters

Multilingual machine translation suffers from negative interference acro...
research
06/30/2022

"Diversity and Uncertainty in Moderation" are the Key to Data Selection for Multilingual Few-shot Transfer

Few-shot transfer often shows substantial gain over zero-shot transfer <...
research
04/03/2022

On Efficiently Acquiring Annotations for Multilingual Models

When tasked with supporting multiple languages for a given problem, two ...
research
09/23/2020

Worst-Case-Aware Curriculum Learning for Zero and Few Shot Transfer

Multi-task transfer learning based on pre-trained language encoders achi...
research
09/29/2022

Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights

Learning representations of neural network weights given a model zoo is ...
research
08/06/2021

Towards Zero-shot Language Modeling

Can we construct a neural model that is inductively biased towards learn...

Please sign up or login with your details

Forgot password? Click here to reset