Sluice networks: Learning what to share between loosely related tasks

05/23/2017
by   Sebastian Ruder, et al.
0

Multi-task learning is partly motivated by the observation that humans bring to bear what they know about related problems when solving new ones. Similarly, deep neural networks can profit from related tasks by sharing parameters with other networks. However, humans do not consciously decide to transfer knowledge between tasks (and are typically not aware of the transfer). In machine learning, it is hard to estimate if sharing will lead to improvements; especially if tasks are only loosely related. To overcome this, we introduce Sluice Networks, a general framework for multi-task learning where trainable parameters control the amount of sharing -- including which parts of the models to share. Our framework goes beyond and generalizes over previous proposals in enabling hard or soft sharing of all combinations of subspaces, layers, and skip connections. We perform experiments on three task pairs from natural language processing, and across seven different domains, using data from OntoNotes 5.0, and achieve up to 15 approaches to multi-task learning. We analyze when the architecture is particularly helpful, as well as its ability to fit noise. We show that a) label entropy is predictive of gains in sluice networks, confirming findings for hard parameter sharing, and b) while sluice networks easily fit noise, they are robust across domains in practice.

READ FULL TEXT
research
07/23/2021

Rethinking Hard-Parameter Sharing in Multi-Task Learning

Hard parameter sharing in multi-task learning (MTL) allows tasks to shar...
research
10/28/2017

Multi-Task Learning by Deep Collaboration and Application in Facial Landmark Detection

Convolutional neural networks (CNN) have become the most successful and ...
research
09/01/2020

Boosting share routing for multi-task learning

Multi-task learning (MTL) aims to make full use of the knowledge contain...
research
05/31/2019

Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains

As deep learning applications continue to become more diverse, an intere...
research
08/29/2018

What can we learn from Semantic Tagging?

We investigate the effects of multi-task learning using the recently int...
research
02/27/2017

Identifying beneficial task relations for multi-task learning in deep neural networks

Multi-task learning (MTL) in deep neural networks for NLP has recently r...
research
01/22/2021

Network Clustering for Multi-task Learning

The Multi-Task Learning (MTL) technique has been widely studied by word-...

Please sign up or login with your details

Forgot password? Click here to reset