One Network Fits All? Modular versus Monolithic Task Formulations in Neural Networks

03/29/2021
by   Atish Agarwala, et al.
0

Can deep learning solve multiple tasks simultaneously, even when they are unrelated and very different? We investigate how the representations of the underlying tasks affect the ability of a single neural network to learn them jointly. We present theoretical and empirical findings that a single neural network is capable of simultaneously learning multiple tasks from a combined data set, for a variety of methods for representing tasks – for example, when the distinct tasks are encoded by well-separated clusters or decision trees over certain task-code attributes. More concretely, we present a novel analysis that shows that families of simple programming-like constructs for the codes encoding the tasks are learnable by two-layer neural networks with standard training. We study more generally how the complexity of learning such combined tasks grows with the complexity of the task codes; we find that combining many tasks may incur a sample complexity penalty, even though the individual tasks are easy to learn. We provide empirical support for the usefulness of the learning bounds by training networks on clusters, decision trees, and SQL-style aggregation.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

02/23/2017

Neural Decision Trees

In this paper we propose a synergistic melting of neural networks and de...
12/07/2017

End-to-end Learning of Deterministic Decision Trees

Conventional decision trees have a number of favorable properties, inclu...
01/04/2022

Time and space complexity of deterministic and nondeterministic decision trees

In this paper, we study arbitrary infinite binary information systems ea...
06/01/2011

Popular Ensemble Methods: An Empirical Study

An ensemble consists of a set of individually trained classifiers (such ...
09/02/2019

Bayesian Neural Tree Models for Nonparametric Regression

Frequentist and Bayesian methods differ in many aspects, but share some ...
06/02/2017

Hyperparameter Optimization: A Spectral Approach

We give a simple, fast algorithm for hyperparameter optimization inspire...
01/16/2020

Extracting more from boosted decision trees: A high energy physics case study

Particle identification is one of the core tasks in the data analysis pi...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.