Training Neural Networks to Produce Compatible Features

by   Michael Gygli, et al.

This paper makes a first step towards compatible and hence reusable network components. Rather than training networks for different tasks independently, we adapt the training process to produce network components that are compatible across tasks. We propose and compare several different approaches to accomplish compatibility. Our experiments on CIFAR-10 show that: (i) we can train networks to produce compatible features, without degrading task accuracy compared to training networks independently; (ii) the degree of compatibility is highly dependent on where we split the network into a feature extractor and a classification head; (iii) random initialization has a large effect on compatibility; (iv) we can train incrementally: given previously trained components, we can train new ones which are also compatible with them. This work is part of a larger goal to increase network reusability: we envision that compatibility will enable solving new tasks by mixing and matching suitable components.



There are no comments yet.


page 4


Towards Backward-Compatible Representation Learning

We propose a way to learn visual features that are compatible with previ...

An Automated Compatibility Prediction Engine using DISC Theory Based Classification and Neural Networks

Traditionally psychometric tests were used for profiling incoming worker...

Roommate Compatibility Detection Through Machine Learning Techniques

Our objective is to develop an artificially intelligent system which aim...

Compatibility of Partitions, Hierarchies, and Split Systems

The question whether a partition 𝒫 and a hierarchy ℋ or a tree-like spli...

CoReS: Compatible Representations via Stationarity

In this paper, we propose a novel method to learn internal feature repre...

Compatible Learning for Deep Photonic Neural Network

Realization of deep learning with coherent optical field has attracted r...

Neural Loop Combiner: Neural Network Models for Assessing the Compatibility of Loops

Music producers who use loops may have access to thousands in loop libra...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.