Set Interdependence Transformer: Set-to-Sequence Neural Networks for Permutation Learning and Structure Prediction

06/08/2022
by   Mateusz Jurewicz, et al.
0

The task of learning to map an input set onto a permuted sequence of its elements is challenging for neural networks. Set-to-sequence problems occur in natural language processing, computer vision and structure prediction, where interactions between elements of large sets define the optimal output. Models must exhibit relational reasoning, handle varying cardinalities and manage combinatorial complexity. Previous attention-based methods require n layers of their set transformations to explicitly represent n-th order relations. Our aim is to enhance their ability to efficiently model higher-order interactions through an additional interdependence component. We propose a novel neural set encoding method called the Set Interdependence Transformer, capable of relating the set's permutation invariant representation to its elements within sets of any cardinality. We combine it with a permutation learning module into a complete, 3-part set-to-sequence model and demonstrate its state-of-the-art performance on a number of tasks. These range from combinatorial optimization problems, through permutation learning challenges on both synthetic and established NLP datasets for sentence ordering, to a novel domain of product catalog structure prediction. Additionally, the network's ability to generalize to unseen sequence lengths is investigated and a comparative empirical analysis of the existing methods' ability to learn higher-order interactions is provided.

READ FULL TEXT
research
10/01/2018

Set Transformer

Many machine learning tasks such as multiple instance learning, 3D shape...
research
06/01/2022

Higher-Order Attention Networks

This paper introduces higher-order attention networks (HOANs), a novel c...
research
09/11/2019

An Iterative Approach for Multiple Instance Learning Problems

Multiple Instance learning (MIL) algorithms are tasked with learning how...
research
06/26/2020

Conditional Set Generation with Transformers

A set is an unordered collection of unique elements–and yet many machine...
research
05/27/2019

OrderNet: Ordering by Example

In this paper we introduce a new neural architecture for sorting unorder...
research
01/30/2020

Learn to Predict Sets Using Feed-Forward Neural Networks

This paper addresses the task of set prediction using deep feed-forward ...
research
04/08/2020

The general theory of permutation equivarant neural networks and higher order graph variational encoders

Previous work on symmetric group equivariant neural networks generally o...

Please sign up or login with your details

Forgot password? Click here to reset