DeepAI AI Chat
Log In Sign Up

Learn to Predict Sets Using Feed-Forward Neural Networks

by   Hamid Rezatofighi, et al.

This paper addresses the task of set prediction using deep feed-forward neural networks. A set is a collection of elements which is invariant under permutation and the size of a set is not fixed in advance. Many real-world problems, such as image tagging and object detection, have outputs that are naturally expressed as sets of entities. This creates a challenge for traditional deep neural networks which naturally deal with structured outputs such as vectors, matrices or tensors. We present a novel approach for learning to predict sets with unknown permutation and cardinality using deep neural networks. In our formulation we define a likelihood for a set distribution represented by a) two discrete distributions defining the set cardinally and permutation variables, and b) a joint distribution over set elements with a fixed cardinality. Depending on the problem under consideration, we define different training models for set prediction using deep neural networks. We demonstrate the validity of our set formulations on relevant vision problems such as: 1)multi-label image classification where we achieve state-of-the-art performance on the PASCAL VOC and MS COCO datasets, 2) object detection, for which our formulation outperforms state-of-the-art detectors such as Faster R-CNN and YOLO v3, and 3) a complex CAPTCHA test, where we observe that, surprisingly, our set-based network acquired the ability of mimicking arithmetics without any rules being coded.


page 9

page 10

page 11

page 12

page 14


DeepSetNet: Predicting Sets with Deep Neural Networks

This paper addresses the task of set prediction using deep learning. Thi...

Joint Learning of Set Cardinality and State Distribution

We present a novel approach for learning to predict sets using deep lear...

Interpretable Set Functions

We propose learning flexible but interpretable functions that aggregate ...

Learning Representations of Sets through Optimized Permutations

Representations of sets are challenging to learn because operations on s...

Geometric Prediction: Moving Beyond Scalars

Many quantities we are interested in predicting are geometric tensors; w...