Towards Unsupervised Visual Reasoning: Do Off-The-Shelf Features Know How to Reason?

12/20/2022
by   Monika Wysoczanska, et al.
0

Recent advances in visual representation learning allowed to build an abundance of powerful off-the-shelf features that are ready-to-use for numerous downstream tasks. This work aims to assess how well these features preserve information about the objects, such as their spatial location, their visual properties and their relative relationships. We propose to do so by evaluating them in the context of visual reasoning, where multiple objects with complex relationships and different attributes are at play. More specifically, we introduce a protocol to evaluate visual representations for the task of Visual Question Answering. In order to decouple visual feature extraction from reasoning, we design a specific attention-based reasoning module which is trained on the frozen visual representations to be evaluated, in a spirit similar to standard feature evaluations relying on shallow networks. We compare two types of visual representations, densely extracted local features and object-centric ones, against the performances of a perfect image representation using ground truth. Our main findings are two-fold. First, despite excellent performances on classical proxy tasks, such representations fall short for solving complex reasoning problem. Second, object-centric features better preserve the critical information necessary to perform visual reasoning. In our proposed framework we show how to methodologically approach this evaluation.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
12/31/2020

Language-Mediated, Object-Centric Representation Learning

We present Language-mediated, Object-centric Representation Learning (LO...
research
12/15/2021

Object Pursuit: Building a Space of Objects via Discriminative Weight Generation

We propose a framework to continuously learn object-centric representati...
research
05/25/2018

Think Visually: Question Answering through Virtual Imagery

In this paper, we study the problem of geometric reasoning in the contex...
research
04/12/2021

Object-Centric Representation Learning for Video Question Answering

Video question answering (Video QA) presents a powerful testbed for huma...
research
09/20/2023

StructChart: Perception, Structuring, Reasoning for Visual Chart Understanding

Charts are common in literature across different scientific fields, conv...
research
07/16/2019

2nd Place Solution to the GQA Challenge 2019

We present a simple method that achieves unexpectedly superior performan...
research
04/14/2023

The role of object-centric representations, guided attention, and external memory on generalizing visual relations

Visual reasoning is a long-term goal of vision research. In the last dec...

Please sign up or login with your details

Forgot password? Click here to reset