DeepAI AI Chat
Log In Sign Up

V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matrices

by   Damien Teney, et al.

One of the primary challenges faced by deep learning is the degree to which current methods exploit superficial statistics and dataset bias, rather than learning to generalise over the specific representations they have experienced. This is a critical concern because generalisation enables robust reasoning over unseen data, whereas leveraging superficial statistics is fragile to even small changes in data distribution. To illuminate the issue and drive progress towards a solution, we propose a test that explicitly evaluates abstract reasoning over visual data. We introduce a large-scale benchmark of visual questions that involve operations fundamental to many high-level vision tasks, such as comparisons of counts and logical operations on complex visual properties. The benchmark directly measures a method's ability to infer high-level relationships and to generalise them over image-based concepts. It includes multiple training/test splits that require controlled levels of generalization. We evaluate a range of deep learning architectures, and find that existing models, including those popular for vision-and-language tasks, are unable to solve seemingly-simple instances. Models using relational networks fare better but leave substantial room for improvement.


page 1

page 3

page 7

page 8

page 14

page 15


RAVEN: A Dataset for Relational and Analogical Visual rEasoNing

Dramatic progress has been witnessed in basic vision tasks involving low...

A Benchmark for Compositional Visual Reasoning

A fundamental component of human vision is our ability to parse complex ...

ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning

Charts are very popular for analyzing data. When exploring charts, peopl...

Deep Non-Monotonic Reasoning for Visual Abstract Reasoning Tasks

While achieving unmatched performance on many well-defined tasks, deep l...

Generalisable Relational Reasoning With Comparators in Low-Dimensional Manifolds

While modern deep neural architectures generalise well when test data is...

Blackbird's language matrices (BLMs): a new benchmark to investigate disentangled generalisation in neural networks

Current successes of machine learning architectures are based on computa...

Hierarchical Rule Induction Network for Abstract Visual Reasoning

Abstract reasoning refers to the ability to analyze information, discove...