Sherlock: Scalable Fact Learning in Images

11/16/2015
by   Mohamed Elhoseiny, et al.
0

We study scalable and uniform understanding of facts in images. Existing visual recognition systems are typically modeled differently for each fact type such as objects, actions, and interactions. We propose a setting where all these facts can be modeled simultaneously with a capacity to understand unbounded number of facts in a structured way. The training data comes as structured facts in images, including (1) objects (e.g., <boy>), (2) attributes (e.g., <boy, tall>), (3) actions (e.g., <boy, playing>), and (4) interactions (e.g., <boy, riding, a horse >). Each fact has a semantic language view (e.g., < boy, playing>) and a visual view (an image with this fact). We show that learning visual facts in a structured way enables not only a uniform but also generalizable visual understanding. We propose and investigate recent and strong approaches from the multiview learning literature and also introduce two learning representation models as potential baselines. We applied the investigated methods on several datasets that we augmented with structured facts and a large scale dataset of more than 202,000 facts and 814,000 images. Our experiments show the advantage of relating facts by the structure by the proposed models compared to the designed baselines on bidirectional fact retrieval.

READ FULL TEXT

page 26

page 28

page 29

page 30

page 31

page 36

page 37

page 38

research
04/02/2016

Automatic Annotation of Structured Facts in Images

Motivated by the application of fact-level image understanding, we prese...
research
12/26/2018

Exploring the Challenges towards Lifelong Fact Learning

So far life-long learning (LLL) has been studied in relatively small-sca...
research
11/21/2022

Higher-Order, Data-Parallel Structured Deduction

State-of-the-art Datalog engines include expressive features such as ADT...
research
12/24/2020

Robotic Following of Flexible Extended Objects: Relevant Technical Facts on the Kinematics of a Moving Continuum

The paper offers general technical facts on the kinematics of a moving c...
research
05/07/2018

Weakly-supervised Contextualization of Knowledge Graph Facts

Knowledge graphs (KGs) model facts about the world, they consist of node...
research
08/12/2023

Calliope-Net: Automatic Generation of Graph Data Facts via Annotated Node-link Diagrams

Graph or network data are widely studied in both data mining and visuali...
research
04/12/2021

Factual Probing Is [MASK]: Learning vs. Learning to Recall

Petroni et al. (2019) demonstrated that it is possible to retrieve world...

Please sign up or login with your details

Forgot password? Click here to reset