Towards Task Understanding in Visual Settings

11/28/2018
by   Sebastin Santy, et al.
0

We consider the problem of understanding real world tasks depicted in visual images. While most existing image captioning methods excel in producing natural language descriptions of visual scenes involving human tasks, there is often the need for an understanding of the exact task being undertaken rather than a literal description of the scene. We leverage insights from real world task understanding systems, and propose a framework composed of convolutional neural networks, and an external hierarchical task ontology to produce task descriptions from input images. Detailed experiments highlight the efficacy of the extracted descriptions, which could potentially find their way in many applications, including image alt text generation.

READ FULL TEXT
research
08/05/2023

A Comprehensive Analysis of Real-World Image Captioning and Scene Identification

Image captioning is a computer vision task that involves generating natu...
research
09/07/2023

A Function Interpretation Benchmark for Evaluating Interpretability Methods

Labeling neural network submodules with human-legible descriptions is us...
research
01/17/2023

Embodied Agents for Efficient Exploration and Smart Scene Description

The development of embodied agents that can communicate with humans in n...
research
07/11/2018

Towards Understanding End-of-trip Instructions in a Taxi Ride Scenario

We introduce a dataset containing human-authored descriptions of target ...
research
03/15/2021

Knowledge driven Description Synthesis for Floor Plan Interpretation

Image captioning is a widely known problem in the area of AI. Caption ge...
research
10/11/2022

Underspecification in Scene Description-to-Depiction Tasks

Questions regarding implicitness, ambiguity and underspecification are c...
research
03/30/2018

Guide Me: Interacting with Deep Networks

Interaction and collaboration between humans and intelligent machines ha...

Please sign up or login with your details

Forgot password? Click here to reset