Deep Embedding for Spatial Role Labeling

03/28/2016
by   Oswaldo Ludwig, et al.
0

This paper introduces the visually informed embedding of word (VIEW), a continuous vector representation for a word extracted from a deep neural model trained using the Microsoft COCO data set to forecast the spatial arrangements between visual objects, given a textual description. The model is composed of a deep multilayer perceptron (MLP) stacked on the top of a Long Short Term Memory (LSTM) network, the latter being preceded by an embedding layer. The VIEW is applied to transferring multimodal background knowledge to Spatial Role Labeling (SpRL) algorithms, which recognize spatial relations between objects mentioned in the text. This work also contributes with a new method to select complementary features and a fine-tuning method for MLP that improves the F1 measure in classifying the words into spatial roles. The VIEW is evaluated with the Task 3 of SemEval-2013 benchmark data set, SpaceEval.

READ FULL TEXT
research
08/11/2017

Argument Labeling of Explicit Discourse Relations using LSTM Neural Networks

Argument labeling of explicit discourse relations is a challenging task....
research
01/29/2016

Lipreading with Long Short-Term Memory

Lipreading, i.e. speech recognition from visual-only recordings of a spe...
research
06/08/2017

Image Captioning with Object Detection and Localization

Automatically generating a natural language description of an image is a...
research
09/30/2019

A Video Recognition Method by using Adaptive Structural Learning of Long Short Term Memory based Deep Belief Network

Deep learning builds deep architectures such as multi-layered artificial...
research
04/18/2016

LSTM-CF: Unifying Context Modeling and Fusion with LSTMs for RGB-D Scene Labeling

Semantic labeling of RGB-D scenes is crucial to many intelligent applica...
research
11/04/2019

A Model for Spatial Outlier Detection Based on Weighted Neighborhood Relationship

Spatial outliers are used to discover inconsistent objects producing imp...

Please sign up or login with your details

Forgot password? Click here to reset