Deep Structured Scene Parsing by Learning with Image Descriptions

04/08/2016
by   Liang Lin, et al.
0

This paper addresses a fundamental problem of scene understanding: How to parse the scene image into a structured configuration (i.e., a semantic object hierarchy with object interaction relations) that finely accords with human perception. We propose a deep architecture consisting of two networks: i) a convolutional neural network (CNN) extracting the image representation for pixelwise object labeling and ii) a recursive neural network (RNN) discovering the hierarchical object structure and the inter-object relations. Rather than relying on elaborative user annotations (e.g., manually labeling semantic maps and relations), we train our deep model in a weakly-supervised manner by leveraging the descriptive sentences of the training images. Specifically, we decompose each sentence into a semantic tree consisting of nouns and verb phrases, and facilitate these trees discovering the configurations of the training images. Once these scene configurations are determined, then the parameters of both the CNN and RNN are updated accordingly by back propagation. The entire model training is accomplished through an Expectation-Maximization method. Extensive experiments suggest that our model is capable of producing meaningful and structured scene configurations and achieving more favorable scene labeling performance on PASCAL VOC 2012 over other state-of-the-art weakly-supervised methods.

READ FULL TEXT
research
09/27/2017

Scene Parsing by Weakly Supervised Learning with Image Descriptions

This paper investigates a fundamental problem of scene understanding: ho...
research
01/08/2020

Weakly Supervised Visual Semantic Parsing

Scene Graph Generation (SGG) aims to extract entities, predicates and th...
research
12/06/2019

Controlling Style and Semantics in Weakly-Supervised Image Generation

We propose a weakly-supervised approach for conditional image generation...
research
06/11/2015

Constrained Convolutional Neural Networks for Weakly Supervised Segmentation

We present an approach to learn a dense pixel-wise labeling from image-l...
research
04/07/2016

Geometric Scene Parsing with Hierarchical LSTM

This paper addresses the problem of geometric scene parsing, i.e. simult...
research
10/01/2013

Deep and Wide Multiscale Recursive Networks for Robust Image Labeling

Feedforward multilayer networks trained by supervised learning have rece...

Please sign up or login with your details

Forgot password? Click here to reset