Mining Interpretable AOG Representations from Convolutional Networks via Active Question Answering

12/18/2018
by   Quanshi Zhang, et al.
6

In this paper, we present a method to mine object-part patterns from conv-layers of a pre-trained convolutional neural network (CNN). The mined object-part patterns are organized by an And-Or graph (AOG). This interpretable AOG representation consists of a four-layer semantic hierarchy, i.e., semantic parts, part templates, latent patterns, and neural units. The AOG associates each object part with certain neural units in feature maps of conv-layers. The AOG is constructed in a weakly-supervised manner, i.e., very few annotations (e.g., 3-20) of object parts are used to guide the learning of AOGs. We develop a question-answering (QA) method that uses active human-computer communications to mine patterns from a pre-trained CNN, in order to incrementally explain more features in conv-layers. During the learning process, our QA method uses the current AOG for part localization. The QA method actively identifies objects, whose feature maps cannot be explained by the AOG. Then, our method asks people to annotate parts on the unexplained objects, and uses answers to discover CNN patterns corresponding to the newly labeled parts. In this way, our method gradually grows new branches and refines existing branches on the AOG to semanticize CNN representations. In experiments, our method exhibited a high learning efficiency. Our method used about 1/6-1/3 of the part annotations for training, but achieved similar or better part-localization performance than fast-RCNN methods.

READ FULL TEXT

page 2

page 3

page 11

page 14

research
04/11/2017

Mining Object Parts from CNNs via Active Question-Answering

Given a convolutional neural network (CNN) that is pre-trained for objec...
research
11/14/2016

Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning

This paper proposes a learning strategy that extracts object-part concep...
research
05/18/2018

Unsupervised Learning of Neural Networks to Explain Neural Networks

This paper presents an unsupervised method to learn a neural network, na...
research
01/08/2019

Interpretable CNNs

This paper proposes a generic method to learn interpretable convolutiona...
research
08/13/2017

A Cost-Sensitive Visual Question-Answer Framework for Mining a Deep And-OR Object Semantics from Web Images

This paper presents a cost-sensitive Question-Answering (QA) framework f...
research
08/05/2017

Interactively Transferring CNN Patterns for Part Localization

In the scenario of one/multi-shot learning, conventional end-to-end lear...
research
02/26/2019

Mining Objects: Fully Unsupervised Object Discovery and Localization From a Single Image

The goal of our work is to discover dominant objects without using any a...

Please sign up or login with your details

Forgot password? Click here to reset