Write a Classifier: Predicting Visual Classifiers from Unstructured Text

by   Mohamed Elhoseiny, et al.

People typically learn through exposure to visual concepts associated with linguistic descriptions. For instance, teaching visual object categories to children is often accompanied by descriptions in text or speech. In a machine learning context, these observations motivates us to ask whether this learning process could be computationally modeled to learn visual classifiers. More specifically, the main question of this work is how to utilize purely textual description of visual classes with no training images, to learn explicit visual classifiers for them. We propose and investigate two baseline formulations, based on regression and domain transfer, that predict a linear classifier. Then, we propose a new constrained optimization formulation that combines a regression function and a knowledge transfer function with additional constraints to predict the parameters of a linear classifier. We also propose a generic kernelized models where a kernel classifier is predicted in the form defined by the representer theorem. The kernelized models allow defining and utilizing any two RKHS (Reproducing Kernel Hilbert Space) kernel functions in the visual space and text space, respectively. We finally propose a kernel function between unstructured text descriptions that builds on distributional semantics, which shows an advantage in our setting and could be useful for other applications. We applied all the studied models to predict visual classifiers on two fine-grained and challenging categorization datasets (CU Birds and Flower Datasets), and the results indicate successful predictions of our final model over several baselines that we designed.


page 1

page 2

page 6


Tell and Predict: Kernel Classifier Prediction for Unseen Visual Classes from Unstructured Text Descriptions

In this paper we propose a framework for predicting kernelized classifie...

Link the head to the "beak": Zero Shot Learning from Noisy Text Description at Part Precision

In this paper, we study learning visual classifiers from unstructured te...

ZEST: Zero-shot Learning from Text Descriptions using Textual Similarity and Visual Summarization

We study the problem of recognizing visual entities from the textual des...

Alpha Net: Adaptation with Composition in Classifier Space

Deep learning classification models typically train poorly on classes wi...

A kernel function for Signal Temporal Logic formulae

We discuss how to define a kernel for Signal Temporal Logic (STL) formul...

TV News Commercials Detection using Success based Locally Weighted Kernel Combination

Commercial detection in news broadcast videos involves judicious selecti...

Neural Naturalist: Generating Fine-Grained Image Comparisons

We introduce the new Birds-to-Words dataset of 41k sentences describing ...

Please sign up or login with your details

Forgot password? Click here to reset