Visual Data Synthesis via GAN for Zero-Shot Video Classification

04/26/2018
by   Chenrui Zhang, et al.
0

Zero-Shot Learning (ZSL) in video classification is a promising research direction, which aims to tackle the challenge from explosive growth of video categories. Most existing methods exploit seen-to-unseen correlation via learning a projection between visual and semantic spaces. However, such projection-based paradigms cannot fully utilize the discriminative information implied in data distribution, and commonly suffer from the information degradation issue caused by "heterogeneity gap". In this paper, we propose a visual data synthesis framework via GAN to address these problems. Specifically, both semantic knowledge and visual distribution are leveraged to synthesize video feature of unseen categories, and ZSL can be turned into typical supervised problem with the synthetic features. First, we propose multi-level semantic inference to boost video feature synthesis, which captures the discriminative information implied in joint visual-semantic distribution via feature-level and label-level semantic inference. Second, we propose Matching-aware Mutual Information Correlation to overcome information degradation issue, which captures seen-to-unseen correlation in matched and mismatched visual-semantic pairs by mutual information, providing the zero-shot synthesis procedure with robust guidance signals. Experimental results on four video datasets demonstrate that our approach can improve the zero-shot video classification performance significantly.

READ FULL TEXT
research
03/17/2020

Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification

Zero-shot learning strives to classify unseen categories for which no da...
research
10/19/2018

Zero and Few Shot Learning with Semantic Feature Synthesis and Competitive Learning

Zero-shot learning (ZSL) is made possible by learning a projection funct...
research
05/04/2017

From Zero-shot Learning to Conventional Supervised Classification: Unseen Visual Data Synthesis

Robust object recognition systems usually rely on powerful feature extra...
research
06/19/2023

Primitive Generation and Semantic-related Alignment for Universal Zero-Shot Segmentation

We study universal zero-shot segmentation in this work to achieve panopt...
research
05/22/2023

Zero-shot Multi-level Feature Transmission Policy Powered by Semantic Knowledge Base

Remote zero-shot object recognition, i.e., offloading zero-shot object r...
research
03/29/2022

Alignment-Uniformity aware Representation Learning for Zero-shot Video Classification

Most methods tackle zero-shot video classification by aligning visual-se...
research
08/09/2019

Zero-shot Feature Selection via Exploiting Semantic Knowledge

Feature selection plays an important role in pattern recognition and mac...

Please sign up or login with your details

Forgot password? Click here to reset