Visually explaining 3D-CNN predictions for video classification with an adaptive occlusion sensitivity analysis

07/26/2022
by   Tomoki Uchiyama, et al.
0

This paper proposes a method for visually explaining the decision-making process of 3D convolutional neural networks (CNN) with a temporal extension of occlusion sensitivity analysis. The key idea here is to occlude a specific volume of data by a 3D mask in an input 3D temporal-spatial data space and then measure the change degree in the output score. The occluded volume data that produces a larger change degree is regarded as a more critical element for classification. However, while the occlusion sensitivity analysis is commonly used to analyze single image classification, it is not so straightforward to apply this idea to video classification as a simple fixed cuboid cannot deal with the motions. To this end, we adapt the shape of a 3D occlusion mask to complicated motions of target objects. Our flexible mask adaptation is performed by considering the temporal continuity and spatial co-occurrence of the optical flows extracted from the input video data. We further propose to approximate our method by using the first-order partial derivative of the score with respect to an input image to reduce its computational cost. We demonstrate the effectiveness of our method through various and extensive comparisons with the conventional methods in terms of the deletion/insertion metric and the pointing metric on the UCF-101. The code is available at: https://github.com/uchiyama33/AOSA.

READ FULL TEXT

page 5

page 8

research
03/24/2020

MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Mask

Feature warping is a core technique in optical flow estimation; however,...
research
07/12/2022

Occluded Human Body Capture with Self-Supervised Spatial-Temporal Motion Prior

Although significant progress has been achieved on monocular maker-less ...
research
07/23/2020

Right for the Right Reason: Making Image Classification Robust

Convolutional neural networks (CNNs) have achieved astonishing performan...
research
11/24/2017

End-to-End Deep HDR Imaging with Large Foreground Motions

This paper proposes the first end-to-end deep framework for high dynamic...
research
08/12/2021

MT-ORL: Multi-Task Occlusion Relationship Learning

Retrieving occlusion relation among objects in a single image is challen...
research
01/26/2023

Distilling Cognitive Backdoor Patterns within an Image

This paper proposes a simple method to distill and detect backdoor patte...
research
06/12/2017

SmoothGrad: removing noise by adding noise

Explaining the output of a deep network remains a challenge. In the case...

Please sign up or login with your details

Forgot password? Click here to reset