Log In Sign Up

Sharp Multiple Instance Learning for DeepFake Video Detection

by   Xiaodan Li, et al.

With the rapid development of facial manipulation techniques, face forgery has received considerable attention in multimedia and computer vision community due to security concerns. Existing methods are mostly designed for single-frame detection trained with precise image-level labels or for video-level prediction by only modeling the inter-frame inconsistency, leaving potential high risks for DeepFake attackers. In this paper, we introduce a new problem of partial face attack in DeepFake video, where only video-level labels are provided but not all the faces in the fake videos are manipulated. We address this problem by multiple instance learning framework, treating faces and input video as instances and bag respectively. A sharp MIL (S-MIL) is proposed which builds direct mapping from instance embeddings to bag prediction, rather than from instance embeddings to instance prediction and then to bag prediction in traditional MIL. Theoretical analysis proves that the gradient vanishing in traditional MIL is relieved in S-MIL. To generate instances that can accurately incorporate the partially manipulated faces, spatial-temporal encoded instance is designed to fully model the intra-frame and inter-frame inconsistency, which further helps to promote the detection performance. We also construct a new dataset FFPMS for partially attacked DeepFake video detection, which can benefit the evaluation of different methods at both frame and video levels. Experiments on FFPMS and the widely used DFDC dataset verify that S-MIL is superior to other counterparts for partially attacked DeepFake video detection. In addition, S-MIL can also be adapted to traditional DeepFake image detection tasks and achieve state-of-the-art performance on single-frame datasets.


page 1

page 4

page 5

page 6

page 7

page 8

page 11


Address Instance-level Label Prediction in Multiple Instance Learning

Multiple Instance Learning (MIL) is concerned with learning from bags of...

FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces

With recent advances in computer vision and graphics, it is now possible...

Dual Contrastive Learning for General Face Forgery Detection

With various facial manipulation techniques arising, face forgery detect...

Detection of Deepfake Videos Using Long Distance Attention

With the rapid progress of deepfake techniques in recent years, facial v...

Multiple Instance-Based Video Anomaly Detection using Deep Temporal Encoding-Decoding

In this paper, we propose a weakly supervised deep temporal encoding-dec...

Face Forensics in the Wild

On existing public benchmarks, face forgery detection techniques have ac...

Interventional Multi-Instance Learning with Deconfounded Instance-Level Prediction

When applying multi-instance learning (MIL) to make predictions for bags...