Multi-Frame Content Integration with a Spatio-Temporal Attention Mechanism for Person Video Motion Transfer

by   Kun Cheng, et al.

Existing person video generation methods either lack the flexibility in controlling both the appearance and motion, or fail to preserve detailed appearance and temporal consistency. In this paper, we tackle the problem of motion transfer for generating person videos, which provides controls on both the appearance and the motion. Specifically, we transfer the motion of one person in a target video to another person in a source video, while preserving the appearance of the source person. Besides only relying on one source frame as the existing state-of-the-art methods, our proposed method integrates information from multiple source frames based on a spatio-temporal attention mechanism to preserve rich appearance details. In addition to a spatial discriminator employed for encouraging the frame-level fidelity, a multi-range temporal discriminator is adopted to enforce the generated video to resemble temporal dynamics of a real video in various time ranges. A challenging real-world dataset, which contains about 500 dancing video clips with complex and unpredictable motions, is collected for the training and testing. Extensive experiments show that the proposed method can produce more photo-realistic and temporally consistent person videos than previous methods. As our method decomposes the syntheses of the foreground and background into two branches, a flexible background substitution application can also be achieved.


page 2

page 3

page 4

page 7

page 8


Image Comes Dancing with Collaborative Parsing-Flow Video Synthesis

Transferring human motion from a source to a target person poses great p...

Dynamics Transfer GAN: Generating Video by Transferring Arbitrary Temporal Dynamics from a Source Video to a Single Target Image

In this paper, we propose Dynamics Transfer GAN; a new method for genera...

C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer

Human video motion transfer (HVMT) aims to synthesize videos that one pe...

Towards Using Clothes Style Transfer for Scenario-aware Person Video Generation

Clothes style transfer for person video generation is a challenging task...

Delving into the Frequency: Temporally Consistent Human Motion Transfer in the Fourier Space

Human motion transfer refers to synthesizing photo-realistic and tempora...

Copy Motion From One to Another: Fake Motion Video Generation

One compelling application of artificial intelligence is to generate a v...

A Spatio-temporal Attention-based Model for Infant Movement Assessment from Videos

The absence or abnormality of fidgety movements of joints or limbs is st...

Please sign up or login with your details

Forgot password? Click here to reset