Sketch Me A Video

10/10/2021
by   Haichao Zhang, et al.
4

Video creation has been an attractive yet challenging task for artists to explore. With the advancement of deep learning, recent works try to utilize deep convolutional neural networks to synthesize a video with the aid of a guiding video, and have achieved promising results. However, the acquisition of guiding videos, or other forms of guiding temporal information is costly expensive and difficult in reality. Therefore, in this work we introduce a new video synthesis task by employing two rough bad-drwan sketches only as input to create a realistic portrait video. A two-stage Sketch-to-Video model is proposed, which consists of two key novelties: 1) a feature retrieve and projection (FRP) module, which parititions the input sketch into different parts and utilizes these parts for synthesizing a realistic start or end frame and meanwhile generating rich semantic features, is designed to alleviate the sketch out-of-domain problem due to arbitrarily drawn free-form sketch styles by different users. 2) A motion projection followed by feature blending module, which projects a video (used only in training phase) into a motion space modeled by normal distribution and blends the motion variables with semantic features extracted above, is proposed to alleviate the guiding temporal information missing problem in the test phase. Experiments conducted on a combination of CelebAMask-HQ and VoxCeleb2 dataset well validate that, our method can acheive both good quantitative and qualitative results in synthesizing high-quality videos from two rough bad-drawn sketches.

READ FULL TEXT

page 1

page 4

page 7

page 8

research
08/10/2020

Deep Sketch-guided Cartoon Video Synthesis

We propose a novel framework to produce cartoon videos by fetching the c...
research
02/10/2015

Video Primal Sketch: A Unified Middle-Level Representation for Video

This paper presents a middle-level video representation named Video Prim...
research
04/02/2023

Sketch-based Video Object Localization

We introduce Sketch-based Video Object Localization (SVOL), a new task a...
research
04/12/2021

Adversarial Open Domain Adaption for Sketch-to-Photo Synthesis

In this paper, we explore the open-domain sketch-to-photo translation, w...
research
06/13/2023

AniFaceDrawing: Anime Portrait Exploration during Your Sketching

In this paper, we focus on how artificial intelligence (AI) can be used ...
research
08/31/2023

StyleInV: A Temporal Style Modulated Inversion Network for Unconditional Video Generation

Unconditional video generation is a challenging task that involves synth...
research
08/14/2022

SketchSampler: Sketch-based 3D Reconstruction via View-dependent Depth Sampling

Reconstructing a 3D shape based on a single sketch image is challenging ...

Please sign up or login with your details

Forgot password? Click here to reset