With the ubiquity of multimedia videos, there has been a massive interest from the advertisement and marketing agencies to provide targeted advertisements for the customers. Such targeted advertisements are useful, both from the perspectives of marketing agents and end users. The advertisement agencies can use a powerful media for marketing and publicity; and the users can interact via a personalized consumer experience. In this paper, we attempt to solve this by designing an online advert creation system for next-gen publicity. We develop and implement an end-to-end system for automatically detecting and seamlessly changing an existing billboard in a video by inserting a new advert. This system will be helpful for online marketers and content developers, to develop video contents for targeted audience.
Figure 1 illustrates our system. Our system automatically detects the presence of a billboard in an image frame from the video sequence. Post billboard detection, our system also localizes its position in the image frame. The user is given an opportunity to manually adjust and refine the detected four corners of the billboard. Finally, a new advertisement is integrated into the image, and tracked across all frames of the video sequence. Thereby, we generate a new composite video with the integrated advert.
Currently, there are no such existing framework available in the literature that aid the marketing agents to seamlessly integrate a new advertisement, into an original video sequence. However, a few companies viz. Mirriad  uses patented advertisement plantation technique to integrate 3D objects in a video sequence.
The backbone of our advert creation system is based on state-of-the-art techniques from deep learning and image processing. In this section, we briefly describe the underlying techniques used in the various components of the demo system. The different modules of our system are: advert- recognition, localization, and integration.
2.1 Advert Recognition
The first module of our advert creation system is used for the recognition of billboard 111In this paper, we interchangeably use both the terms, billboard and advert to indicate a candidate object for new advertisement integration in an image frame.
– does an image frame from the video sequence contain billboard? This helps the system user to automatically detect the presence of billboard in an image frame of the video. We use a deep neural network (DNN) as a binary classifier where classes representpresence and absence of billboard in video frame respectively. We use a VGG-based network  layers. We add fully connected layers with a softmax layer as the output layer. We train this deep network on our annotated dataset, containing both billboard and non-billboard images, and achieve good accuracy on billboard recognition.
2.2 Advert Localization
The second module of our advert creation system is used for localizing the position of recognized billboard – where is the billboard located in image frame? We use a encoder-decoder based deep neural network that localizes the billboard position in an image. We train this model on our billboard dataset comprising input images (cf. Fig. 2(a)) and corresponding binary ground truth image (cf. Fig. 22.
2.3 Advert Integration
The third and final module of our system is advert integration – how to integrate a new advert in the video? In this stage, the localized billboard is replaced with a new advert in a seamless and temporally consistent manner. We use Poisson image editing  on the new advert, to achieve similar local illumination and local color tone, as the original video sequence. Furthermore, the relative motion of the billboard within the scene is tracked using Kanade-Lucas-Tomasi (KLT)  tracking technique.
3 Design and Interface
Figure 3 illustrates a sample snapshot of our developed web-based tool. The web interface consists of primarily three sections: Home, Demo and Images. The page Home provides an overview of the system. The next page Demo
describes the entire working prototype of our system. The user selects a sample video from the list, runs the billboard detection module to accurately localize the billboard at sample image frames of the video. The detection module estimates the four corners of the billboard. However, the user also gets an option torefine the four corners manually, if the detected four corners are not completely accurate. The refined four corners of the billboard are subsequently used for tracking and integration of a new advertisement into the video sequence. The third and final web page Images contains the list of all candidate adverts that can be integrated into the selected video sequence.
Finally, our system integrates the new advertisement into the detected billboard position, and generates a new composite video with the implanted advertisement.
4 Conclusion and Future Work
In this paper, we have presented an online advert creation system on multimedia videos for a personalized and targeted advertisement. We use techniques from deep neural networks and image processing, for a seamless integration of new adverts into existing videos. Our system is trained on datasets that comprises outdoor scenes and views. Our future work involve further refining the performance of the system, and also generalizing it to other video sequence types.
The ADAPT Centre for Digital Content Technology is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund.
-  Mirriad: Scalable, effective campaigns (Accessed 7-May-2018 2018), http://www.mirriad.com/
-  Lucas, B.D., Kanade, T., et al.: An iterative image registration technique with an application to stereo vision (1981)
-  Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Transactions on graphics (TOG) 22(3)
-  Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)