On- Device Information Extraction from Screenshots in form of tags

by   Sumit Kumar, et al.

We propose a method to make mobile screenshots easily searchable. In this paper, we present the workflow in which we: 1) preprocessed a collection of screenshots, 2) identified script presentin image, 3) extracted unstructured text from images, 4) identifiedlanguage of the extracted text, 5) extracted keywords from the text, 6) identified tags based on image features, 7) expanded tag set by identifying related keywords, 8) inserted image tags with relevant images after ranking and indexed them to make it searchable on device. We made the pipeline which supports multiple languages and executed it on-device, which addressed privacy concerns. We developed novel architectures for components in the pipeline, optimized performance and memory for on-device computation. We observed from experimentation that the solution developed can reduce overall user effort and improve end user experience while searching, whose results are published.



There are no comments yet.


page 7


On-Device Tag Generation for Unstructured Text

With the overwhelming transition to smart phones, storing important info...

Automatic Visual Theme Discovery from Joint Image and Text Corpora

A popular approach to semantic image understanding is to manually tag im...

Image Privacy Prediction Using Deep Neural Networks

Images today are increasingly shared online on social networking sites s...

ScreenSeg: On-Device Screenshot Layout Analysis

We propose a novel end-to-end solution that performs a Hierarchical Layo...

On-Device Sentence Similarity for SMS Dataset

Determining the sentence similarity between Short Message Service (SMS) ...

FONTNET: On-Device Font Understanding and Prediction Pipeline

Fonts are one of the most basic and core design concepts. Numerous use c...

Analysing Word Importance for Image Annotation

Image annotation provides several keywords automatically for a given ima...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.