DeepAI AI Chat
Log In Sign Up

Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

by   Kevin Moran, et al.

It is common practice for developers of user-facing software to transform a mock-up of a graphical user interface (GUI) into code. This process takes place both at an application's inception and in an evolutionary context as GUI changes keep pace with evolving features. Unfortunately, this practice is challenging and time-consuming. In this paper, we present an approach that automates this process by enabling accurate prototyping of GUIs via three tasks: detection, classification, and assembly. First, logical components of a GUI are detected from a mock-up artifact using either computer vision techniques or mock-up metadata. Then, software repository mining, automated dynamic analysis, and deep convolutional neural networks are utilized to accurately classify GUI-components into domain-specific types (e.g., toggle-button). Finally, a data-driven, K-nearest-neighbors algorithm generates a suitable hierarchical GUI structure from which a prototype application can be automatically assembled. We implemented this approach for Android in a system called ReDraw. Our evaluation illustrates that ReDraw achieves an average GUI-component classification accuracy of 91 applications that closely mirror target mock-ups in terms of visual affinity while exhibiting reasonable code structure. Interviews with industrial practitioners illustrate ReDraw's potential to improve real development workflows.


page 7

page 10

page 18

page 23


Automatic code generation from sketches of mobile applications in end-user development using Deep Learning

A common need for mobile application development by end-users or in comp...

Automated Reporting of GUI Design Violations for Mobile Apps

The inception of a mobile app often takes form of a mock-up of the Graph...

Detecting and Summarizing GUI Changes in Evolving Mobile Apps

Mobile applications have become a popular software development domain in...

Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios

Screen recordings of mobile applications are easy to obtain and capture ...

pix2code: Generating Code from a Graphical User Interface Screenshot

Transforming a graphical user interface screenshot created by a designer...

BabelView: Evaluating the Impact of Code Injection Attacks in Mobile Webviews

A Webview embeds a full-fledged browser in a mobile application and allo...

ZeroIn: Characterizing the Data Distributions of Commits in Software Repositories

Modern software development is based on a series of rapid incremental ch...