DeepAI AI Chat
Log In Sign Up

Automatic code generation from sketches of mobile applications in end-user development using Deep Learning

by   Daniel Baulé, et al.

A common need for mobile application development by end-users or in computing education is to transform a sketch of a user interface into wireframe code using App Inventor, a popular block-based programming environment. As this task is challenging and time-consuming, we present the Sketch2aia approach that automates this process. Sketch2aia employs deep learning to detect the most frequent user interface components and their position on a hand-drawn sketch creating an intermediate representation of the user interface and then automatically generates the App Inventor code of the wireframe. The approach achieves an average user interface component classification accuracy of 87,72 and results of a preliminary user evaluation indicate that it generates wireframes that closely mirror the sketches in terms of visual similarity. The approach has been implemented as a web tool and can be used to support the end-user development of mobile applications effectively and efficiently as well as the teaching of user interface design in K-12.


page 11

page 12

page 13


Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

It is common practice for developers of user-facing software to transfor...

Sketch2Code: Transformation of Sketches to UI in Real-time Using Deep Neural Network

User Interface (UI) prototyping is a necessary step in the early stages ...

Magic Layouts: Structural Prior for Component Detection in User Interface Designs

We present Magic Layouts; a method for parsing screenshots or hand-drawn...

Intelligent Exploration for User Interface Modules of Mobile App with Collective Learning

A mobile app interface usually consists of a set of user interface modul...

Sketch-based Creativity Support Tools using Deep Learning

Sketching is a natural and effective visual communication medium commonl...

PrototypeML: A Neural Network Integrated Design and Development Environment

Neural network architectures are most often conceptually designed and de...

User Interface Factors of Mobile UX: A Study with an Incident Reporting Application

Smartphones are now ubiquitous, yet our understanding of user interface ...