Automatic code generation from sketches of mobile applications in end-user development using Deep Learning

03/09/2021
by   Daniel Baulé, et al.
0

A common need for mobile application development by end-users or in computing education is to transform a sketch of a user interface into wireframe code using App Inventor, a popular block-based programming environment. As this task is challenging and time-consuming, we present the Sketch2aia approach that automates this process. Sketch2aia employs deep learning to detect the most frequent user interface components and their position on a hand-drawn sketch creating an intermediate representation of the user interface and then automatically generates the App Inventor code of the wireframe. The approach achieves an average user interface component classification accuracy of 87,72 and results of a preliminary user evaluation indicate that it generates wireframes that closely mirror the sketches in terms of visual similarity. The approach has been implemented as a web tool and can be used to support the end-user development of mobile applications effectively and efficiently as well as the teaching of user interface design in K-12.

READ FULL TEXT

page 11

page 12

page 13

research
02/07/2018

Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps

It is common practice for developers of user-facing software to transfor...
research
10/20/2019

Sketch2Code: Transformation of Sketches to UI in Real-time Using Deep Neural Network

User Interface (UI) prototyping is a necessary step in the early stages ...
research
06/14/2021

Magic Layouts: Structural Prior for Component Detection in User Interface Designs

We present Magic Layouts; a method for parsing screenshots or hand-drawn...
research
07/21/2020

Intelligent Exploration for User Interface Modules of Mobile App with Collective Learning

A mobile app interface usually consists of a set of user interface modul...
research
11/19/2021

Sketch-based Creativity Support Tools using Deep Learning

Sketching is a natural and effective visual communication medium commonl...
research
07/01/2020

PrototypeML: A Neural Network Integrated Design and Development Environment

Neural network architectures are most often conceptually designed and de...
research
02/04/2021

User Interface Factors of Mobile UX: A Study with an Incident Reporting Application

Smartphones are now ubiquitous, yet our understanding of user interface ...

Please sign up or login with your details

Forgot password? Click here to reset