Learning to Infer User Interface Attributes from Images

12/31/2019
by   Philippe Schlattner, et al.
21

We explore a new domain of learning to infer user interface attributes that helps developers automate the process of user interface implementation. Concretely, given an input image created by a designer, we learn to infer its implementation which when rendered, looks visually the same as the input image. To achieve this, we take a black box rendering engine and a set of attributes it supports (e.g., colors, border radius, shadow or text properties), use it to generate a suitable synthetic training dataset, and then train specialized neural models to predict each of the attribute values. To improve pixel-level accuracy, we additionally use imitation learning to train a neural policy that refines the predicted attribute values by learning to compute the similarity of the original and rendered images in their attribute space, rather than based on the difference of pixel values. We instantiate our approach to the task of inferring Android Button attribute values and achieve 92.5 dataset consisting of real-world Google Play Store applications.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 14

page 16

page 17

06/01/2017

Fader Networks: Manipulating Images by Sliding Attributes

This paper introduces a new encoder-decoder architecture that is trained...
12/28/2019

MulGAN: Facial Attribute Editing by Exemplar

Recent studies on face attribute editing by exemplars have achieved prom...
10/22/2021

Multi-attribute Pizza Generator: Cross-domain Attribute Control with Conditional StyleGAN

Multi-attribute conditional image generation is a challenging problem in...
05/22/2017

pix2code: Generating Code from a Graphical User Interface Screenshot

Transforming a graphical user interface screenshot created by a designer...
01/01/1997

Improved Heterogeneous Distance Functions

Instance-based learning techniques typically handle continuous and linea...
05/16/2020

Attribute2Font: Creating Fonts You Want From Attributes

Font design is now still considered as an exclusive privilege of profess...
12/18/2019

Simulating Content Consistent Vehicle Datasets with Attribute Descent

We simulate data using a graphic engine to augment real-world datasets, ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.