Log In Sign Up

Screen2Vec: Semantic Embedding of GUI Screens and GUI Components

by   Toby Jia-Jun Li, et al.

Representing the semantics of GUI screens and components is crucial to data-driven computational methods for modeling user-GUI interactions and mining GUI designs. Existing GUI semantic representations are limited to encoding either the textual content, the visual design and layout patterns, or the app contexts. Many representation techniques also require significant manual data annotation efforts. This paper presents Screen2Vec, a new self-supervised technique for generating representations in embedding vectors of GUI screens and components that encode all of the above GUI features without requiring manual annotation using the context of user interaction traces. Screen2Vec is inspired by the word embedding method Word2Vec, but uses a new two-layer pipeline informed by the structure of GUIs and interaction traces and incorporates screen- and app-specific metadata. Through several sample downstream tasks, we demonstrate Screen2Vec's key useful properties: representing between-screen similarity through nearest neighbors, composability, and capability to represent user tasks.


Enhancing Semantic Word Representations by Embedding Deeper Word Relationships

Word representations are created using analogy context-based statistics ...

Explicit Pairwise Word Interaction Modeling Improves Pretrained Transformers for English Semantic Similarity Tasks

In English semantic similarity tasks, classic word embedding-based appro...

Exploiting Behavioral Consistence for Universal User Representation

User modeling is critical for developing personalized services in indust...

Magic Layouts: Structural Prior for Component Detection in User Interface Designs

We present Magic Layouts; a method for parsing screenshots or hand-drawn...

ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces

As mobile devices are becoming ubiquitous, regularly interacting with a ...

Learning Continuous User Representations through Hybrid Filtering with doc2vec

Players in the online ad ecosystem are struggling to acquire the user da...