Log In Sign Up

Psychologically-Inspired, Unsupervised Inference of Perceptual Groups of GUI Widgets from GUI Images

by   Mulong Xie, et al.

Graphical User Interface (GUI) is not merely a collection of individual and unrelated widgets, but rather partitions discrete widgets into groups by various visual cues, thus forming higher-order perceptual units such as tab, menu, card or list. The ability to automatically segment a GUI into perceptual groups of widgets constitutes a fundamental component of visual intelligence to automate GUI design, implementation and automation tasks. Although humans can partition a GUI into meaningful perceptual groups of widgets in a highly reliable way, perceptual grouping is still an open challenge for computational approaches. Existing methods rely on ad-hoc heuristics or supervised machine learning that is dependent on specific GUI implementations and runtime information. Research in psychology and biological vision has formulated a set of principles (i.e., Gestalt theory of perception) that describe how humans group elements in visual scenes based on visual cues like connectivity, similarity, proximity and continuity. These principles are domain-independent and have been widely adopted by practitioners to structure content on GUIs to improve aesthetic pleasant and usability. Inspired by these principles, we present a novel unsupervised image-based method for inferring perceptual groups of GUI widgets. Our method requires only GUI pixel images, is independent of GUI implementation, and does not require any training data. The evaluation on a dataset of 1,091 GUIs collected from 772 mobile apps and 20 UI design mockups shows that our method significantly outperforms the state-of-the-art ad-hoc heuristics-based baseline. Our perceptual grouping method creates the opportunities for improving UI-related software engineering tasks.


page 1

page 6

page 9


Disentangling neural mechanisms for perceptual grouping

Forming perceptual groups and individuating objects in visual scenes is ...

Principles of perceptual grouping: implications for image-guided surgery

Gestalt theory has provided perceptual science with a conceptual framewo...

Contour-guided Image Completion with Perceptual Grouping

Humans are excellent at perceiving illusory outlines. We are readily abl...

UI Layers Group Detector: Grouping UI Layers via Text Fusion and Box Attention

Graphic User Interface (GUI) is facing great demand with the popularizat...

Tagger: Deep Unsupervised Perceptual Grouping

We present a framework for efficient perceptual inference that explicitl...

ReGroup: Recursive Neural Networks for Hierarchical Grouping of Vector Graphic Primitives

Selection functionality is as fundamental to vector graphics as it is fo...

Spatial Mixture Models with Learnable Deep Priors for Perceptual Grouping

Humans perceive the seemingly chaotic world in a structured and composit...