DeepAI
Log In Sign Up

Psychologically-Inspired, Unsupervised Inference of Perceptual Groups of GUI Widgets from GUI Images

06/15/2022
by   Mulong Xie, et al.
0

Graphical User Interface (GUI) is not merely a collection of individual and unrelated widgets, but rather partitions discrete widgets into groups by various visual cues, thus forming higher-order perceptual units such as tab, menu, card or list. The ability to automatically segment a GUI into perceptual groups of widgets constitutes a fundamental component of visual intelligence to automate GUI design, implementation and automation tasks. Although humans can partition a GUI into meaningful perceptual groups of widgets in a highly reliable way, perceptual grouping is still an open challenge for computational approaches. Existing methods rely on ad-hoc heuristics or supervised machine learning that is dependent on specific GUI implementations and runtime information. Research in psychology and biological vision has formulated a set of principles (i.e., Gestalt theory of perception) that describe how humans group elements in visual scenes based on visual cues like connectivity, similarity, proximity and continuity. These principles are domain-independent and have been widely adopted by practitioners to structure content on GUIs to improve aesthetic pleasant and usability. Inspired by these principles, we present a novel unsupervised image-based method for inferring perceptual groups of GUI widgets. Our method requires only GUI pixel images, is independent of GUI implementation, and does not require any training data. The evaluation on a dataset of 1,091 GUIs collected from 772 mobile apps and 20 UI design mockups shows that our method significantly outperforms the state-of-the-art ad-hoc heuristics-based baseline. Our perceptual grouping method creates the opportunities for improving UI-related software engineering tasks.

READ FULL TEXT

page 1

page 6

page 9

06/04/2019

Disentangling neural mechanisms for perceptual grouping

Forming perceptual groups and individuating objects in visual scenes is ...
08/05/2018

Principles of perceptual grouping: implications for image-guided surgery

Gestalt theory has provided perceptual science with a conceptual framewo...
11/22/2021

Contour-guided Image Completion with Perceptual Grouping

Humans are excellent at perceiving illusory outlines. We are readily abl...
12/07/2022

UI Layers Group Detector: Grouping UI Layers via Text Fusion and Box Attention

Graphic User Interface (GUI) is facing great demand with the popularizat...
06/21/2016

Tagger: Deep Unsupervised Perceptual Grouping

We present a framework for efficient perceptual inference that explicitl...
11/23/2021

ReGroup: Recursive Neural Networks for Hierarchical Grouping of Vector Graphic Primitives

Selection functionality is as fundamental to vector graphics as it is fo...
02/07/2019

Spatial Mixture Models with Learnable Deep Priors for Perceptual Grouping

Humans perceive the seemingly chaotic world in a structured and composit...