Desk Organization: Effect of Multimodal Inputs on Spatial Relational Learning

by   Ryan Rowe, et al.

For robots to operate in a three dimensional world and interact with humans, learning spatial relationships among objects in the surrounding is necessary. Reasoning about the state of the world requires inputs from many different sensory modalities including vision (V) and haptics (H). We examine the problem of desk organization: learning how humans spatially position different objects on a planar surface according to organizational ”preference”. We model this problem by examining how humans position objects given multiple features received from vision and haptic modalities. However, organizational habits vary greatly between people both in structure and adherence. To deal with user organizational preferences, we add an additional modality, ”utility” (U), which informs on a particular human's perceived usefulness of a given object. Models were trained as generalized (over many different people) or tailored (per person). We use two types of models: random forests, which focus on precise multi-task classification, and Markov logic networks, which provide an easily interpretable insight into organizational habits. The models were applied to both synthetic data, which proved to be learnable when using fixed organizational constraints, and human-study data, on which the random forest achieved over 90 V}modalities,UVandHUVwere the most informative for organization. In a follow-up study, we gauged participants preference of desk organizations by a generalized random forest organization vs. by a random model. On average, participants rated the random forest models as 4.15 on a 5-point Likert scale compared to 1.84 for the random model


page 1

page 6

page 7


Interpretable Data-driven Methods for Subgrid-scale Closure in LES for Transcritical LOX/GCH4 Combustion

Many practical combustion systems such as those in rockets, gas turbines...

Seeker or Avoider? User Modeling for Inspiration Deployment in Large-Scale Ideation

People react differently to inspirations shown to them during brainstorm...

Spatially Coherent Random Forests

Spatially Coherent Random Forest (SCRF) extends Random Forest to create ...

Forest Guided Smoothing

We use the output of a random forest to define a family of local smoothe...

An application of an Embedded Model Estimator to a synthetic non-stationary reservoir model with multiple secondary variables

A method (Ember) for non-stationary spatial modelling with multiple seco...

ADD-Lib: Decision Diagrams in Practice

In the paper, we present the ADD-Lib, our efficient and easy to use fram...

Supporting User Autonomy with Multimodal Fusion to Detect when a User Needs Assistance from a Social Robot

It is crucial for any assistive robot to prioritize the autonomy of the ...