Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications

by   Ritam Jyoti Sarmah, et al.
Carnegie Mellon University

Supporting voice commands in applications presents significant benefits to users. However, adding such support to existing GUI-based web apps is effort-consuming with a high learning barrier, as shown in our formative study, due to the lack of unified support for creating multimodal interfaces. We present Geno—a developer tool for adding the voice input modality to existing web apps without requiring significant NLP expertise. Geno provides a high-level workflow for developers to specify functionalities to be supported by voice (intents), create language models for detecting intents and the relevant information (parameters) from user utterances, and fulfill the intents by either programmatically invoking the corresponding functions or replaying GUI actions on the web app. Geno further supports multimodal references to GUI context in voice commands (e.g. "move this [event] to next week" while pointing at an event with the cursor). In a study, developers with little NLP expertise were able to add multimodal voice command support for two existing web apps using Geno.


page 2

page 6

page 7

page 10


Measuring the Effectiveness of Privacy Policies for Voice Assistant Applications

Voice Assistants (VA) such as Amazon Alexa and Google Assistant are quic...

HIVA: Holographic Intellectual Voice Assistant

Holographic Intellectual Voice Assistant (HIVA) aims to facilitate human...

Consent on the Fly: Developing Ethical Verbal Consent for Voice Assistants

Determining how voice assistants should broker consent to share data wit...

SkillBot: Identifying Risky Content for Children in Alexa Skills

Many households include children who use voice personal assistants (VPA)...

Threshy: Supporting Safe Usage of Intelligent Web Services

Increased popularity of `intelligent' web services provides end-users wi...

Software Product Line for Metaverse: Preliminary Results

The Metaverse is a network of eXtended Reality applications (XR apps) co...

Please sign up or login with your details

Forgot password? Click here to reset