NLU for Game-based Learning in Real: Initial Evaluations

05/27/2022
by   Eda Okur, et al.
0

Intelligent systems designed for play-based interactions should be contextually aware of the users and their surroundings. Spoken Dialogue Systems (SDS) are critical for these interactive agents to carry out effective goal-oriented communication with users in real-time. For the real-world (i.e., in-the-wild) deployment of such conversational agents, improving the Natural Language Understanding (NLU) module of the goal-oriented SDS pipeline is crucial, especially with limited task-specific datasets. This study explores the potential benefits of a recently proposed transformer-based multi-task NLU architecture, mainly to perform Intent Recognition on small-size domain-specific educational game datasets. The evaluation datasets were collected from children practicing basic math concepts via play-based interactions in game-based learning settings. We investigate the NLU performances on the initial proof-of-concept game datasets versus the real-world deployment datasets and observe anticipated performance drops in-the-wild. We have shown that compared to the more straightforward baseline approaches, Dual Intent and Entity Transformer (DIET) architecture is robust enough to handle real-world data to a large extent for the Intent Recognition task on these domain-specific in-the-wild game datasets.

READ FULL TEXT
research
05/09/2022

Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System

Contextually aware intelligent agents are often required to understand t...
research
11/07/2022

End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics

The advances in language-based Artificial Intelligence (AI) technologies...
research
06/01/2023

Inspecting Spoken Language Understanding from Kids for Basic Math Learning at Home

Enriching the quality of early childhood education with interactive math...
research
09/29/2020

HINT3: Raising the bar for Intent Detection in the Wild

Intent Detection systems in the real world are exposed to complexities o...
research
12/20/2019

Modeling Intent, Dialog Policies and Response Adaptation for Goal-Oriented Interactions

Building a machine learning driven spoken dialog system for goal-oriente...
research
05/22/2023

clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents

Recent work has proposed a methodology for the systematic evaluation of ...
research
05/02/2017

Chat Detection in an Intelligent Assistant: Combining Task-oriented and Non-task-oriented Spoken Dialogue Systems

Recently emerged intelligent assistants on smartphones and home electron...

Please sign up or login with your details

Forgot password? Click here to reset