The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues

05/16/2022
by   Anaïs Tack, et al.
0

How can we test whether state-of-the-art generative models, such as Blender and GPT-3, are good AI teachers, capable of replying to a student in an educational dialogue? Designing an AI teacher test is challenging: although evaluation methods are much-needed, there is no off-the-shelf solution to measuring pedagogical ability. This paper reports on a first attempt at an AI teacher test. We built a solution around the insight that you can run conversational agents in parallel to human teachers in real-world dialogues, simulate how different agents would respond to a student, and compare these counterpart responses in terms of three abilities: speak like a teacher, understand a student, help a student. Our method builds on the reliability of comparative judgments in education and uses a probabilistic model and Bayesian sampling to infer estimates of pedagogical ability. We find that, even though conversational agents (Blender in particular) perform well on conversational uptake, they are quantifiably worse than real teachers on several pedagogical dimensions, especially with regard to helpfulness (Blender: Δ ability = -0.75; GPT-3: Δ ability = -0.93).

READ FULL TEXT
research
06/12/2023

The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues

This paper describes the results of the first shared task on the generat...
research
06/07/2021

Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions

In conversation, uptake happens when a speaker builds on the contributio...
research
06/08/2023

The ADAIO System at the BEA-2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues

This paper presents the ADAIO team's system entry in the Building Educat...
research
06/15/2023

Can Language Models Teach Weaker Agents? Teacher Explanations Improve Students via Theory of Mind

Large Language Models (LLMs) perform complex reasoning by generating exp...
research
11/13/2020

The Teacher-Student Chatroom Corpus

The Teacher-Student Chatroom Corpus (TSCC) is a collection of written co...
research
05/15/2020

Siamese Neural Networks for Class Activity Detection

Classroom activity detection (CAD) aims at accurately recognizing speake...
research
12/02/2021

Improving mathematical questioning in teacher training

High-fidelity, AI-based simulated classroom systems enable teachers to r...

Please sign up or login with your details

Forgot password? Click here to reset