AI Chat AI Image Generator AI Video Text to Speech

Large Language Models in Introductory Programming Education: ChatGPT's Performance and Implications for Assessments

08/15/2023

∙

by Natalie Kiesler, et al.

∙

∙

This paper investigates the performance of the Large Language Models (LLMs) ChatGPT-3.5 and GPT-4 in solving introductory programming tasks. Based on the performance, implications for didactic scenarios and assessment formats utilizing LLMs are derived. For the analysis, 72 Python tasks for novice programmers were selected from the free site CodingBat. Full task descriptions were used as input to the LLMs, while the generated replies were evaluated using CodingBat's unit tests. In addition, the general availability of textual explanations and program code was analyzed. The results show high scores of 94.4 to 95.8 explanations and program code, which opens new ways to incorporate LLMs into programming education and assessment.

Natalie Kiesler
4 publications
Daniel Schiffner
1 publication

research

∙ 06/03/2022

Automatic Generation of Programming Exercises and Code Explanations using Large Language Models

This article explores the natural language generation capabilities of la...

0 Sami Sarsa, et al. ∙

research

∙ 12/09/2022

Automatically Generating CS Learning Materials with Large Language Models

Recent breakthroughs in Large Language Models (LLMs), such as GPT-3 and ...

0 Stephen MacNeil, et al. ∙

research

∙ 11/27/2019

Automated Generation of Computer Graded Unit Testing-Based Programming Assessments for Education

Automatic assessment of code, in particular to support education, is an ...

0 Sébastien Combéfis, et al. ∙

research

∙ 06/29/2023

Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors

Generative AI and large language models hold great promise in enhancing ...

0 Tung Phung, et al. ∙

research

∙ 07/30/2023

Evaluating ChatGPT and GPT-4 for Visual Programming

Generative AI and large language models have the potential to drasticall...

0 Adish Singla, et al. ∙

research

∙ 06/09/2023

Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests

Background and Context: Over the past year, large language models (LLMs)...

0 Arto Hellas, et al. ∙

research

∙ 06/15/2023

Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses

This paper studies recent developments in large language models' (LLM) a...

0 Jaromír Šavelka, et al. ∙