Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?

03/16/2023
by   Jaromír Šavelka, et al.
0

We evaluated the capability of generative pre-trained transformers (GPT), to pass assessments in introductory and intermediate Python programming courses at the postsecondary level. Discussions of potential uses (e.g., exercise generation, code explanation) and misuses (e.g., cheating) of this emerging technology in programming education have intensified, but to date there has not been a rigorous analysis of the models' capabilities in the realistic context of a full-fledged programming course with diverse set of assessment instruments. We evaluated GPT on three Python courses that employ assessments ranging from simple multiple-choice questions (no code involved) to complex programming projects with code bases distributed into multiple files (599 exercises overall). Further, we studied if and how successfully GPT models leverage feedback provided by an auto-grader. We found that the current models are not capable of passing the full spectrum of assessments typically involved in a Python programming course (<70 clear that a straightforward application of these easily accessible models could enable a learner to obtain a non-trivial portion of the overall available score (>55 exhibit remarkable capabilities, including correcting solutions based on auto-grader's feedback, some limitations exist (e.g., poor handling of exercises requiring complex chains of reasoning steps). These findings can be leveraged by instructors wishing to adapt their assessments so that GPT becomes a valuable assistant for a learner as opposed to an end-to-end solution.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
06/15/2023

Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses

This paper studies recent developments in large language models' (LLM) a...
research
03/09/2023

Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code

We analyzed effectiveness of three generative pre-trained transformer (G...
research
07/30/2023

Evaluating ChatGPT and GPT-4 for Visual Programming

Generative AI and large language models have the potential to drasticall...
research
04/29/2023

Can ChatGPT Pass An Introductory Level Functional Language Programming Course?

The recent introduction of ChatGPT has drawn significant attention from ...
research
06/30/2023

Harnessing LLMs in Curricular Design: Using GPT-4 to Support Authoring of Learning Objectives

We evaluated the capability of a generative pre-trained transformer (GPT...
research
03/07/2023

Many bioinformatics programming tasks can be automated with ChatGPT

Computer programming is a fundamental tool for life scientists, allowing...
research
07/10/2023

ChatGPT for Digital Forensic Investigation: The Good, The Bad, and The Unknown

The disruptive application of ChatGPT (GPT-3.5, GPT-4) to a variety of d...

Please sign up or login with your details

Forgot password? Click here to reset