Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators

by   Paul Denny, et al.

With their remarkable ability to generate code, large language models (LLMs) are a transformative technology for computing education practice. They have created an urgent need for educators to rethink pedagogical approaches and teaching strategies for newly emerging skill sets. Traditional approaches to learning programming have focused on frequent and repeated practice at writing code. The ease with which code can now be generated has resulted in a shift in focus towards reading, understanding and evaluating LLM-generated code. In parallel with this shift, a new essential skill is emerging – the ability to construct good prompts for code-generating models. This paper introduces a novel pedagogical concept known as a `Prompt Problem', designed to help students learn how to craft effective prompts for LLMs. A Prompt Problem challenges a student to create a natural language prompt that leads an LLM to produce the correct code for a specific problem. To support the delivery of Prompt Problems at scale, in this paper we also present a novel tool called Promptly which hosts a repository of Prompt Problems and automates the evaluation of prompt-generated code. We report empirical findings from a field study in which Promptly was deployed in a first-year Python programming course (n=54). We explore student interactions with the tool and their perceptions of the Prompt Problem concept. We found that Promptly was largely well-received by students for its ability to engage their computational thinking skills and expose them to new programming constructs. We also discuss avenues for future work, including variations on the design of Prompt Problems and the need to study their integration into the curriculum and teaching practice.


page 5

page 6

page 9


Computing Education in the Era of Generative AI

The computing education community has a rich history of pedagogical inno...

CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes

Computing educators face significant challenges in providing timely supp...

Comparing Code Explanations Created by Students and Large Language Models

Reasoning about code and explaining its purpose are fundamental skills f...

"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers

Recent developments in deep learning have resulted in code-generation mo...

Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language

GitHub Copilot is an artificial intelligence model for automatically gen...

StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code

Code LLMs are being rapidly deployed and there is evidence that they can...

Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests

Background and Context: Over the past year, large language models (LLMs)...

Please sign up or login with your details

Forgot password? Click here to reset