Play to Grade: Testing Coding Games as Classifying Markov Decision Process

10/27/2021
by   Allen Nie, et al.
0

Contemporary coding education often presents students with the task of developing programs that have user interaction and complex dynamic systems, such as mouse based games. While pedagogically compelling, there are no contemporary autonomous methods for providing feedback. Notably, interactive programs are impossible to grade by traditional unit tests. In this paper we formalize the challenge of providing feedback to interactive programs as a task of classifying Markov Decision Processes (MDPs). Each student's program fully specifies an MDP where the agent needs to operate and decide, under reasonable generalization, if the dynamics and reward model of the input MDP should be categorized as correct or broken. We demonstrate that by designing a cooperative objective between an agent and an autoregressive model, we can use the agent to sample differential trajectories from the input MDP that allows a classifier to determine membership: Play to Grade. Our method enables an automatic feedback system for interactive code assignments. We release a dataset of 711,274 anonymized student submissions to a single assignment with hand-coded bug labels to support future research.

READ FULL TEXT

page 7

page 13

page 14

page 15

research
11/16/2022

Giving Feedback on Interactive Student Programs with Meta-Exploration

Developing interactive software, such as websites or games, is a particu...
research
07/10/2019

Markov Decision Process for MOOC users behavioral inference

Studies on massive open online courses (MOOCs) users discuss the existen...
research
08/15/2013

Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizations

Control applications often feature tasks with similar, but not identical...
research
11/17/2020

REALab: An Embedded Perspective on Tampering

This paper describes REALab, a platform for embedded agency research in ...
research
11/19/2021

Expert-Guided Symmetry Detection in Markov Decision Processes

Learning a Markov Decision Process (MDP) from a fixed batch of trajector...
research
06/10/2022

Conformal Prediction Intervals for Markov Decision Process Trajectories

Before delegating a task to an autonomous system, a human operator may w...
research
01/15/2021

Automating Program Structure Classification

When students write programs, their program structure provides insight i...

Please sign up or login with your details

Forgot password? Click here to reset