CamChoice: A Corpus of Multiple Choice Questions and Candidate Response Distributions

by   Adian Liusie, et al.

Multiple Choice examinations are a ubiquitous form of assessment that is used to measure the ability of candidates across various domains and tasks. Maintaining the quality of proposed questions is of great importance to test designers, and therefore newly proposed questions go through several pre-test evaluation stages before they can be deployed into real-world exams. This process is currently quite manual, which can lead to time lags in the question development cycle. Automating this process would lead to a large improvement in efficiency, however, current datasets do not contain sufficient pre-test analysis information. In this paper, we introduce CamChoice; a multiple-choice comprehension dataset with questions at different target levels, where questions have the true candidate selected options distributions. We introduce the task of candidate distribution matching, propose several evaluation metrics for the task, and demonstrate that automatic systems trained on RACE++ can be leveraged as baselines for our task. We further demonstrate that these automatic systems can be used for practical pre-test evaluation tasks such as detecting underperforming distractors, where our detection systems can automatically identify poor distractors that few candidates select. We release the data publicly for future research.


page 1

page 2

page 3

page 4


"World Knowledge" in Multiple Choice Reading Comprehension

Recently it has been shown that without any access to the contextual pas...

Convolutional Spatial Attention Model for Reading Comprehension with Multiple-Choice Questions

Machine Reading Comprehension (MRC) with multiple-choice questions requi...

Multiple-Choice Question Generation: Towards an Automated Assessment Framework

Automated question generation is an important approach to enable persona...

Learning to Reuse Distractors to support Multiple Choice Question Generation in Education

Multiple choice questions (MCQs) are widely used in digital learning sys...

NPCs Vote! Changing Voter Reactions Over Time Using the Extreme AI Personality Engine

Can non-player characters have human-realistic personalities, changing o...

Mining Precise Test Oracle Modelled by FSM

Precise test oracles for reactive systems such as critical control syste...

Exploring Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning

Multiple-choice questions (MCQs) are ubiquitous in almost all levels of ...

Please sign up or login with your details

Forgot password? Click here to reset