Towards Continuous Compounding Effects and Agile Practices in Educational Experimentation

by   Luis M. Vaquero, et al.

Randomised control trials are currently the definitive gold standard approach for formal educational experiments. Although conclusions from these experiments are highly credible, their relatively slow experimentation rate, high expense and rigid framework can be seen to limit scope on: 1. metrics: automation of the consistent rigorous computation of hundreds of metrics for every experiment; 2. concurrency: fast automated releases of hundreds of concurrent experiments daily; and 3. safeguards: safety net tests and ramping up/rolling back treatments quickly to minimise negative impact. This paper defines a framework for categorising different experimental processes, and places a particular emphasis on technology readiness. On the basis of our analysis, our thesis is that the next generation of education technology successes will be heralded by recognising the context of experiments and collectively embracing the full set of processes that are at hand: from rapid ideation and prototyping produced in small scale experiments on the one hand, to influencing recommendations of best teaching practices with large-scale and technology-enabled online A/B testing on the other. A key benefit of the latter is that the running costs tend towards zero (leading to `free experimentation'). This offers low-risk opportunities to explore and drive value though well-planned lasting campaigns that iterate quickly at a large scale. Importantly, because these experimental platforms are so adaptable, the cumulative effect of the experimental campaign delivers compounding value exponentially over time even if each individual experiment delivers a small effect.


Assign Experiment Variants at Scale in Online Controlled Experiments

Online controlled experiments (A/B tests) have become the gold standard ...

Experimenting with Experimentation: Rethinking The Role of Experimentation in Educational Design

What if we take a broader view of what it means to run an education expe...

Quantifying the Value of Iterative Experimentation

Over the past decade, most technology companies and a growing number of ...

Ensure A/B Test Quality at Scale with Automated Randomization Validation and Sample Ratio Mismatch Detection

eBay's experimentation platform runs hundreds of A/B tests on any given ...

Learning Analytics in Massive Open Online Courses

Educational technology has obtained great importance over the last fifte...

Clickbait in Education Positive or Negative? Machine Learning Answers

The topic of clickbait has garnered lot of attention since the advent of...

Trustworthy Experimentation Under Telemetry Loss

Failure to accurately measure the outcomes of an experiment can lead to ...

Please sign up or login with your details

Forgot password? Click here to reset