A Test for Evaluating Performance in Human-Computer Systems

by   Andres Campero, et al.

The Turing test for comparing computer performance to that of humans is well known, but, surprisingly, there is no widely used test for comparing how much better human-computer systems perform relative to humans alone, computers alone, or other baselines. Here, we show how to perform such a test using the ratio of means as a measure of effect size. Then we demonstrate the use of this test in three ways. First, in an analysis of 79 recently published experimental results, we find that, surprisingly, over half of the studies find a decrease in performance, the mean and median ratios of performance improvement are both approximately 1 (corresponding to no improvement at all), and the maximum ratio is 1.36 (a 36 higher performance improvement ratio is obtained when 100 human programmers generate software using GPT-3, a massive, state-of-the-art AI system. In this case, we find a speed improvement ratio of 1.27 (a 27 we find that 50 human non-programmers using GPT-3 can perform the task about as well as–and less expensively than–the human programmers. In this case, neither the non-programmers nor the computer would have been able to perform the task alone, so this is an example of a very strong form of human-computer synergy.


page 5

page 21


Consistent ANOVA-type tests for various effect sizes

Analysis of variance (ANOVA) reveals some disadvantages, such as non-rob...

Human or Not? A Gamified Approach to the Turing Test

We present "Human or Not?", an online game inspired by the Turing test, ...

Human or Machine? Turing Tests for Vision and Language

As AI algorithms increasingly participate in daily activities that used ...

Improvised Comedy as a Turing Test

The best improvisational theatre actors can make any scene partner, of a...

A data-driven approach for learning to control computers

It would be useful for machines to use computers as humans do so that th...

Large language models are not zero-shot communicators

Despite widespread use of LLMs as conversational agents, evaluations of ...

Deceiving computers in Reverse Turing Test through Deep Learning

It is increasingly becoming difficult for human beings to work on their ...

Please sign up or login with your details

Forgot password? Click here to reset