DeepAI AI Chat
Log In Sign Up

An Empirical Evaluation of Competitive Programming AI: A Case Study of AlphaCode

08/18/2022
by   Sila Lertbanjongngam, et al.
Nara Institute of Science and Technology
ku.th
Kasetsart University
0

AlphaCode is a code generation system for assisting software developers in solving competitive programming problems using natural language problem descriptions. Despite the advantages of the code generating system, the open source community expressed concerns about practicality and data licensing. However, there is no research investigating generated codes in terms of code clone and performance. In this paper, we conduct an empirical study to find code similarities and performance differences between AlphaCode-generated codes and human codes. The results show that (i) the generated codes from AlphaCode are similar to human codes (i.e., the average maximum similarity score is 0.56) and (ii) the generated code performs on par with or worse than the human code in terms of execution time and memory usage. Moreover, AlphaCode tends to generate more similar codes to humans for low-difficulty problems (i.e., four cases have the exact same codes). It also employs excessive nested loops and unnecessary variable declarations for high-difficulty problems, which cause low performance regarding our manual investigation. The replication package is available at https:/doi.org/10.5281/zenodo.6820681

READ FULL TEXT

page 1

page 2

page 3

page 4

10/16/2021

AugmentedCode: Examining the Effects of Natural Language Resources in Code Retrieval Models

Code retrieval is allowing software engineers to search codes through a ...
07/14/2020

Coding theory package for Macaulay2

In this Macaulay2 <cit.> package we define an object called linear code....
06/09/2020

Guiding Optimizations with Meliora: A Deep Walk down Memory Lane

Performance models can be very useful for understanding the behavior of ...
06/17/2020

Synthesizing Tasks for Block-based Programming

Block-based visual programming environments play a critical role in intr...
02/01/2023

On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot

Software engineering research has always being concerned with the improv...
03/16/2021

From Innovations to Prospects: What Is Hidden Behind Cryptocurrencies?

The great influence of Bitcoin has promoted the rapid development of blo...