An Empirical Study on the Usage of Transformer Models for Code Completion

08/03/2021
by   Matteo Ciniselli, et al.
0

Code completion aims at speeding up code writing by predicting the next code token(s) the developer is likely to write. Works in this field focused on improving the accuracy of the generated predictions, with substantial leaps forward made possible by deep learning (DL) models. However, code completion techniques are mostly evaluated in the scenario of predicting the next token to type, with few exceptions pushing the boundaries to the prediction of an entire code statement. Thus, little is known about the performance of state-of-the-art code completion approaches in more challenging scenarios in which, for example, an entire code block must be generated. We present a large-scale study exploring the capabilities of state-of-the-art Transformer-based models in supporting code completion at different granularity levels, including single tokens, one or multiple entire statements, up to entire code blocks (e.g., the iterated block of a for loop). We experimented with several variants of two recently proposed Transformer-based models, namely RoBERTa and the Text-To-Text Transfer Transformer (T5), for the task of code completion. The achieved results show that Transformer-based models, and in particular the T5, represent a viable solution for code completion, with perfect predictions ranging from  29 reached in the simpler scenario of few tokens masked from the same code statement.

READ FULL TEXT

page 9

page 11

page 18

research
03/12/2021

An Empirical Study on the Usage of BERT Models for Code Completion

Code completion is one of the main features of modern Integrated Develop...
research
07/09/2022

A Closer Look into Transformer-Based Code Intelligence Through Code Transformation: Challenges and Opportunities

Transformer-based models have demonstrated state-of-the-art performance ...
research
01/10/2023

Practitioners' Expectations on Code Completion

Code completion has become a common practice for programmers during thei...
research
04/21/2022

Non-autoregressive Model for Full-line Code Completion

Code completion tools are frequently used by software developers to acce...
research
11/09/2022

Syntax-Aware On-the-Fly Code Completion

Code completion aims to help improve developers' productivity by suggest...
research
03/08/2021

Siri, Write the Next Method

Code completion is one of the killer features of Integrated Development ...
research
07/22/2020

DeepClone: Modeling Clones to Generate Code Predictions

During software development, programmers often tend to reuse the code fo...

Please sign up or login with your details

Forgot password? Click here to reset