An Empirical Study on the Usage of BERT Models for Code Completion

by   Matteo Ciniselli, et al.

Code completion is one of the main features of modern Integrated Development Environments (IDEs). Its objective is to speed up code writing by predicting the next code token(s) the developer is likely to write. Research in this area has substantially bolstered the predictive performance of these techniques. However, the support to developers is still limited to the prediction of the next few tokens to type. In this work, we take a step further in this direction by presenting a large-scale empirical study aimed at exploring the capabilities of state-of-the-art deep learning (DL) models in supporting code completion at different granularity levels, including single tokens, one or multiple entire statements, up to entire code blocks (e.g., the iterated block of a for loop). To this aim, we train and test several adapted variants of the recently proposed RoBERTa model, and evaluate its predictions from several perspectives, including: (i) metrics usually adopted when assessing DL generative models (i.e., BLEU score and Levenshtein distance); (ii) the percentage of perfect predictions (i.e., the predicted code snippets that match those written by developers); and (iii) the "semantic" equivalence of the generated code as compared to the one written by developers. The achieved results show that BERT models represent a viable solution for code completion, with perfect predictions ranging from  7 blocks, up to  58 the same code statement.



There are no comments yet.


page 1

page 7


An Empirical Study on the Usage of Transformer Models for Code Completion

Code completion aims at speeding up code writing by predicting the next ...

An Empirical Study on Code Comment Completion

Code comments play a prominent role in program comprehension activities....

Towards Full-line Code Completion with Neural Language Models

A code completion system suggests future code elements to developers giv...

Siri, Write the Next Method

Code completion is one of the killer features of Integrated Development ...

Fast and Memory-Efficient Neural Code Completion

Code completion is one of the most widely used features of modern integr...

Toward Less Hidden Cost of Code Completion with Acceptance and Ranking Models

Code completion is widely used by software developers to provide coding ...

SEED: Semantic Graph based Deep detection for type-4 clone

Background: Type-4 clones refer to a pair of code snippets with similar ...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.