All You Need Is Logs: Improving Code Completion by Learning from Anonymous IDE Usage Logs

05/21/2022
by   Vitaliy Bibaev, et al.
0

Integrated Development Environments (IDE) are designed to make users more productive, as well as to make their work more comfortable. To achieve this, a lot of diverse tools are embedded into IDEs, and the developers of IDEs can employ anonymous usage logs to collect the data about how they are being used to improve them. A particularly important component that this can be applied to is code completion, since improving code completion using statistical learning techniques is a well-established research area. In this work, we propose an approach for collecting completion usage logs from the users in an IDE and using them to train a machine learning based model for ranking completion candidates. We developed a set of features that describe completion candidates and their context, and deployed their anonymized collection in the Early Access Program of IntelliJ-based IDEs. We used the logs to collect a dataset of code completions from users, and employed it to train a ranking CatBoost model. Then, we evaluated it in two settings: on a held-out set of the collected completions and in a separate A/B test on two different groups of users in the IDE. Our evaluation shows that using a simple ranking model trained on the past user behavior logs significantly improved code completion experience. Compared to the default heuristics-based ranking, our model demonstrated a decrease in the number of typing actions necessary to perform the completion in the IDE from 2.073 to 1.832. The approach adheres to privacy requirements and legal constraints, since it does not require collecting personal information, performing all the necessary anonymization on the client's side. Importantly, it can be improved continuously: implementing new features, collecting new data, and evaluating new models - this way, we have been using it in production since the end of 2020.

READ FULL TEXT

page 2

page 3

page 4

page 5

page 8

page 9

page 10

page 11

research
06/26/2021

Toward Less Hidden Cost of Code Completion with Acceptance and Ranking Models

Code completion is widely used by software developers to provide coding ...
research
06/21/2018

Par4Sim -- Adaptive Paraphrasing for Text Simplification

Learning from a real-world data stream and continuously updating the mod...
research
07/24/2019

Counterfactual Learning from Logs for Improved Ranking of E-Commerce Products

Improved search quality enhances users' satisfaction, which directly imp...
research
05/21/2021

De-Biased Modelling of Search Click Behavior with Reinforcement Learning

Users' clicks on Web search results are one of the key signals for evalu...
research
04/12/2012

Enabling Semantic Analysis of User Browsing Patterns in the Web of Data

A useful step towards better interpretation and analysis of the usage pa...
research
04/20/2021

GDPR-Compliant Use of Blockchain for Secure Usage Logs

The unique properties of blockchain enable central requirements of distr...
research
12/07/2022

Towards using Few-Shot Prompt Learning for Automating Model Completion

We propose a simple yet a novel approach to improve completion in domain...

Please sign up or login with your details

Forgot password? Click here to reset