DeepAI AI Chat
Log In Sign Up

A Comparison of Code Embeddings and Beyond

by   Siqi Han, et al.
East China Normal University

Program representation learning is a fundamental task in software engineering applications. With the availability of "big code" and the development of deep learning techniques, various program representation learning models have been proposed to understand the semantic properties of programs and applied on different software engineering tasks. However, no previous study has comprehensively assessed the generalizability of these deep models on different tasks, so that the pros and cons of the models are unclear. In this experience paper, we try to bridge this gap by systemically evaluating the performance of eight program representation learning models on three common tasks, where six models are based on abstract syntax trees and two models are based on plain text of source code. We kindly explain the criteria for selecting the models and tasks, as well as the method for enabling end-to-end learning in each task. The results of performance evaluation show that they perform diversely in each task and the performance of the AST-based models is generally unstable over different tasks. In order to further explain the results, we apply a prediction attribution technique to find what elements are captured by the models and responsible for the predictions in each task. Based on the findings, we discuss some general principles for better capturing the information in the source code, and hope to inspire researchers to improve program representation learning methods for software engineering tasks.


page 18

page 19


Adding Context to Source Code Representations for Deep Learning

Deep learning models have been successfully applied to a variety of soft...

Authorship Attribution of Source Code: A Language-Agnostic Approach and Applicability in Software Engineering

Authorship attribution of source code has been an established research t...

Evaluating the Impact of Source Code Parsers on ML4SE Models

As researchers and practitioners apply Machine Learning to increasingly ...

Multimodal Deep Learning for Flaw Detection in Software Programs

We explore the use of multiple deep learning models for detecting flaws ...

IR2Vec: A Flow Analysis based Scalable Infrastructure for Program Encodings

We propose IR2Vec, a Concise and Scalable encoding infrastructure to rep...

Software Defect Prediction Based On Deep Learning Models: Performance Study

In recent years, defect prediction, one of the major software engineerin...

Explainable AI for Pre-Trained Code Models: What Do They Learn? When They Do Not Work?

In recent years, there has been a wide interest in designing deep neural...