Is this Change the Answer to that Problem? Correlating Descriptions of Bug and Code Changes for Evaluating Patch Correctness

by   Haoye Tian, et al.

In this work, we propose a novel perspective to the problem of patch correctness assessment: a correct patch implements changes that "answer" to a problem posed by buggy behaviour. Concretely, we turn the patch correctness assessment into a Question Answering problem. To tackle this problem, our intuition is that natural language processing can provide the necessary representations and models for assessing the semantic correlation between a bug (question) and a patch (answer). Specifically, we consider as inputs the bug reports as well as the natural language description of the generated patches. Our approach, Quatrain, first considers state of the art commit message generation models to produce the relevant inputs associated to each generated patch. Then we leverage a neural network architecture to learn the semantic correlation between bug reports and commit messages. Experiments on a large dataset of 9135 patches generated for three bug datasets (Defects4j, Bugs.jar and Bears) show that Quatrain can achieve an AUC of 0.886 on predicting patch correctness, and recalling 93 incorrect patches. Our experimental results further demonstrate the influence of inputs quality on prediction performance. We further perform experiments to highlight that the model indeed learns the relationship between bug reports and code change descriptions for the prediction. Finally, we compare against prior work and discuss the benefits of our approach.


page 1

page 2

page 3

page 4


A Recurrent Neural Network Based Patch Recommender for Linux Kernel Bugs

Software bugs in a production environment have an undesirable impact on ...

CC2Vec: Distributed Representations of Code Changes

Existing work on software patches often use features specific to a singl...

Towards Generating Functionally Correct Code Edits from Natural Language Issue Descriptions

Large language models (LLMs), such as OpenAI's Codex, have demonstrated ...

Checking Patch Behaviour against Test Specification

Towards predicting patch correctness in APR, we propose a simple, but no...

Evaluating Representation Learning of Code Changes for Predicting Patch Correctness in Program Repair

A large body of the literature of automated program repair develops appr...

Fixing Model Bugs with Natural Language Patches

Current approaches for fixing systematic problems in NLP models (e.g. re...

PatchZero: Zero-Shot Automatic Patch Correctness Assessment

Automated Program Repair (APR) techniques have shown more and more promi...

Please sign up or login with your details

Forgot password? Click here to reset