How Effective Are Neural Networks for Fixing Security Vulnerabilities

05/29/2023
by   Yi Wu, et al.
0

Security vulnerability repair is a difficult task that is in dire need of automation. Two groups of techniques have shown promise: (1) large code language models (LLMs) that have been pre-trained on source code for tasks such as code completion, and (2) automated program repair (APR) techniques that use deep learning (DL) models to automatically fix software bugs. This paper is the first to study and compare Java vulnerability repair capabilities of LLMs and DL-based APR models. The contributions include that we (1) apply and evaluate five LLMs (Codex, CodeGen, CodeT5, PLBART and InCoder), four fine-tuned LLMs, and four DL-based APR techniques on two real-world Java vulnerability benchmarks (Vul4J and VJBench), (2) design code transformations to address the training and test data overlapping threat to Codex, (3) create a new Java vulnerability repair benchmark VJBench, and its transformed version VJBench-trans and (4) evaluate LLMs and APR techniques on the transformed vulnerabilities in VJBench-trans. Our findings include that (1) existing LLMs and APR models fix very few Java vulnerabilities. Codex fixes 10.2 (20.4 (2) Fine-tuning with general APR data improves LLMs' vulnerability-fixing capabilities. (3) Our new VJBench reveals that LLMs and APR models fail to fix many Common Weakness Enumeration (CWE) types, such as CWE-325 Missing cryptographic step and CWE-444 HTTP request smuggling. (4) Codex still fixes 8.3 transformed vulnerabilities, outperforming all the other LLMs and APR models on transformed vulnerabilities. The results call for innovations to enhance automated Java vulnerability repair such as creating larger vulnerability repair training data, tuning LLMs with such data, and applying code simplification transformation to facilitate vulnerability repair.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/07/2022

Enabling Automatic Repair of Source Code Vulnerabilities Using Data-Driven Methods

Users around the world rely on software-intensive systems in their day-t...
research
02/10/2023

Impact of Code Language Models on Automated Program Repair

Automated program repair (APR) aims to help developers improve software ...
research
10/31/2022

Poison Attack and Defense on Deep Source Code Processing Models

In the software engineering community, deep learning (DL) has recently b...
research
04/16/2021

Neural Transfer Learning for Repairing Security Vulnerabilities in C Code

In this paper, we address the problem of automatic repair of software vu...
research
08/24/2023

Pre-trained Model-based Automated Software Vulnerability Repair: How Far are We?

Various approaches are proposed to help under-resourced security researc...
research
02/24/2022

Automatically Mitigating Vulnerabilities in x86 Binary Programs via Partially Recompilable Decompilation

When vulnerabilities are discovered after software is deployed, source c...
research
07/13/2023

SecureFalcon: The Next Cyber Reasoning System for Cyber Security

Software vulnerabilities leading to various detriments such as crashes, ...

Please sign up or login with your details

Forgot password? Click here to reset