DeepAI AI Chat
Log In Sign Up

Can OpenAI Codex and Other Large Language Models Help Us Fix Security Bugs?

by   Hammond Pearce, et al.

Human developers can produce code with cybersecurity weaknesses. Can emerging 'smart' code completion tools help repair those weaknesses? In this work, we examine the use of large language models (LLMs) for code (such as OpenAI's Codex and AI21's Jurassic J-1) for zero-shot vulnerability repair. We investigate challenges in the design of prompts that coax LLMs into generating repaired versions of insecure code. This is difficult due to the numerous ways to phrase key information – both semantically and syntactically – with natural languages. By performing a large scale study of four commercially available, black-box, "off-the-shelf" LLMs, as well as a locally-trained model, on a mix of synthetic, hand-crafted, and real-world security bug scenarios, our experiments show that LLMs could collectively repair 100 generated and hand-crafted scenarios, as well as 58 selection of historical bugs in real-world open-source projects.


page 1

page 6

page 8

page 12

page 16


Fixing Hardware Security Bugs with Large Language Models

Novel AI-based code-writing Large Language Models (LLMs) such as OpenAI'...

Can we learn from developer mistakes? Learning to localize and repair real bugs from real bug fixes

Real bug fixes found in open source repositories seem to be the perfect ...

Systematically Finding Security Vulnerabilities in Black-Box Code Generation Models

Recently, large language models for code generation have achieved breakt...

Enabling Automatic Repair of Source Code Vulnerabilities Using Data-Driven Methods

Users around the world rely on software-intensive systems in their day-t...

Large Language Models and Simple, Stupid Bugs

With the advent of powerful neural language models, AI-based systems to ...

FLAG: Finding Line Anomalies (in code) with Generative AI

Code contains security and functional bugs. The process of identifying a...

An Empirical Cybersecurity Evaluation of GitHub Copilot's Code Contributions

There is burgeoning interest in designing AI-based systems to assist hum...