Evaluating Large Language Models Trained on Code

07/07/2021
by   Mark Chen, et al.
6

We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8 solves 0 from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2 problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

READ FULL TEXT

page 1

page 2

page 3

page 4

05/09/2023

StarCoder: may the source be with you!

The BigCode community, an open-scientific collaboration working on the r...
07/28/2023

VeriGen: A Large Language Model for Verilog Code Generation

In this study, we explore the capability of Large Language Models (LLMs)...
08/20/2022

Security Implications of Large Language Model Code Assistants: A User Study

Advances in Deep Learning have led to the emergence of Large Language Mo...
08/31/2023

BioCoder: A Benchmark for Bioinformatics Code Generation with Contextual Pragmatic Knowledge

Pre-trained language models like ChatGPT have significantly improved cod...
07/25/2022

A Hazard Analysis Framework for Code Synthesis Large Language Models

Codex, a large language model (LLM) trained on a variety of codebases, e...
08/20/2021

An Empirical Cybersecurity Evaluation of GitHub Copilot's Code Contributions

There is burgeoning interest in designing AI-based systems to assist hum...
03/31/2023

Pair Programming with Large Language Models for Sampling and Estimation of Copulas

Without writing a single line of code by a human, an example Monte Carlo...

Code Repositories

PythonProgrammingPuzzles

A Dataset of Python Challenges for AI Research


view repo

human-eval

Code for the paper "Evaluating Large Language Models Trained on Code"


view repo

gpt-code-clippy

Full description can be found here: https://discuss.huggingface.co/t/pretrain-gpt-neo-for-open-source-github-copilot-model/7678?u=ncoop57


view repo

Please sign up or login with your details

Forgot password? Click here to reset