A Systematic Evaluation of Large Language Models of Code

02/26/2022
by   Frank F. Xu, et al.
0

Large language models (LMs) of code have recently shown tremendous promise in completing code and synthesizing code from natural language descriptions. However, the current state-of-the-art code LMs (e.g., Codex (Chen et al., 2021)) are not publicly available, leaving many questions about their model and data design decisions. We aim to fill in some of these blanks through a systematic evaluation of the largest existing models: Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot, across various programming languages. Although Codex itself is not open-source, we find that existing open-source models do achieve close results in some programming languages, although targeted mainly for natural language modeling. We further identify an important missing piece in the form of a large open-source model trained exclusively on a multi-lingual corpus of code. We release a new model, PolyCoder, with 2.7B parameters based on the GPT-2 architecture, which was trained on 249GB of code across 12 programming languages on a single machine. In the C programming language, PolyCoder outperforms all models including Codex. Our trained models are open-source and publicly available at https://github.com/VHellendoorn/Code-LMs, which enables future research and application in this area.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/01/2021

The Comprehensive Blub Archive Network: Towards Design Principals for Open Source Programming Language Repositories

Many popular open source programming languages (Perl, Ruby or Python for...
research
07/27/2023

Multilingual Code Co-Evolution Using Large Language Models

Many software projects implement APIs and algorithms in multiple program...
research
08/22/2023

Halo: Estimation and Reduction of Hallucinations in Open-Source Weak Large Language Models

Large Language Models (LLMs) have revolutionized Natural Language Proces...
research
09/11/2023

OpenFashionCLIP: Vision-and-Language Contrastive Learning with Open-Source Fashion Data

The inexorable growth of online shopping and e-commerce demands scalable...
research
04/08/2023

ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application

This paper demonstrates how OpenAI's ChatGPT can be used in a few-shot s...
research
04/12/2023

Programming Language Assisted Waveform Analysis: A Case Study on the Instruction Performance of SERV

RISC-Vs growing traction leads to the release of new RISC-V cores on a n...
research
09/04/2023

LLM and Infrastructure as a Code use case

Cloud computing and the evolution of management methodologies such as Le...

Please sign up or login with your details

Forgot password? Click here to reset