Natural Language to Code Generation in Interactive Data Science Notebooks

12/19/2022
by   Pengcheng Yin, et al.
0

Computational notebooks, such as Jupyter notebooks, are interactive computing environments that are ubiquitous among data scientists to perform data wrangling and analytic tasks. To measure the performance of AI pair programmers that automatically synthesize programs for those tasks given natural language (NL) intents from users, we build ARCADE, a benchmark of 1082 code generation problems using the pandas data analysis framework in data science notebooks. ARCADE features multiple rounds of NL-to-code problems from the same notebook. It requires a model to understand rich multi-modal contexts, such as existing notebook cells and their execution states as well as previous turns of interaction. To establish a strong baseline on this challenging task, we develop PaChiNCo, a 62B code language model (LM) for Python computational notebooks, which significantly outperforms public code LMs. Finally, we explore few-shot prompting strategies to elicit better code with step-by-step decomposition and NL explanation, showing the potential to improve the diversity and explainability of model predictions.

READ FULL TEXT

page 11

page 25

research
01/30/2022

Training and Evaluating a Jupyter Notebook Data Science Assistant

We study the feasibility of a Data Science assistant powered by a sequen...
research
04/11/2023

Teaching Large Language Models to Self-Debug

Large language models (LLMs) have achieved impressive performance on cod...
research
11/17/2022

Execution-based Evaluation for Data Science Code Generation Models

Code generation models can benefit data scientists' productivity by auto...
research
07/16/2018

Teaching machines to understand data science code by semantic enrichment of dataflow graphs

Your computer is continuously executing programs, but does it really und...
research
08/20/2021

An Empirical Cybersecurity Evaluation of GitHub Copilot's Code Contributions

There is burgeoning interest in designing AI-based systems to assist hum...
research
06/26/2023

InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback

Humans write code in a fundamentally interactive manner and rely on cons...
research
10/15/2021

A Static Analysis Framework for Data Science Notebooks

Notebooks provide an interactive environment for programmers to develop ...

Please sign up or login with your details

Forgot password? Click here to reset