DeepAI AI Chat
Log In Sign Up

Teaching Large Language Models to Self-Debug

by   Xinyun Chen, et al.

Large language models (LLMs) have achieved impressive performance on code generation. However, for complex programming tasks, generating the correct solution in one go becomes challenging, thus some prior works have designed program repair approaches to improve code generation performance. In this work, we propose Self-Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations. In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i.e., without any feedback on the code correctness or error messages, the model is able to identify its mistakes by explaining the generated code in natural language. Self-Debugging achieves the state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation. On the Spider benchmark where there are no unit tests to verify the correctness of predictions, Self-Debugging with code explanation consistently improves the baseline by 2-3 9 improves the baseline accuracy by up to 12 messages and reusing failed predictions, Self-Debugging notably improves sample efficiency, and can match or outperform baseline models that generate more than 10x candidate programs.


page 5

page 7

page 11

page 12

page 13


LEVER: Learning to Verify Language-to-Code Generation with Execution

The advent of large language models trained on code (code LLMs) has led ...

I Speak, You Verify: Toward Trustworthy Neural Program Synthesis

We develop an approach for improving the trustworthiness and overall acc...

Demystifying GPT Self-Repair for Code Generation

Large Language Models (LLMs) have shown remarkable aptitude in code gene...

Self-planning Code Generation with Large Language Model

Although large language models have demonstrated impressive ability in c...

Natural Language to Code Generation in Interactive Data Science Notebooks

Computational notebooks, such as Jupyter notebooks, are interactive comp...

Fault-Aware Neural Code Rankers

Large language models (LLMs) have demonstrated an impressive ability to ...

RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model

Inspired by the recent success of large language models (LLMs) like Chat...