PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models

09/10/2021
by   Torsten Scholak, et al.
0

Large pre-trained language models for textual data have an unconstrained output space; at each decoding step, they can produce any of 10,000s of sub-word tokens. When fine-tuned to target constrained formal languages like SQL, these models often generate invalid code, rendering it unusable. We propose PICARD (code and trained models available at https://github.com/ElementAI/picard), a method for constraining auto-regressive decoders of language models through incremental parsing. PICARD helps to find valid output sequences by rejecting inadmissible tokens at each decoding step. On the challenging Spider and CoSQL text-to-SQL translation tasks, we show that PICARD transforms fine-tuned T5 models with passable performance into state-of-the-art solutions.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
05/22/2023

Text-to-SQL Error Correction with Language Models of Code

Despite recent progress in text-to-SQL parsing, current semantic parsers...
research
08/24/2022

Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Models

The ability to extrapolate, i.e., to make predictions on sequences that ...
research
05/23/2023

Flexible Grammar-Based Constrained Decoding for Language Models

LLMs have shown impressive few-shot performance across many tasks. Howev...
research
06/21/2022

BenchCLAMP: A Benchmark for Evaluating Language Models on Semantic Parsing

We introduce BenchCLAMP, a Benchmark to evaluate Constrained LAnguage Mo...
research
09/06/2023

Improving Code Generation by Dynamic Temperature Sampling

Recently, Large Language Models (LLMs) have shown impressive results in ...
research
02/13/2023

Machine Learning Model Attribution Challenge

We present the findings of the Machine Learning Model Attribution Challe...
research
05/08/2023

ComputeGPT: A computational chat model for numerical problems

Language models are not accurate in numerical problems. Their architectu...

Please sign up or login with your details

Forgot password? Click here to reset