DeepAI AI Chat
Log In Sign Up

Semantic Evaluation for Text-to-SQL with Distilled Test Suites

10/06/2020
by   Ruiqi Zhong, et al.
0

We propose test suite accuracy to approximate semantic accuracy for Text-to-SQL models. Our method distills a small test suite of databases that achieves high code coverage for the gold query from a large number of randomly generated databases. At evaluation time, it computes the denotation accuracy of the predicted queries on the distilled test suite, hence calculating a tight upper-bound for semantic accuracy efficiently. We use our proposed method to evaluate 21 models submitted to the Spider leader board and manually verify that our method is always correct on 100 examples. In contrast, the current Spider metric leads to a 2.5 worst case, indicating that test suite accuracy is needed. Our implementation, along with distilled test suites for eleven Text-to-SQL datasets, is publicly available.

READ FULL TEXT
05/25/2023

UNITE: A Unified Benchmark for Text-to-SQL Evaluation

A practical text-to-SQL system should generalize well on a wide variety ...
09/07/2019

Compiling PL/SQL Away

"PL/SQL functions are slow," is common developer wisdom that derives fro...
08/12/2021

Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-based Hate

Detecting online hate is a complex task, and low-performing models have ...
09/13/2018

IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles

We present a sequence-to-action parsing approach for the natural languag...
09/08/2023

Seeding Contradiction: a fast method for generating full-coverage test suites

The regression test suite, a key resource for managing program evolution...
01/03/2017

Experiences with Some Benchmarks for Deductive Databases and Implementations of Bottom-Up Evaluation

OpenRuleBench is a large benchmark suite for rule engines, which include...
05/28/2019

One SQL to Rule Them All

Real-time data analysis and management are increasingly critical for tod...