Table2answer: Read the database and answer without SQL

by   Tong Guo, et al.

Semantic parsing is the task of mapping natural language to logic form. In question answering, semantic parsing can be used to map the question to logic form and execute the logic form to get the answer. One key problem for semantic parsing is the hard label work. We study this problem in another way: we do not use the logic form any more. Instead we only use the schema and answer info. We think that the logic form step can be injected into the deep model. The reason why we think removing the logic form step is possible is that human can do the task without explicit logic form. We use BERT-based model and do the experiment in the WikiSQL dataset, which is a large natural language to SQL dataset. Our experimental evaluations that show that our model can achieves the baseline results in WikiSQL dataset.


page 1

page 2

page 3

page 4


Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedback

We study the task of semantic parse correction with natural language fee...

Towards Transparent Interactive Semantic Parsing via Step-by-Step Correction

Existing studies on semantic parsing focus primarily on mapping a natura...

Abductive Matching in Question Answering

We study question-answering over semi-structured data. We introduce a ne...

AutoQA: From Databases To QA Semantic Parsers With Only Synthetic Training Data

We propose AutoQA, a methodology and toolkit to generate semantic parser...

Translating Natural Language to SQL using Pointer-Generator Networks and How Decoding Order Matters

Translating natural language to SQL queries for table-based question ans...

Right Answer for the Wrong Reason: Discovery and Mitigation

Exposing the weaknesses of neural models is crucial for improving their ...

Semantic Parsing to Manipulate Relational Database For a Management System

Chatbots and AI assistants have claimed their importance in today life. ...

1 Introduction

One way to construct a question-answering system over database is leveraging the semantic parsing. Semantic parsing is a task that transform the natural language to logic form which computer can execute. Transforming from natural language to SQL (NL2SQL) is kind of semantic parsing task. The generated SQL can be executed in the database system to can the answer from the database. In recent years, deep learning techniques [1] is applied to semantic parsing[4][7][8][9]

. But deep learning need large amount of labeled data, when the parameter number is very large. The logic form of natural language is also very hard to label, compared to other natural language processing (NLP) tasks such as text classification or sentence similarity. There are some works

[14] solve the hard labeling problem in weak supervision methods. Our work try to solve this problem in another way. The upper level view of the problem of semantic parsing for question answering is retrieve the answer from the database when given a question. Human can solve this problem even without an explicit logic form. Human can read the schema or columns’ info in the database and answer the question. We think the deep model can integrate the logic or reasoning modules like [2] or other deep model to search on the database without an explicit logic form. We present our idea in Fig. 1. and Fig. 2.

Figure 1: The illustration of natural language to SQL for question answering
Figure 2: The illustration of question answering over database without SQL

We use BERT-based [3] to implement our idea. BERT is a new language representation model, which stands for Bidirectional Encoder Representations from Transformers. Pre-trained word representations on a large (unlabeled) language corpus, such as [15] , have shown promising results in a lot of NLP tasks. BERT is also a pre-trained deep model[11][12][13] which use large amount of plain text to pre-train. As a result, the pre-trained BERT representations can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and text classification.

Our main contributions in this work are two-fold. First, we introduce our idea of finding the answer from database with out semantic parsing to solve the problem of hard label work for semantic parsing. Second, we use BERT-based model to implement our idea and achieve a baseline level experiment result. The code is available. 111

2 Task Description

In the NL2SQL without SQL task (Table2answer), given a question and a database table, the deep model needs to find the answer in the database table. The question is described as a sequence of word tokens: , where is the number of words in the question, and the table is described as a sequence of column names or headers , where is the number of columns in the table. Each table contains a number of rows which contains the answer or cells to the question. The answer for the model is the pointers to the table cells. We denote the cells as , where r is the number of rows. Note that each is not one word in the table, but each is one cell in the table. In experiment, we concatenate the word embeddings in a cell to represent one cell. The cell representation is the input of the transformer layers of BERT.

We now describe the WikiSQL dataset [4], a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables from Wikipedia. We present an example in Fig. 3.

We extract the question, answer index and table content from the WikiSQL dataset and construct the dataset for Table2answer. One table corresponding to several questions with answers in the table. For an elementary consideration, we only extract the data case which SQL of question only contains one condition in the WHERE clause. We leave it as future work which solves all kinds of questions with SQL in the WikiSQL. We present an example in Fig. 4. Also, Our model works under the condition that the table which contains the answer is determined. In other words, our model need not predict the exact table in the all tables. We use other methods to find the exact table in industry application.

Figure 3: An example of the WikiSQL semantic parsing dataset. The inputs consist of a table and a question. The outputs consist of a ground truth SQL query and the corresponding result from execution.
Figure 4: An example of the Table2anwser dataset. The inputs consist of a table and a question. The outputs only consist the corresponding result from SQL execution.

3 Model

In this section, we describe the details of our BERT-based model to solve the problem of question answering over database without semantic parsing. We present the overall solution for the Table2answer problem in Fig. 5. The reason and inspiration to use BERT-based model is that we want to leverage the attention info between question and table header to point the exact cell in the table.

Figure 5: The overall model for Table2answer

We follow the BERT convention of data input format for encoding the natural language question together with the headers and cells of the table. We use [SEP] to separate between the cells and the question. We average word embedding of one cell for the representation of one cell. At last, each headers , table cells and question is encoded as following:

In the SQuAD [5] machine comprehension task, we input the paragraph and question to the model and find the answer string in the paragraph. And our Table2answer task, we input the question, headers and table cells to find the answer cell in the table. The two tasks are very similar in this perspective. So we append one pointer [6] after the output of the BERT main module. It is different from machine comprehension task because we only have one cell to point as we have simplified the dataset. We leave it as future work to choose more than one cell as answer.

4 Experiments

In this section, we present more details of the model and the evaluation on the dataset. Pre-trained BERT models (BERT-Base-Uncased) are loaded and fine-tuned with Adam optimizer with learning rate

. The batch size is 16. We use the origin BERT tokenizer with the same vocabulary of BERT-Base-Uncased. We fix the parameters of 1-9 layers of BERT-Base and fine-tune the last 3 layers, as we observe that fine-tuning all the layers do not get a better evaluation result. Our neural network model is implemented in TensorFlow.

4.1 Data augmentation

We randomly shuffle the rows of all the tables and get a training dataset of 503881 data and test dataset of 1874 data. Note that the answer cell index is corresponding to the shuffled rows. We have not shuffled the columns of table as we observe bad result of it. See Tab. 1. for detail.

Training Data Size Test Accuracy
76301 20.3
321536 47.2
503881 54.0
Table 1: The evaluation of our experiment for data augmentation.

4.2 Evaluation

We evaluate our model on the dataset that extract from WikiSQL. The results are presented in Tab. 2. The training accuracy is around 96 and we leave it as future work to further improve the result on the test dataset. We also do the experiment just the same as SQuAD machine comprehension task. That is, we concatenate all the words in the table cells and append two pointer after BERT for the start index and end index. And the result for this kind of data feeding methods is 2-3 lower.

Model Test Accuracy
transformer baseline 11.0
Our model 54.0
Our model without data augmentation 17.7
Our model without position embedding 20.5
Table 2: The evaluation of our experiment. Our baseline is a transformer[10] without pre-training. As there may be same answers in different cells, we consider the final word match as the accuracy.

5 Conclusion

In this paper, in order to solve the problem that the labeling work for semantic parsing is too hard, we introduce our idea that inject the reasoning part into the deep model to remove the logic form step for question answering. We think that human can do the logic operation even without the SQL so we believe our idea will work. Then We design the BERT-based model and achieve the baseline results in the sub-WikiSQL dataset. It is trained end-to-end and can retrieve the answer directly. The dataset for table2answer is simpler than WikiSQL and there will be a lot of work to research.