share
1 Introduction
One way to construct a question-answering system over database is leveraging the semantic parsing. Semantic parsing is a task that transform the natural language to logic form which computer can execute. Transforming from natural language to SQL (NL2SQL) is kind of semantic parsing task. The generated SQL can be executed in the database system to can the answer from the database. In recent years, deep learning techniques [1] is applied to semantic parsing[4][7][8][9]
. But deep learning need large amount of labeled data, when the parameter number is very large. The logic form of natural language is also very hard to label, compared to other natural language processing (NLP) tasks such as text classification or sentence similarity. There are some works
[14] solve the hard labeling problem in weak supervision methods. Our work try to solve this problem in another way. The upper level view of the problem of semantic parsing for question answering is retrieve the answer from the database when given a question. Human can solve this problem even without an explicit logic form. Human can read the schema or columns’ info in the database and answer the question. We think the deep model can integrate the logic or reasoning modules like [2] or other deep model to search on the database without an explicit logic form. We present our idea in Fig. 1. and Fig. 2.
We use BERT-based [3] to implement our idea. BERT is a new language representation model, which stands for Bidirectional Encoder Representations from Transformers. Pre-trained word representations on a large (unlabeled) language corpus, such as [15] , have shown promising results in a lot of NLP tasks. BERT is also a pre-trained deep model[11][12][13] which use large amount of plain text to pre-train. As a result, the pre-trained BERT representations can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and text classification.
Our main contributions in this work are two-fold. First, we introduce our idea of finding the answer from database with out semantic parsing to solve the problem of hard label work for semantic parsing. Second, we use BERT-based model to implement our idea and achieve a baseline level experiment result. The code is available. 111https://github.com/guotong1988/table2answer
2 Task Description
In the NL2SQL without SQL task (Table2answer), given a question and a database table, the deep model needs to find the answer in the database table. The question is described as a sequence of word tokens: , where is the number of words in the question, and the table is described as a sequence of column names or headers , where is the number of columns in the table. Each table contains a number of rows which contains the answer or cells to the question. The answer for the model is the pointers to the table cells. We denote the cells as , where r is the number of rows. Note that each is not one word in the table, but each is one cell in the table. In experiment, we concatenate the word embeddings in a cell to represent one cell. The cell representation is the input of the transformer layers of BERT.
We now describe the WikiSQL dataset [4], a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables from Wikipedia. We present an example in Fig. 3.
We extract the question, answer index and table content from the WikiSQL dataset and construct the dataset for Table2answer. One table corresponding to several questions with answers in the table. For an elementary consideration, we only extract the data case which SQL of question only contains one condition in the WHERE clause. We leave it as future work which solves all kinds of questions with SQL in the WikiSQL. We present an example in Fig. 4. Also, Our model works under the condition that the table which contains the answer is determined. In other words, our model need not predict the exact table in the all tables. We use other methods to find the exact table in industry application.
3 Model
In this section, we describe the details of our BERT-based model to solve the problem of question answering over database without semantic parsing. We present the overall solution for the Table2answer problem in Fig. 5. The reason and inspiration to use BERT-based model is that we want to leverage the attention info between question and table header to point the exact cell in the table.
We follow the BERT convention of data input format for encoding the natural language question together with the headers and cells of the table. We use [SEP] to separate between the cells and the question. We average word embedding of one cell for the representation of one cell. At last, each headers , table cells and question is encoded as following:
In the SQuAD [5] machine comprehension task, we input the paragraph and question to the model and find the answer string in the paragraph. And our Table2answer task, we input the question, headers and table cells to find the answer cell in the table. The two tasks are very similar in this perspective. So we append one pointer [6] after the output of the BERT main module. It is different from machine comprehension task because we only have one cell to point as we have simplified the dataset. We leave it as future work to choose more than one cell as answer.
4 Experiments
In this section, we present more details of the model and the evaluation on the dataset. Pre-trained BERT models (BERT-Base-Uncased) are loaded and fine-tuned with Adam optimizer with learning rate
. The batch size is 16. We use the origin BERT tokenizer with the same vocabulary of BERT-Base-Uncased. We fix the parameters of 1-9 layers of BERT-Base and fine-tune the last 3 layers, as we observe that fine-tuning all the layers do not get a better evaluation result. Our neural network model is implemented in TensorFlow.
4.1 Data augmentation
We randomly shuffle the rows of all the tables and get a training dataset of 503881 data and test dataset of 1874 data. Note that the answer cell index is corresponding to the shuffled rows. We have not shuffled the columns of table as we observe bad result of it. See Tab. 1. for detail.
| Training Data Size | Test Accuracy |
|---|---|
| 76301 | 20.3 |
| 321536 | 47.2 |
| 503881 | 54.0 |
4.2 Evaluation
We evaluate our model on the dataset that extract from WikiSQL. The results are presented in Tab. 2. The training accuracy is around 96 and we leave it as future work to further improve the result on the test dataset. We also do the experiment just the same as SQuAD machine comprehension task. That is, we concatenate all the words in the table cells and append two pointer after BERT for the start index and end index. And the result for this kind of data feeding methods is 2-3 lower.
| Model | Test Accuracy |
|---|---|
| transformer baseline | 11.0 |
| Our model | 54.0 |
| Our model without data augmentation | 17.7 |
| Our model without position embedding | 20.5 |
5 Conclusion
In this paper, in order to solve the problem that the labeling work for semantic parsing is too hard, we introduce our idea that inject the reasoning part into the deep model to remove the logic form step for question answering. We think that human can do the logic operation even without the SQL so we believe our idea will work. Then We design the BERT-based model and achieve the baseline results in the sub-WikiSQL dataset. It is trained end-to-end and can retrieve the answer directly. The dataset for table2answer is simpler than WikiSQL and there will be a lot of work to research.
References
-
[1]
A. Krizhevsky, I. Sutskever, and G. Hinton.: Imagenet classification with deep convolutional neural networks. In NIPS (2012)
- [2] Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer, Phil Blunsom: Neural Arithmetic Logic Units. arxiv.org/abs/1808.00508
- [3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
- [4] Victor Zhong, C. Xiong, and R. Socher. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. arXiv preprint arxiv:1709.00103, Nov 2017
- [5] Rajpurkar P, Zhang J, Lopyrev K, et al: Squad: 100,000+ questions for machine comprehension of text[J]. arXiv pre-print arXiv:1606.05250 (2016)
- [6] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly: Pointer networks. International Conference on Neural Information Processing Systems. MIT Press (2015)
-
[7]
Xiaojun Xu, Chang Liu, Dawn Song. 2017. SQLNet: Generating Structured Queries from Natural Language Without Reinforcement Learning.
- [8] Yu T, Li Z, Zhang Z, et al. TypeSQL: Knowledge-based Type-Aware Neural Text-to-SQL Generation[J]. arXiv preprint arXiv:1804.09769, 2018.
- [9] Dong, Li, and Mirella Lapata. ”Coarse-to-Fine Decoding for Neural Semantic Parsing.” arXiv preprint arXiv:1805.04793 (2018).
- [10] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762
- [11] Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL.
-
[12]
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI.
- [13] Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079-3087
- [14] Liang C, Berant J, Le Q, et al. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision[J]. arXiv preprint arXiv:1611.00020, 2016.
-
[15]
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.