Teaching computers to answer complex natural language questions requires sophisticated reasoning and human language understanding. We investigate generic natural language interfaces for simple arithmetic questions on semi-structured tables. Typical questions for this task are topic independent and may require performing multiple discrete operations such as aggregation, comparison, superlatives or arithmetics.
We propose a weakly supervised neural model that eliminates the need for expensive feature engineering in the candidate ranking stage. Each natural language question is translated using the method of  into a set of machine understandable candidate representations, called logical forms or programs. Then, the most likely such program is retrieved in two steps: i) using a simple algorithm, logical forms are transformed back into paraphrases (textual representations) understandable by non-expert users, ii) next, these strings are further embedded together with their respective questions in a jointly learned vector space using convolutional neural networks over character and word embeddings. Multi-layer neural networks and bilinear mappings are further employed as effective similarity measures and combined to score the candidate interpretations. Finally, the highest ranked logical form is executed against the input data to retrieve the answer. Our method uses only weak-supervision from question-answer-table input triples, without requiring expensive annotations of gold logical forms.
We empirically test our approach on a series of experiments on WikiTableQuestions, to our knowledge the only dataset designed for this task. An ensemble of our best models reached state-of-the-art accuracy of 38.7% at the moment of publication.
2 Related Work
We briefly mention here two main types of QA systems related to our task111An extensive list of open-domain QA publications can be found here: https://aclweb.org/aclwiki/Question_Answering_(State_of_the_art): semantic parsing-based and embedding-based. Semantic parsing-based methods perform a functional parse of the question that is further converted to a machine understandable program and executed on a knowledgebase or database. For QA on semi-structured tables with multi-compositional queries,  generate and rank candidate logical forms with a log-linear model, resorting to hand-crafted features for scoring. As opposed, we learn neural features for each question and the paraphrase of each candidate logical form. Paraphrases and hand-crafted features have successfully facilitated semantic parsers targeting simple factoid  and compositional questions . Compositional questions are also the focus of  that construct logical forms from the question embedding through operations parametrized by RNNs, thus losing interpretability. A similar fully neural, end-to-end differentiable network was proposed by .
We describe our QA system. For every question : i) a set of candidate logical forms is generated using the method of ; ii) each such candidate program is transformed in an interpretable textual representation ; iii) all ’s are jointly embedded with in the same vector space and scored using a neural similarity function; iv) the logical form corresponding to the highest ranked is selected as the machine-understandable translation of question and executed on the input table to retrieve the final answer. Our contributions are the novel models that perform steps ii) and iii), while for step i) we rely on the work of  (henceforth: PL2015).
3.1 Candidate Logical Form Generation
We generate a set of candidate logical forms from a question using the method of . Only briefly, we review this method. Specifically, a question is parsed into a set of candidate logical forms using a semantic parser that recursively applies deduction rules. Logical forms are represented in Lambda DCS form  and can be executed on a table to yield an answer. An example of a question and its correct logical form are below:
How many people attended the last Rolling Stones concert?
3.2 Converting Logical Forms to Text
In Algorithm 1 we describe how logical forms are transformed into interpretable textual representations called ”paraphrases”. We choose to embed paraphrases in low dimensional vectors and compare these against the question embedding. Working directly with paraphrases instead of logical forms is a design choice, justified by their interpretability, comprehensibility (understandability by non-technical users) and empirical accuracy gains. Our method recursively traverses the tree representation of the logical form starting at the root. For example, the correct candidate logical form for the question mentioned in section 3.1, namely How many people attended the last Rolling Stones concert?, is mapped to the paraphrase Attendance as number of last table row where act is Rolling Stones.
3.3 Joint Embedding Model
We embed the question together with the paraphrases of candidate logical forms in a jointly learned vector space. We use two convolutional neural networks (CNNs) for question and paraphrase embeddings, on top of which a max-pooling operation is applied. The CNNs receive as input token embeddings obtained as described below.
3.3.1 Token Embedding
The embedding of an input word sequence (e.g. question, paraphrase) is depicted in Figure 1 and is similar to . Every token is parametrized by learnable word and character embeddings. The latter help dealing with unknown tokens (e.g. rare words, misspellings, numbers or dates). Token vectors are then obtained using a CNN (with multiple filter widths) over the constituent characters , followed by a max-over-time pooling layer and concatenation with the word vector.
3.3.2 Sentence Embedding
We map both the question and the paraphrase into a joint vector space using sentence embeddings obtained from two jointly trained CNNs. CNNs’ filters span a different number of tokens from a width set . For each filter width , we learn different filters, each of dimension , where is the word embedding size. After the convolution layer, we apply a max-over-time pooling on the resulting feature matrices which yields, per filter-width, a vector of dimension . Next, we concatenate the resulting max-over-time pooling vectors of the different filter-widths in to form our sentence embedding. The final sentence embedding size is .
3.3.3 Neural Similarity Measures
Let be the sentence embeddings of question and of paraphrase . We experiment with the following similarity scores: i) DOTPRODUCT : ; ii) BILIN : , with being a trainable matrix; iii) FC: u and v concatenated, followed by two sequential fully connected layers with ELU non-linearities; iv) FC-BILIN: weighted average of BILIN and FC. These models define parametrized similarity scoring functions , where is the set of natural language questions and is the set of paraphrases of logical forms.
3.4 Training Algorithm
For training, we build two sets (positive) and (negative) consisting of all pairs of questions and paraphrases of candidate logical forms generated as described in Section 3.1
. A pair is positive or negative if its logical form gives the correct or respectively incorrect gold answer when executed on the corresponding table. During training, we use the ranking hinge loss function (with margin):
Dataset: For training and testing we use the train-validation-test split of WikiTableQuestions , a dataset containing 22,033 pairs of questions and answers based on 2,108 Wikipedia tables. This dataset is also used by our baselines, [8, 7]. Tables are not shared across these splits, which requires models to generalize to unseen data. We obtain about 3.8 million training triples , where is a binary indicator of whether the logical form gives the correct gold answer when executed on the corresponding table. 76.7% of the questions have at least one correct candidate logical form when generated with the model of .
|Question||Paraphrase (gold vs predicted)|
|Which association entered last?||association of last row|
|association of row with highest number of joining year|
|What is the total of all the medals?||count all rows|
|number of total of nation is total|
|count original air date as date <= 12 1965|
|count original air date as date <12 1965|
Our models are implemented using TensorFlow and trained on a single Tesla P100 GPU. Training takes approximately 6 hours. We initialize word vectors with 200 dimensional GloVe () pre-trained vectors. For the character CNN we use widths spanning 1, 2 and 3 characters. The sentence embedding CNNs use widths of
. The fully connected layers in the FC models have 500 hidden neurons, which we regularize using 0.8-dropout. The loss marginis set to 0.2. Optimization is done using Adam 
with a learning rate of 7e-4. Hyperparameters are tunned on the development data split of the Wiki-TableQuestions table. We choose the best performing model on the validation set using early stopping.
Results: Experimental results are shown in Table 1
. Our best performing single model is FC-BILIN with CNNs, Intuitively, BILIN and FC are able to extract different interaction features between the two input vectors, while their linear combination retains the best of both models. An ensemble of 15 single CNN-FC-BILIN models was setting (at the moment of publication) a new state-of-the-art precision@1 for this dataset: 38.7%. This shows that the same model initialized differently can learn different features. We also experimented with recurrent neural networks (RNNs) for the sentence embedding since these are known to capture word order better than CNNs. However, RNN-FC-BILIN performs worse than its CNN variant.
There are a few reasons that contributed to the low accuracy obtained on this task by various methods (including ours) compared to other NLP problems: weak supervision, small training size and a high percentage of unanswerable questions.
Error Analysis: The questions our models do not answer correctly can be split into two categories: either a correct logical form is not generated, or our scoring models do not rank the correct one at the top. We perform a qualitative analysis presented in Table 2 to reveal common question types our models often rank incorrectly. The first two examples show questions whose correct logical form depends on the structure of the table. In these cases a bias towards the more general logical form is often exhibited. The third example shows that our model has difficulty distinguishing operands with slight modification (e.g. smaller and smaller equals), which may be due to weak-supervision.
Ablation Studies: For a better understanding of our model, we investigate the usefulness of various components with an ablation study shown in Table 3. In particular, we emphasize that replacing the paraphrasing stage with the raw strings of the Lambda DCS expressions resulted in lower precision@1, which confirms the utility of this stage.
Analysis of Correct Answers: We analyze how well our best single model performs on various question types. For this, we manually annotate 80 randomly chosen questions that are correctly answered by our model and report statistics in Table 3.
In this paper we propose a neural network QA system for semi-structured tables that eliminates the need for manually designed features. Experiments show that an ensemble of our models reaches competitive accuracy on the WikiTableQuestions dataset, thus indicating its capability to answer complex, multi-compositional questions. Our code is available at https://github.com/dalab/neural_qa .
This research was supported by the Swiss National Science Foundation (SNSF) grant number 407540_167176 under the project ”Conversational Agent for Interactive Access to Information”.
-  Jonathan Berant and Percy Liang. Semantic parsing via paraphrasing. In ACL (1), pages 1415–1425, 2014.
-  Antoine Bordes, Sumit Chopra, and Jason Weston. Question answering with subgraph embeddings. arXiv preprint arXiv:1406.3676, 2014.
-  Yoon Kim. Convolutional neural networks for sentence classification. In In EMNLP. Citeseer, 2014.
Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush.
Character-aware neural language models.
Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 2741–2749. AAAI Press, 2016.
-  Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
-  Percy Liang. Lambda dependency-based compositional semantics. arXiv:1309.4408, 2013.
-  Arvind Neelakantan, Quoc V Le, Martin Abadi, Andrew McCallum, and Dario Amodei. Learning a natural language interface with neural programmer. arXiv:1611.08945, 2016.
-  Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In In Proceedings of the Annual Meeting of the Association for Computational Linguistics. Citeseer, 2015.
-  Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532–1543, 2014.
-  Yushi Wang, Jonathan Berant, Percy Liang, et al. Building a semantic parser overnight. In ACL (1), pages 1332–1342, 2015.
-  Pengcheng Yin, Zhengdong Lu, Hang Li, and Ben Kao. Neural enquirer: learning to query tables in natural language. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pages 2308–2314. AAAI Press, 2016.