Neural Enquirer: Learning to Query Tables with Natural Language

12/03/2015 ∙ by Pengcheng Yin, et al. ∙ HUAWEI Technologies Co., Ltd. The University of Hong Kong 0

We proposed Neural Enquirer as a neural network architecture to execute a natural language (NL) query on a knowledge-base (KB) for answers. Basically, Neural Enquirer finds the distributed representation of a query and then executes it on knowledge-base tables to obtain the answer as one of the values in the tables. Unlike similar efforts in end-to-end training of semantic parsers, Neural Enquirer is fully "neuralized": it not only gives distributional representation of the query and the knowledge-base, but also realizes the execution of compositional queries as a series of differentiable operations, with intermediate results (consisting of annotations of the tables at different levels) saved on multiple layers of memory. Neural Enquirer can be trained with gradient descent, with which not only the parameters of the controlling components and semantic parsing component, but also the embeddings of the tables and query words can be learned from scratch. The training can be done in an end-to-end fashion, but it can take stronger guidance, e.g., the step-by-step supervision for complicated queries, and benefit from it. Neural Enquirer is one step towards building neural network systems which seek to understand language by executing it on real-world. Our experiments show that Neural Enquirer can learn to execute fairly complicated NL queries on tables with rich structures.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

In models for natural language dialogue and question answering, there is ubiquitous need for querying a knowledge-base [13, 11]. The traditional pipeline is to put the query through a semantic parser to obtain some “executable” representations, typically logical forms, and then apply this representation to a knowledge-base for the answer. Both the semantic parsing and the query execution part can get quite messy for complicated queries like : “Which city hosted the longest game before the game in Beijing?” in Figure 1, and need carefully devised systems with hand-crafted features or rules to derive the correct logical form (written in SQL-like style). Partially to overcome this difficulty, there has been effort [11]

to “backpropagate” the result of query execution to revise the semantic representation of the query, which actually falls into the thread of work on learning from grounding 

[5]. One drawback of these semantic parsing models is rather symbolic with rule-based features, leaving only a handful of tunable parameters to cater to the supervision signal from the execution result.

On the other hand, neural network-based models are previously successful mostly on tasks with direct and strong supervision in natural language processing or related domain, with examples including machine translation and syntactic parsing. The recent work on learning to execute simple Python code with LSTM 

[15]

pioneers in the direction on learning to parse structured objects through executing it in a purely neural way, while the later work on Neural Turing Machine (NTM) 

[6] introduces more modeling flexibility by equipping the LSTM with external memory and various means of interacting with it.

Our work, inspired by above-mentioned threads of research, aims to design a neural network system that can learn to understand the query and execute it on a knowledge-base table from examples of queries and answers. Our proposed Neural Enquirer encodes queries and KBs into distributed representations, and executes compositional queries against the KB through a series of differentiable operations. It can be trained using Query-Answer pairs, where the distributed representations of queries and the KB are optimized together with the query execution logic in an end-to-end fashion. We then demonstrates using a synthetic question-answering task that our proposed Neural Enquirer is capable of learning to execute compositional natural language queries with complex structures.

2 Overview of Neural Enquirer

Given an NL query and a KB table , Neural Enquirer executes the query against the table and outputs a ranked list of query answers. The execution is done by first using Encoders to encode the query and table into distributed representations, which are then sent to a cascaded pipeline of Executors to derive the answer. Figure 1 gives an illustrative example (with five executors) of various types of components involved:

Figure 1: An overview of Neural Enquirer with five executors

xx Query Encoder (Section 3.1), which encodes the query into a distributed representation that carries the semantic information of the original query. The encoded query embedding will be sent to various executors to compute its execution result.

xx Table Encoder (Section 3.2

), which encodes entries in the table into distributed vectors. Table Encoder outputs an embedding vector for each table entry, which retains the two-dimensional structure of the table.

xx Executor (Section 3.3), which executes the query against the table and outputs annotations that encode intermediate execution results, which are stored in the memory of each layer to be accessed by subsequent executor. Our basic assumption is that complex, compositional queries can be answered through multiple steps of computation, where each executor models a specific type of operation conditioned on the query. Figure 1 illustrates the operation each executor is supposed to perform in answering . Different from classical semantic parsing approaches which require a predefined set of all possible logical operations, Neural Enquirer is capable of learning the logic of executors via end-to-end training using Query-Answer pairs. By stacking several executors, Neural Enquirer is able to answer complex queries involving multiple steps of computation.

3 Model

In this section we give a more detailed exposition of different types of components in the Neural Enquirer model.

3.1 Query Encoder

Given an NL query composed of a sequence of words , Query Encoder parses into a -dimensional vectorial representation : . In our implementation of Neural Enquirer, we employ a bidirectional RNN for this mission111

Other choices of sentence encoder such as LSTM or even convolutional neural networks are possible too

. More specifically, the RNN summarizes the sequence of word embeddings of , , into a vector as the representation of , where , and is the embedding matrix. See Appendix A for details.

It is worth noting that our Query Encoder can find the representation of rather general class of symbol sequences, agnostic to the actual representation of queries (e.g., natural language, SQL-like, etc). Neural Enquirer is capable of learning the execution logic expressed in the input query through end-to-end training, making it a generic model for query execution.

3.2 Table Encoder

Table Encoder converts a knowledge-base table into its distributional representation as input to Neural Enquirer. Suppose the table has rows and columns, where each column comes with a field name (e.g., host_city), and the value of each table entry is a word (e.g., Beijing) in our vocabulary, Table Encoder first finds the embedding for field names and values of table, and then it computes the (field, value) composite embedding for each of the entries in the table. More specifically, for the entry in the -th row and -th column with a value of , Table Encoder computes a -dimensional embedding vector by fusing the embedding of the entry value with the embedding of its corresponding field name as follows:

where is the embedding of the field name (of the -th column). and denote the weight matrices, and

the concatenation of vectors. The output of Table Encoder is a tensor of shape

, consisting of embeddings of length for all entries.

Our Table Encoder functions differently from classical knowledge embedding models (e.g., TransE [4]), where embeddings of entities (entry values) and relations (field names) are learned in a unsupervised fashion via minimizing certain reconstruction errors. Embeddings in Neural Enquirer

are optimized via supervised learning towards end-to-end QA tasks. Additionally, as will shown in the experiments, those embeddings function in a way as indices, which not necessarily encode the exact semantic meaning of their corresponding words.

3.3 Executor

Figure 2: Overview of an Executor-

Neural Enquirer executes an input query on a KB table through layers of execution. Each layer of executor captures a certain type of operation (e.g., select, where, max, etc.) relevant to the input query222Depending on the query, an executor may perform different operations., and returns intermediate execution results, referred to as annotations, saved in an external memory of the same layer. A query is executed step-by-step through a sequence of stacked executors. Such a cascaded architecture enables Neural Enquirer to answer complex, compositional queries. An illustrative example is given in Figure 1, with each executor annotated with the operation it is assumed to perform. We will demonstrate in Section 5 that Neural Enquirer is capable of learning the operation logic of each executor via end-to-end training.

As illustrated in Figure 2, an executor at Layer- (denoted as Executor-) has two major neural network components: Reader and Annotator. An executor processes a table row-by-row. For the -th row, with (field, value) composite embeddings , the Reader fetches a read vector from , which is sent to the Annotator to compute a row annotation :

Read Vector: (1)
Row Annotation: (2)

where denotes the content in memory Layer-, and is the set of field name embeddings. Once all row annotations are obtained, Executor- then generates the table annotation through the following pooling process:

A row annotation captures the local execution result on each row, while a table annotation, derived from all row annotations, summarizes the global computational result on the whole table. Both row annotations and table annotation are saved in memory Layer-: .

Figure 3: Illustration of the Reader for Executor-.

Our design of executor is inspired by Neural Turing Machines [6], where data is fetched from an external memory using a read head, and subsequently processed by a controller, whose outputs are flushed back in to memories. An executor functions similarly by reading data from each row of the table, using a Reader, and then calling an Annotator to calculate intermediate computational results as annotations, which are stored in the executor’s memory. We assume that row annotations are able to handle operations which require only row-wise, local information (e.g., select, where), while table annotations can model superlative operations (e.g., max, min) by aggregating table-wise, global execution results. Therefore, a combination of row and table annotations enables Neural Enquirer to capture a variety of real-world query operations.

3.3.1 Reader

As illustrated in Figure 3, an executor at Layer- reads in a vector for each row , defined as the weighted sum of composite embeddings for entries in this row:

where is the normalized attention weights given by:

(3)

and is modeled as a DNN (denoted as ).

Note that the is agnostic to the values of entries in the row, i.e., in an executor all rows share the same set of weights . Since each executor models a specific type of computation, it should only attend to a subset of entries pertain to its execution, which is modeled by the Reader. This is related to the content-based addressing of Neural Turing Machines [6]

and the attention mechanism in neural machine translation models 

[2].

3.3.2 Annotator

In Executor-, the Annotator computes row and table annotations based on the fetched read vector of the Reader, which are then stored in the -th memory layer accessible to Executor-. This process is repeated in intermediate layers, until the executor in the last layer to finally generate the answer.

Row annotations

A row annotation encodes the local computational result on a specific row. As illustrated in Figure 4, a row annotation for row in Executor-, given by

(4)

fuses the corresponding read vector , the results saved in previous memory layer (row and table annotations , ), and the query embedding . Basically,

  • row annotation represents the local status of the execution before Layer-;

  • table annotation summarizes the global status of the execution before Layer-;

  • read vector stores the value of current attention;

  • query embedding encodes the overall execution agenda,

all of which are combined through to form the annotation of row in the current layer.

Figure 4: Illustration of Annotator for Executor-.
Table annotations

Capturing the global execution state, a table annotation is summarized from all row annotations via a global pooling operation. In our implementation of Neural Enquirer

we employ max pooling:

(5)

where is the maximum value among the -th elements of all row annotations. It is possible to use other pooling operations (e.g., gated pooling), but we find max pooling yields the best results.

3.3.3 Last Layer of Executor

Instead of computing annotations based on read vectors, the last executor in Neural Enquirer

directly outputs the probability of the value of each entry in

being the answer:

(6)

where is modeled as a DNN. Note that the last executor, which is devoted to returning answers, carries out a specific kind of execution using based on the entry value, the query, and annotations from previous layer.

3.4 Handling Multiple Tables

Real-world KBs are often modeled by a schema involving various tables, where each table stores a specific type of factual information. We present Neural Enquirer-M, adapted for simultaneously operating on multiple KB tables. A key challenge in this scenario is that the multiplicity of tables requires modeling interaction between them. For example, Neural Enquirer-M needs to serve join queries, whose answer is derived by joining fields in different tables. Details of the modeling and experiments of Neural Enquirer-M are given in Appendix C.

4 Learning

Neural Enquirer can be trained in an end-to-end (N2N) fashion in Question Answering tasks. During training, both the representations of queries and table entries, as well as the execution logic captured by weights of executors are learned. More specifically, given a set of query-table-answer triples , we optimize the model parameters by maximizing the log-likelihood of gold-standard answers:

(7)

In end-to-end training, each executor discovers its operation logic from training data in a purely data-driven manner, which could be difficult for complicated queries requiring four or five sequential operations.

This can be alleviated by softly guiding the learning process via controlling the attention weights in Eq. (3). By enforcing to bias towards a field pertain to a specific operation, we can “coerce” the executor to figure out the logic of this particular operation relative to the field. As an example, for Executor-1 in Figure 1, by biasing the weight of the host_city field towards 1.0, only the value of host_city field will be fetched and sent for computing annotations, in this way we can force the executor to learn to find the row whose host_city is Beijing. This setting will be referred to as step-by-step (SbS) training. Formally, this is done by introducing additional supervision signal to Eq. (7):

(8)

where is a scalar and is the embedding of the field name known a priori to be relevant to the executor at Layer- in the -th example.

5 Experiments

In this section we evaluate Neural Enquirer on synthetic QA tasks with queries with varying compositional depth. We will first briefly describe our synthetic QA task for benchmark and experimental setup, and then discuss the results under different settings.

5.1 Synthetic QA Task

We present a synthetic QA task to evaluate the performance of Neural Enquirer, where a large amount of QA examples at various levels of complexity are generated to evaluate the single table and multiple tables cases of the model. Starting with “artificial” tasks eases the process of developing novel deep models [14], and has gained increasing popularity in recent advances of the research on modeling symbolic computation using DNNs [6, 15].

Our synthetic dataset consists of query-table-answer triples . To generate such a triple, we first randomly sample a table of size from a synthetic schema of Olympic Games, which has 10 fields, whose values are drawn from a vocabulary of size 240, with 120 country and city names, and 120 numbers. Figure 5 gives an example table with one row. Next, we generate a query using predefined templates associated with its gold-standard answer on . Our task consists of four types of natural language queries as summarized in Table 1, with annotated SQL-like logical forms for easy interpretation. We generate NL queries at various levels of compositionality to mimic the real world scenario. The complexity of those queries ranges from simple Select_Where queries to more complicated Nest ones involving multiple steps of computation. Those queries are flexible enough to involve complex matching between NL phrases and logical constituents, which makes query understanding and execution nontrivial: (1) the same field is described by different NL phrases (e.g., “How big is the country …” and “What is the size of the country …” for country_size field); (2) different fields may be referred to by the same NL pattern (e.g, in China” for country and in Beijing” for host_city); (3) simple NL constituents may be grounded to complex logical operations (e.g., after the game in Beijing” implies comparing between year fields). In our experiments we use the above procedure to generate benchmark datasets consisting of different types of queries. To make the artificial task harder, we enforce that all queries in the testing set do not appear in the training set.

To simulate the read-world scenario where queries of various types are issued to the model, we constructed two Mixtured datasets for our main experiments, with and training examples respectively, where four types of quires are sampled with the ratio . Both datasets share the same testing set of examples, for each type of query.

year host_city #_participants #_medals #_duration #_audience host_country GDP country_size population
2008 Beijing 4,200 2,500 30 67,000 China 2,300 960 130
Figure 5: An example table in the synthetic QA task (only show one row)
Query Type Example Queries with Annotated SQL-like Logical Forms
Select_Where : How many people participated in the game in Beijing?
  : select #_participants, where host_city = Beijing
: In which country was the game hosted in 2012?
  : select host_country, where year = 2012
Superlative : When was the lastest game hosted?
  : argmax(host_city, year)
: How big is the country which hosted the shortest game?
  : argmin(country_size, #_duration)
Where_Superlative : How long is the game with the most medals that has fewer than 3,000 participants?
  : where #_participants 3,000, argmax(#_duration, #_medals)
: How many medals are in the first game after 2008?
  : where #_year 2008, argmin(#_medals, #_year)
Nest : Which country hosted the longest game before the game in Athens?
  : where year(select year,where host_city=Athens),argmax(host_country,#_duration)
: How many people watched the earliest game that lasts for more days than the game in 1956?
  : where #_duration(select #_duration,where year=1956),argmin(#_audience,#_year)
Table 1: Example queries in our synthetic QA task

5.2 Setup

[Tuning] We use a Neural Enquirer with five executors. The number of layers for and are set to 2 and 3, respectively. We set the dimensionality of word/entity embeddings and row/table annotations to 20, hidden layers to 50, and the hidden states of the GRU in query encoder to 150. in Eq. (8

) is set to 0.2. We pad the beginning of all input queries to a fixed size.

Neural Enquirer is trained via standard back-propagation. Objective functions are optimized using SGD in a mini-batch of size 100 with adaptive learning rates (AdaDelta [16]

). The model converges fast within 100 epochs.

[Baseline] We compare our model with Sempre [11], a state-of-the-art semantic parser.

[Metric] We evaluate the performance of Neural Enquirer and Sempre (baseline) in terms of accuracy, defined as the fraction of correctly answered queries.

5.3 Main Results

Mixtured-25K Mixtured-100K
Sempre N2N SbS N2N - OOV N2N SbS N2N - OOV
Select_Where 93.8% 96.2% 99.7% 90.3% 99.3% 100.0% 97.6%
Superlative 97.8% 98.9% 99.5% 98.2% 99.9% 100.0% 99.7%
Where_Superlative 34.8% 80.4% 94.3% 79.1% 98.5% 99.8% 98.0%
Nest 34.4% 60.5% 92.1% 57.7% 64.7% 99.7% 63.9%
Overall Acc. 65.2% 84.0% 96.4% 81.3% 90.6% 99.9% 89.8%
Table 2: Accuracies on Mixtured datasets

Table 2 summarizes the results of Sempre (baseline) and our Neural Enquirer under end-to-end (N2N) and step-by-step (SbS) settings. We show both the individual performance for each type of query and the overall accuracy. We evaluate Sempre only on Mixtured-25K because of its long training time even on the smaller Mixtured-25K ( 3 days). We give discussion of efficiency issues in Appendix B.

We first discuss Neural Enquirer’s performance under end-to-end (N2N) training setting (the 3rd and 6th column in Table 2), and defer the discussion for SbS setting to Section 5.4. On Mixtured-25K, our model outperforms Sempre on all types of queries, with a marginal gain on simple queries (Select_Where, Superlative), and significant improvement on complex ones (Where_Superlative, Nest). When the size of training set grows (Mixtured-100K), Neural Enquirer achieves near 100% accuracy for the first three types of queries, while registering a decent overall accuracy of 90.6%. These results suggest that our model is very effective in answering compositional natural language queries, especially those with complex semantic structures compared with the state-of-the-art system.

: How long is the game with the most medals that has fewer than 3,000 participants?
Figure 6: Weights visualization of query
: Which country hosted the longest game before the game in Athens?
Figure 7: Weights visualization of query
: How many people watched the earliest game that lasts for more days than the game in 1956?
Figure 8: Weights visualization of query (an incorrectly answered query)

To further understand why our model is capable of handling compositional queries, we study the attention weights of Readers (Eq. 3) for executors in intermediate layers, and the answer probability (Eq. 6) the last executor outputs for each entry in the table. Those statistics are obtained from the model trained on Mixtured-100K. We sampled two queries ( and in Table 1) in the dataset that our model answers correctly and visualized their corresponding values, as illustrated in Figure 6 and 7, respectively. We find that each executor actually learns its execution logic from just the correct answers in end-to-end training, which corresponds with our assumption. For , the model executes the query in three steps, with each of the last three executors performs a specific type of operation. For each row, Executor-3 takes the value of the #_participants field as input and computes intermediate annotations, while Executor-4 focuses on the #_medals field. Finally, the last executor outputs high probability for the #_duration field (the 5-th column) in the 3-rd row. The attention weights for Executor-1 and Executor-2 appear to be meaningless because requires only three steps of execution, and the model learns to defer the meaningful execution to the last three executors. We can guess confidently that in executing , Executor-3 performs the conditional filtering operation (where clause in ), and Executor-4 performs the first part of argmax (finding the maximum value of #_medals), while the last executor finishes the execution by assigning high probability for the #_duration field of the row with the maximum value of #_medals.

Compared with the relatively simple , is more complicated, whose logical form involves a nest sub-query, and requires five steps of execution. From the weights visualized in figure 7, we can find that the last three executors function similarly as the case in answering

, yet the execution logic for the first two executors is a bit obscure. We posit that this is because during end-to-end training, the supervision signal propagated from the top layer has decayed along the long path down to the first two executors, which causes vanishing gradient problem.

We also investigate the case where our model fails to deliver the correct answer for complicated queries. Figure 8 gives such a query in Table 1 together with its visualized weights. Similar as , requires five steps of execution. Besides messing up the weights in the first two executors, the last executor, Executor-5, predicts a wrong entry as the query answer, instead of the highlighted (in red rectangle) correct entry.

5.4 With Additional Step-by-Step Supervision

: Which country hosted the longest game before the game in Athens?
Figure 9: Weights visualization of query in step-by-step training setting

To alleviate the vanishing gradient problem when training on complex queries as described in Section 5.3, in our next set of experiments we trained our Neural Enquirer model using step-by-step (SbS) training (Eq. 8), where we encourage each executor to attend to a specific field that is known a priori to be relevant to its execution logic. The results are shown in the 4-rd and 7-th columns of Table 2. With stronger supervision signal, the model significantly outperforms the results in end-to-end setting, and achieves near accuracy on all types of queries, which shows that our proposed Neural Enquirer is capable of leveraging the additional supervision signal given to intermediate layers in SbS training setting, and answering complex and compositional queries with perfect accuracy.

Let us revisit the query in SbS setting with the weights visualization in Figure 9. In contrast to the result in N2N setting (Figure 7

) where the attention weights for the first two executors are obscure, the weights in every executor are perfectly skewed towards the actual field pertain to each layer of execution (with a weight

). Quite interestingly, the attention weights for Executor-3 and Executor-4 are exactly the same with the result in N2N setting, while the weights for Executor-1 and Executor-2 are significantly different, suggesting Neural Enquirer learned a different execution logic in the SbS setting.

5.5 Dealing with Out-Of-Vocabulary Words

: How many people watched the game in Macau?
Figure 10: Weights visualization of query

One of the major challenges for applying neural network models to NLP applications is to deal with Out-Of-Vocabulary (OOV) words, which is particularly severe for QA. It is hard to cover existing tail entities, while at the same time new entities appear in user-issued queries and back-end KBs everyday. Quite interestingly, we find that a simple variation of Neural Enquirer is able to handle unseen entities almost without any loss of accuracy.

Basically, we divide words in the vocabulary into entity words and operation words. Embeddings of entity words (e.g., Beijing, China) function in a way as index to facilitate the matching between entities in queries and tables during the layer-by-layer execution, and do not need to be updated once initialized; while those of operation words, i.e., all non-entity words (e.g., numbers, longest, before, etc) carry semantic meanings relevant to execution and should be optimized during training. Therefore, after randomly initializing the embedding matrix , we only update the embeddings of operation words in training, while keeping those of entity words unchanged.

To test the model’s performance with OOV words, we modify queries in the testing portion of the Mixtured dataset to replace all entity words (i.e., all country and city names) with OOV ones333They also have embeddings in . unseen in the training set. Results obtained using N2N training are summarized in the 5th and 8th columns of Table 2. As it shows Neural Enquirer training in this OOV setting yields performance comparable to that in the non-OOV setting, indicating that operation words and entity words play different roles in query execution.

An interesting question to investigate in this OOV setting is how Neural Enquirer distinguishes between different types of entity words (i.e., cities and countries) in queries, since their embeddings are randomly initialized and fixed thereafter. An example query is “How many people watched the game in Macau?”, where Macau is an OOV entity. To help understand how the model knows Macau is a city, we give its weights visualization in Figure 10. Interestingly, the model first checks the host_city field in Executor-3, and then host_country in Executor-4, which suggests that the model learns to scan all possible fields where the OOV entity may belong to.

5.6 Simulating Large Knowledge Source

Query Type Select_Where Superlative Where_Superlative Overall
Accuracy 68.2% 84.8% 80.2% 77.7%
Table 3: Accuracies for large knowledge source simulation
Figure 11: Large knowledge source simulation

An important direction in semantic parsing research is to scale to large knowledge source [3, 11]. In this set of experiments we simulate a test case to evaluate Neural Enquirer’s ability to generalize to large knowledge source. We train a model on tables whose field sets are either , where (with ) is a subset of the entire set . We then test the model on tables with all fields and queries whose fields span multiple subsets . Figure 11 illustrates the setting. Note that all testing queries exhibit field combinations unseen in the training data, to mimic the difficulty the system often encounter when scaling to large knowledge source, which usually poses great challenge on model’s generalization ability. We then train and test the model only on a new dataset of the first three types of relatively simple queries (namely Select_Where, Superlative and Where_Superlative). The sizes of training/testing splits are 75,000 and 30,000, with equal numbers for different query types. Table 3 lists the results. Neural Enquirer still maintains a reasonable performance even when the compositionality of testing queries is previously unseen, showing the model’s generalization ability in tackling unseen query patterns through composition of familiar ones, and hence the potential to scale to larger and unseen knowledge sources.

6 Related Work

Our work falls into the research area of Semantic Parsing, where the key problem is to parse Natural Language queries into logical forms executable on KBs. Classical approaches for Semantic Parsing can be broadly divided into two categories. The first line of research resorts to the power of grammatical formalisms (e.g., Combinatory Categorial Grammar) to parse NL queries and generate corresponding logical forms, which requires curated/learned lexicons defining the correspondence between NL phrases and symbolic constituents 

[17, 7, 1, 18]

. The model is tuned with annotated logical forms, and is capable of recovering complex semantics from data, but often constrained on a specific domain due to scalability issues brought by the crisp grammars and the lack of annotated training data. Another line of research takes a semi-supervised learning approach, and adopts the results of query execution (i.e., answers) as supervision signal 

[5, 3, 10, 11, 12]. The parsers, designed towards this new learning paradigm, take different types of forms, ranging from generic chart parsers [3, 11] to more specifically engineered, task-oriented ones [12, 8]. Semantic parsers in this category often scale to open domain knowledge sources, but lack the ability of understanding compositional queries because of the intractable search space incurred by the flexibility of parsing algorithms. Our work follows this line of research in using query answers as indirect supervision to facilitate end-to-end training using QA tasks, but performs semantic parsing in distributional spaces, where logical forms are “neuralized” to an executable distributed representation.

Our work is also related to the recent advances of modeling symbolic computation using Deep Neural Networks. Pioneered by the development of Neural Turing Machines (NTMs) [6], this line of research studies the problem of using differentiable neural networks to perform “hard” symbolic execution. As an independent line of research with similar flavor, Zaremba et al. [15] designed a LSTM-RNN to execute simple Python programs, where the parameters are learned by comparing the neural network output and the correct answer. Our work is related to both lines of work, in that like NTM, we heavily use external memory and flexible way of processing (e.g., the attention-based reading in the operations in Reader) and like  [15], Neural Enquirer learns to execute a sequence with complicated structure, and the model is tuned from the executing them. As a highlight and difference from the previous work, we have a deep architecture with multiple layer of external memory, with the neural network operations highly customized to querying KB tables.

Perhaps the most related work to date is the recently published Neural Programmer proposed by Neelakantan et al. [9], which studies the same task of executing queries on tables with Deep Neural Networks. Neural Programmer uses a neural network model to select operations during query processing. While the query planning (i.e., which operation to execute at each time step) phase is modeled softly using neural networks, the symbolic operations are predefined by users. In contrast Neural Enquirer is fully distributional: it models both the query planning and the operations with neural networks, which are jointly optimized via end-to-end training. Our Neural Enquirer model learns symbolic operations using data-driven approach, and demonstrates that a fully neural, end-to-end differentiable system is capable of modeling and executing compositional arithmetic and logic operations upto certain level of complexity.

7 Conclusion and Future Work

In this paper we propose Neural Enquirer, a fully neural, end-to-end differentiable network that learns to execute queries on tables. We present results on a set of synthetic QA tasks to demonstrate the ability of Neural Enquirer to answer fairly complicated compositional queries across multiple tables. In the future we plan to advance this work in the following directions. First we will apply Neural Enquirer to natural language questions and natural language answers, where both the input query and the output supervision are noisier and less informative. Second, we are going to scale to real world QA task as in [11], for which we have to deal with a large vocabulary and novel predicates. Third, we are going to work on the computational efficiency issue in query execution by heavily borrowing the symbolic operation.

References

  • [1] Y. Artzi, K. Lee, and L. Zettlemoyer. Broad-coverage ccg semantic parsing with amr. In EMNLP, pages 1699–1710, 2015.
  • [2] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015.
  • [3] J. Berant, A. Chou, R. Frostig, and P. Liang. Semantic parsing on freebase from question-answer pairs. In EMNLP, pages 1533–1544, 2013.
  • [4] A. Bordes, N. Usunier, A. García-Durán, J. Weston, and O. Yakhnenko. Translating embeddings for modeling multi-relational data. In NIPS, pages 2787–2795, 2013.
  • [5] D. L. Chen and R. J. Mooney. Learning to sportscast: a test of grounded language acquisition. In ICML, pages 128–135, 2008.
  • [6] A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. CoRR, abs/1410.5401, 2014.
  • [7] T. Kwiatkowski, E. Choi, Y. Artzi, and L. S. Zettlemoyer. Scaling semantic parsers with on-the-fly ontology matching. In EMNLP, pages 1545–1556, 2013.
  • [8] D. K. Misra, K. Tao, P. Liang, and A. Saxena. Environment-driven lexicon induction for high-level instructions. In ACL (1), pages 992–1002, 2015.
  • [9] A. Neelakantan, Q. V. Le, and I. Sutskever. Neural Programmer: Inducing Latent Programs with Gradient Descent. ArXiv e-prints, Nov. 2015.
  • [10] P. Pasupat and P. Liang. Zero-shot entity extraction from web pages. In ACL (1), pages 391–401, 2014.
  • [11] P. Pasupat and P. Liang. Compositional semantic parsing on semi-structured tables. In ACL (1), pages 1470–1480, 2015.
  • [12] W. tau Yih, M.-W. Chang, X. He, and J. Gao. Semantic parsing via staged query graph generation: Question answering with knowledge base. In ACL (1), pages 1321–1331, 2015.
  • [13] T.-H. Wen, M. Gasic, N. Mrksic, P. hao Su, D. Vandyke, and S. J. Young.

    Semantically conditioned lstm-based natural language generation for spoken dialogue systems.

    In EMNLP, pages 1711–1721, 2015.
  • [14] J. Weston, A. Bordes, S. Chopra, and T. Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. CoRR, abs/1502.05698, 2015.
  • [15] W. Zaremba and I. Sutskever. Learning to execute. CoRR, abs/1410.4615, 2014.
  • [16] M. D. Zeiler. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701, 2012.
  • [17] L. S. Zettlemoyer and M. Collins. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In UAI, pages 658–666, 2005.
  • [18] L. S. Zettlemoyer and M. Collins. Online learning of relaxed ccg grammars for parsing to logical form. In EMNLP-CoNLL, pages 678–687, 2007.

Appendix A Computation of Query Encoder

We use a bidirectional RNN as the Query Encoder, which consists of a forward GRU and a backward GRU. Given the sequence of word embeddings of : , at each time step , the forward GRU computes the hidden state as follows:

where , , , , , are parametric matrices, the column vector of all ones, and element-wise multiplication. The backward GRU reads the sequence in reverse order. We concatenate the last hidden states given by the two GRUs as the vectorial representation of the query.

Appendix B Efficiency of Model Learning

We compared the efficiency of Neural Enquirer and Sempre (baseline) in training by plotting the accuracy on testing data by training time. Figure 12 illustrates the results. We train Neural Enquirer-CPU and Sempre on a machine with Intel Core i7-3770@3.40GHz and 16GB memory, while Neural Enquirer-GPU is tuned on Nvidia Tesla K40. Neural Enquirer-CPU is 10 times faster than Sempre, and Neural Enquirer-GPU is 100 times faster.

Figure 12: Accuracy on Testing by Training Time

Appendix C Handling Multiple Tables

c.1 Neural Enquirer-M Model

Figure 13: Executor- and Executor- in multiple tables case
Figure 14: Illustration of Annotator for multiple tables case

Basically, Neural Enquirer-M assigns an executor to each table in every execution layer , denoted as Executor-. Figure 13 pictorially illustrates Executor- and Executor- working on Table-1 and Table-K respectively. Within each executor, the Reader is designed the same way as single table case, while we modify the Annotator to let in the information from other tables. More specifically, for Executor-, we modify its Annotator by extending Eq. (4) to leverage computational results from other tables when computing the annotation for the -th row:

This process is illustrated in Figure 14. Note that we add subscripts to the notations to index tables. To model the interaction between tables, the Annotator incorporates the “relevant” row annotation, , and the “relevant” table annotation, derived from the previous execution results of other tables when computing the current row annotation.

A relevant row annotation stores the data fetched from row annotations of other tables, while a relevant table annotation summarizes the table-wise execution results from other tables. We now describe how to compute those annotations. First, for each table , , we fetch a relevant row annotation from all row annotations of via attentive reading:

Intuitively, the attention weight (modeled by a DNN) captures how important the -th row annotation from table , is with respect to the current step of execution. After getting the set of row annotations fetched from all other tables, , we then compute and via a pooling operation444This operation is trivial in our experiments on two tables. on and :

In summary, relevant row and table annotations encode the local and global computational results from other tables. By incorporating them into calculating row annotations, Neural Enquirer-M is capable of answering queries that involve interaction between multiple tables.

Finally, Neural Enquirer-M outputs the ranked list of answer probabilities by normalizing the value of in Eq. (6) over each entry for very table.

c.2 Experimental Results

country year host_city #_participants #_medals #_duration #_audience
China 2008 Beijing 3,500 4,200 30 67,000
Table-2
country continent population country_size
China Asia 130 960
Figure 15: Multiple tables example in the synthetic QA task (only show one row)
Table-1

We present preliminarily results for Neural Enquirer-M, which we evaluated on SQL-like Select_Where logical forms (like , in Table 1). We sampled a dataset of examples, with each example having two tables as in Figure 15. Out of all Select_Where queries, roughly half of the queries (denoted as “Join”) require joining the two tables to derive answers. We tested on a model with three executors. Table 4 lists the results. The accuracy of join queries is lower than that of non-join queries, which is caused by additional interaction between the two tables involved in answering join queries.

Query Type Non-Join Join Overall
Accuracy
Table 4: Accuracies of Select_Where queries on two tables
: select country_size, where year = 2012
Executor-
Executor-
Executor-
Executor-
Executor-
Executor-
Figure 16: Weights visualization of query

We find that Neural Enquirer-M is capable of identifying that the country field is the foreign key linking the two tables. Figure 16 illustrates the attention weights for a correctly answered join query . Although the query does not contain any hints for the foreign key (country field), Executor- (the executor at Layer-1 on Table-1) operates on an ensemble of embeddings of the country and year fields, whose outputting row annotations (contain information of both the key country and the value year) are sent to Executor- to compare with the country field in Table-2. We posit that the result of comparison is stored in the row annotations of Executor- and subsequently sent to the executors at Layer-3 for computing the answer probability for each entry in the two tables.