Probabilistic Reasoning via Deep Learning: Neural Association Models

03/24/2016 ∙ by Quan Liu, et al. ∙ York University USTC Anhui USTC iFLYTEK Co 0

In this paper, we propose a new deep learning approach, called neural association model (NAM), for probabilistic reasoning in artificial intelligence. We propose to use neural networks to model association between any two events in a domain. Neural networks take one event as input and compute a conditional probability of the other event to model how likely these two events are to be associated. The actual meaning of the conditional probabilities varies between applications and depends on how the models are trained. In this work, as two case studies, we have investigated two NAM structures, namely deep neural networks (DNN) and relation-modulated neural nets (RMNN), on several probabilistic reasoning tasks in AI, including recognizing textual entailment, triple classification in multi-relational knowledge bases and commonsense reasoning. Experimental results on several popular datasets derived from WordNet, FreeBase and ConceptNet have all demonstrated that both DNNs and RMNNs perform equally well and they can significantly outperform the conventional methods available for these reasoning tasks. Moreover, compared with DNNs, RMNNs are superior in knowledge transfer, where a pre-trained model can be quickly extended to an unseen relation after observing only a few training samples. To further prove the effectiveness of the proposed models, in this work, we have applied NAMs to solving challenging Winograd Schema (WS) problems. Experiments conducted on a set of WS problems prove that the proposed models have the potential for commonsense reasoning.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.


Reasoning is an important topic in artificial intelligence (AI), which has attracted considerable attention and research effort in the past few decades [McCarthy1986, Minsky1988, Mueller2014]

. Besides the traditional logic reasoning, probabilistic reasoning has been studied as another typical genre in order to handle knowledge uncertainty in reasoning based on probability theory

[Pearl1988, Neapolitan2012]. The probabilistic reasoning can be used to predict conditional probability of one event given another event

. State-of-the-art methods for probabilistic reasoning include Bayesian Networks

[Jensen1996], Markov Logic Networks [Richardson and Domingos2006] and other graphical models [Koller and Friedman2009]

. Taking Bayesian networks as an example, the conditional probabilities between two associated events are calculated as posterior probabilities according to Bayes theorem, with all possible events being modeled by a pre-defined graph structure. However, these methods quickly become intractable for most practical tasks where the number of all possible events is usually very large.

In recent years, distributed representations that map discrete language units into continuous vector space have gained significant popularity along with the development of neural networks

[Bengio et al.2003, Collobert et al.2011, Mikolov et al.2013]

. The main benefit of embedding in continuous space is its smoothness property, which helps to capture the semantic relatedness between discrete events, potentially generalizable to unseen events. Similar ideas, such as knowledge graph embedding, have been proposed to represent knowledge bases (KB) in low-dimensional continuous space

[Bordes et al.2013, Socher et al.2013, Wang et al.2014, Nickel et al.2015]. Using the smoothed KB representation, it is possible to reason over the relations among various entities. However, human-like reasoning remains as an extremely challenging problem partially because it requires the effective encoding of world knowledge using powerful models. Most of the existing KBs are quite sparse and even recently created large-scale KBs, such as YAGO, NELL and Freebase, can only capture a fraction of world knowledge. In order to take advantage of these sparse knowledge bases, the state-of-the-art approaches for knowledge graph embedding usually adopt simple linear models, such as RESCAL [Nickel, Tresp, and Kriegel2012], TransE [Bordes et al.2013]

and Neural Tensor Networks

[Socher et al.2013, Bowman2013].

Although deep learning techniques achieve great progresses in many domains, e.g. speech and image [LeCun, Bengio, and Hinton2015], the progress in commonsense reasoning seems to be slow. In this paper, we propose to use deep neural networks, called neural association model (NAM)

, for commonsense reasoning. Different from the existing linear models, the proposed NAM model uses multi-layer nonlinear activations in deep neural nets to model the association conditional probabilities between any two possible events. In the proposed NAM framework, all symbolic events are represented in low-dimensional continuous space and there is no need to explicitly specify any dependency structure among events as required in Bayesian networks. Deep neural networks are used to model the association between any two events, taking one event as input to compute a conditional probability of another event. The computed conditional probability for association may be generalized to model various reasoning problems, such as entailment inference, relational learning, causation modelling and so on. In this work, we study two model structures for NAM. The first model is a standard deep neural networks (DNN) and the second model uses a special structure called relation modulated neural nets (RMNN). Experiments on several probabilistic reasoning tasks, including recognizing textual entailment, triple classification in multi-relational KBs and commonsense reasoning, have demonstrated that both DNNs and RMNNs can outperform other conventional methods. Moreover, the RMNN model is shown to be effective in knowledge transfer learning, where a pre-trained model can be quickly extended to a new relation after observing only a few training samples.

Furthermore, we also apply the proposed NAM models to more challenging commonsense reasoning problems, i.e., the recently proposed Winograd Schemas (WS) [Levesque, Davis, and Morgenstern2011]. The WS problems has been viewed as an alternative to the Turing Test [Turing1950]. To support the model training for NAM, we propose a straightforward method to collect associated cause-effect pairs from large unstructured texts. The pair extraction procedure starts from constructing a vocabulary with thousands of common verbs and adjectives. Based on the extracted pairs, this paper extends the NAM models to solve the Winograd Schema problems and achieves a 61% accuracy on a set of cause-effect examples. Undoubtedly, to realize commonsense reasoning, there is still much work be done and many problems to be solved. Detailed discussions would be given at the end of this paper.

Motivation: Association between Events

This paper aims to model the association relationships between events using neural network methods. To make clear our main work, we will first describe the characteristics of events and all the possible association relationships between events. Based on the analysis of event association, we present the motivation for the proposed neural association models. In commonsense reasoning, the main characteristics of events are the following:

  • Massive: In most natural situations, the number of events is massive, which means that the association space we will model is very large.

  • Sparse: All the events occur in our dialy life are very sparse. It is a very challenging task to ideally capture the similarities between all those different events.

At the same time, association between events appears everywhere. Consider a single event play basketball for example, shown in Figure 1. This single event would associate with many other events. A person who plays basketball would win a game. Meanwhile, he would be injured in some cases. The person could make money by playing basketball as well. Moreover, we know that a person who plays basketball should be coached during a regular game. Those are all typical associations between events. However, we need to recognize that the task of modeling event association is not identical to performing classification. In classification, we typically map an event from its feature space into one of pre-defined finite categories or classes. In event association, we need to compute the association probability between two arbitrary events, each of which may be a sample from a possibly infinite set. The mapping relationships in event association would be many-to-many; e.g., not only playing basketball could support us to make money, someone who makes stock trading could make money as well. More specifically, the association relationships between events include cause-effect, spatial, temporal and so on. This paper treats them as a general relation considering the sparseness of useful KBs.

Figure 1: Example of association between events.

In this paper, we believe that modeling the the association relationships between events is a fundamental work for commonsense reasoning. If we could model the event associations very well, we may have the ability to solve many commonsense reasoning problems. Considering the main characteristics of discrete event and event association, two reasons are given for describing our motivation.

  • The advantage of distributed representation methods: representing discrete events into continuous vector space provides a good way to capture the similarities between discrete events.

  • The advantage of neural network methods: neural networks could perform universal approximation while linear models cannot easily do this [Hornik, Stinchcombe, and White1990].

At the same time, this paper takes into account that both distributed representation and neural network methods are data-hungry. In Artificial Intelligence (AI) research, mining large sizes of useful data (or knowledge) for model learning is always challenging. In the following section, this paper presents a preliminary work on data collection and the corresponding experiments we have made for solving commonsense reasoning problems.

Neural Association Models (NAM)

In this paper, we propose to use a nonlinear model, namely neural association model, for probabilistic reasoning. Our main goal is to use neural nets to model the association probability for any two events and in a domain, i.e., of conditioning on . All possible events in the domain are projected into continuous space without specifying any explicit dependency structure among them. In the following, we first introduce neural association models (NAM) as a general modeling framework for probabilistic reasoning. Next, we describe two particular NAM structures for modeling the typical multi-relational data.

NAM in general

Figure 2: The NAM framework in general.

Figure 2 shows the general framework of NAM for associating two events, and . In the general NAM framework, the events are first projected into a low-dimension continuous space. Deep neural networks with multi-layer nonlinearity are used to model how likely these two events are to be associated. Neural networks take the embedding of one event (antecedent) as input and compute a conditional probability of the other event (consequent). If the event is binary (true or false), the NAM models may use a sigmoid node to compute . If takes multiple mutually exclusive values, we use a few softmax nodes for , where it may need to use multiple embeddings for (one per value). NAMs do not explicitly specify how different events are actually related; they may be mutually exclusive, contained, intersected. NAMs are only used to separately compute conditional probabilities, , for each pair of events, and , in a task. The actual physical meaning of the conditional probabilities varies between applications and depends on how the models are trained. Table 1 lists a few possible applications.

language modeling
causal reasoning cause effect
knowledge triple classification
lexical entailment
textual entailment
Table 1: Some applications for NAMs.

In language modeling, the antecedent event is the representation of historical context, , and the consequent event is the next word that takes one out of K values. In causal reasoning, and represent cause and effect respectively. For example, we have = “eating cheesy cakes” and = “being happy”, where indicates how likely it is that may cause the binary (true or false) event . In the same model, we may add more nodes to model different effects from the same , e.g., = “growing fat”. Moreover, we may add 5 softmax nodes to model a multi-valued event, e.g., = “happiness” (scale from 1 to 5). Similarly, for knowledge triple classification of multi-relation data, given one triple , consists of the head entity (subject) and relation (predicate) , and is a binary event indicating whether the tail entity (object) is true or false. Finally, in the applications of recognizing lexical or textual entailment, and may be defined as premise and hypothesis. More generally, NAMs can be used to model an infinite number of events , where each point in a continuous space represents a possible event. In this work, for simplicity, we only consider NAMs for a finite number of binary events but the formulation can be easily extended to more general cases.

Compared with traditional methods, like Bayesian networks, NAMs employ neural nets as a universal approximator to directly model individual pairwise event association probabilities without relying on explicit dependency structure. Therefore, NAMs can be end-to-end learned purely from training samples without strong human prior knowledge, and are potentially more scalable to real-world tasks.

Learning NAMs

Assume we have a set of observed examples (event pairs ), , each of which is denoted as . This training set normally includes both positive and negative samples. We denote all positive samples () as and all negative samples () as . Under the same independence assumption as in statistical relational learning (SRL) [Getoor2007, Nickel et al.2015], the log likelihood function of a NAM model can be expressed as follows:


where denotes a logistic score function derived by the NAM for each , which numerically computes the conditional probability . More details on

will be given later in the paper. Stochastic gradient descent (SGD) methods may be used to maximize the above likelihood function, leading to a maximum likelihood estimation (MLE) for NAMs.

In the following, as two case studies, we consider two NAM structures with a finite number of output nodes to model for any pair of events, where we have only a finite number of and each is binary. The first model is a typical DNN that associates antecedent event () at input and consequent event () at output. We then present another model structure, called relation-modulated neural nets, which is more suitable for multi-relational data.

DNN for NAMs

The first NAM structure is a traditional DNN as shown in Figure 3. Here we use multi-relational data in KB for illustration. Given a KB triple and its corresponding label (true or false), we cast and to compute as follows.

Figure 3: The DNN structure for NAMs.

Firstly, we represent head entity phrase and tail entity phrase by two embedding vectors and . Similarly, relation is also represented by a low-dimensional vector , which we call a relation code hereafter. Secondly, we combine the embeddings of the head entity and the relation to feed into an -layer DNN as input. The DNN consists of

rectified linear (ReLU) hidden layers

[Nair and Hinton2010]. The input is . During the feedforward process, we have


where and represent the weight matrix and bias for layer respectively.

Finally, we propose to calculate a sigmoid score for each triple as the association probability using the last hidden layer’s output and the tail entity vector :


where is the sigmoid function, i.e., .

All network parameters of this NAM structure, represented as , may be jointly learned by maximizing the likelihood function in eq. (1).

Relation-modulated Neural Networks (RMNN)

Particularly for multi-relation data, following the idea in [Xue et al.2014], we propose to use the so-called relation-modulated neural nets (RMNN), as shown in Figure 4.

Figure 4: The relation-modulated neural networks (RMNN).

The RMNN uses the same operations as DNNs to project all entities and relations into low-dimensional continuous space. As shown in Figure 4, we connect the knowledge-specific relation code to all hidden layers in the network. As shown later, this structure is superior in knowledge transfer learning tasks. Therefore, for each layer of RMNNs, instead of using eq.(2), its linear activation signal is computed from the previous layer and the relation code as follows:


where and represent the normal weight matrix and the relation-specific weight matrix for layer . At the topmost layer, we calculate the final score for each triple using the relation code as:


In the same way, all RMNN parameters, including , can be jointly learned based on the above maximum likelihood estimation.

The RMNN models are particularly suitable for knowledge transfer learning, where a pre-trained model can be quickly extended to any new relation after observing a few samples from that relation. In this case, we may estimate a new relation code based on the available new samples while keeping the whole network unchanged. Due to its small size, the new relation code can be reliably estimated from only a small number of new samples. Furthermore, model performance in all original relations will not be affected since the model and all original relation codes are not changed during transfer learning.


In this section, we evaluate the proposed NAM models for various reasoning tasks. We first describe the experimental setup and then we report the results from several reasoning tasks, including textual entailment recognition, triple classification in multi-relational KBs, commonsense reasoning and knowledge transfer learning.

Experimental setup

Here we first introduce some common experimental settings used for all experiments: 1) For entity or sentence representations, we represent them by composing from their word vectors as in [Socher et al.2013]. All word vectors are initialized from a pre-trained skip-gram [Mikolov et al.2013]

word embedding model, trained on a large English Wikipedia corpus. The dimensions for all word embeddings are set to 100 for all experiments; 2) The dimensions of all relation codes are set to 50. All relation codes are randomly initialized; 3) For network structures, we use ReLU as the nonlinear activation function and all network parameters are initialized according to

[Glorot and Bengio2010]. Meanwhile, since the number of training examples for most probabilistic reasoning tasks is relatively small, we adopt the dropout approach [Hinton et al.2012] during the training process to avoid the over-fitting problem; 4) During the learning process of NAMs, we need to use negative samples, which are automatically generated by randomly perturbing positive KB triples as .

For each task, we use the provided development set to tune for the best training hyperparameters. For example, we have tested the number of hidden layers among {1, 2, 3}, the initial learning rate among {0.01, 0.05, 0.1, 0.25, 0.5}, dropout rate among {0, 0.1, 0.2, 0.3, 0.4}. Finally, we select the best setting based on the performance on the development set: the final model structure uses 2 hidden layers, and the learning rate and the dropout rate are set to be 0.1 and 0.2, respectively, for all the experiments. During model training, the learning rate is halved once the performances in the development set decreases. Both DNNs and RMNNs are trained using the stochastic gradient descend (SGD) algorithm. We notice that the NAM models converge quickly after 30 epochs.

Recognizing Textual Entailment

Understanding entailment and contradiction is fundamental to language understanding. Here we conduct experiments on a popular recognizing textual entailment (RTE) task, which aims to recognize the entailment relationship between a pair of English sentences. In this experiment, we use the SNLI dataset in [Bowman et al.2015] to conduct 2-class RTE experiments (entailment or contradiction). All instances that are not labelled as “entailment” are converted to contradiction in our experiments. The SNLI dataset contains hundreds of thousands of training examples, which is useful for training a NAM model. Since this data set does not include multi-relational data, we only investigate the DNN structure for this task. The final NAM result, along with the baseline performance provided in [Bowman et al.2015], is listed in Table 2.

Model Accuracy (%)
Edit Distance [Bowman et al.2015] 71.9
Classifier [Bowman et al.2015] 72.2
Lexical Resources [Bowman et al.2015] 75.0
DNN 84.7
Table 2: Experimental results on the RTE task.

From the results, we can see the proposed DNN based NAM model achieves considerable improvements over various traditional methods. This indicates that we can better model entailment relationship in natural language by representing sentences in continuous space and conducting probabilistic reasoning with deep neural networks.

Triple classification in multi-relational KBs

In this section, we evaluate the proposed NAM models on two popular knowledge triple classification datasets, namely WN11 and FB13 in [Socher et al.2013] (derived from WordNet and FreeBase), to predict whether some new triple relations hold based on other training facts in the database. The WN11 dataset contains 38,696 unique entities involving 11 different relations in total while the FB13 dataset covers 13 relations and 75,043 entities. Table 3 summarizes the statistics of these two datasets.

Dataset # R # Ent # Train # Dev # Test
WN11 11 38,696 112,581 2,609 10,544
FB13 13 75,043 316,232 5,908 23,733
Table 3: The statistics for KBs triple classification datasets. #R is the number of relations. #Ent is the size of the entity set.

The goal of knowledge triple classification is to predict whether a given triple is correct or not. We first use the training data to learn NAM models. Afterwards, we use the development set to tune a global threshold to make a binary decision: the triple is classified as true if ; otherwise it is false. The final accuracy is calculated based on how many triplets in the test set are classified correctly.

Experimental results on both WN11 and FB13 datasets are given in Table 4, where we compare the two NAM models with all other methods reported on these two datasets. The results clearly show that the NAM methods (DNNs and RMNNs) achieve comparable performance on these triple classification tasks, and both yield consistent improvement over all existing methods. In particular, the RMNN model yields 3.7% and 1.9% absolute improvements over the popular neural tensor networks (NTN) [Socher et al.2013] on WN11 and FB13 respectively. Both DNN and RMNN models are much smaller than NTN in the number of parameters and they scale well as the number of relation types increases. For example, both DNN and RMNN models for WN11 have about 7.8 millions of parameters while NTN has about 15 millions. Although the RESCAL and TransE models have about 4 millions of parameters for WN11, their size goes up quickly for other tasks of thousands or more relation types. In addition, the training time of DNN and RMNN is much shorter than that of NTN or TransE since our models converge much faster. For example, we have obtained at least a 5 times speedup over NTN in WN11.

Model WN11 FB13 Avg.
SME [Bordes et al.2012] 70.0 63.7 66.9
TransE [Bordes et al.2013] 75.9 81.5 78.7
TransH [Wang et al.2014] 78.8 83.3 81.1
TransR [Lin et al.2015] 85.9 82.5 84.2
NTN [Socher et al.2013] 86.2 90.0 88.1
DNN 89.3 91.5 90.4
RMNN 89.9 91.9 90.9
Table 4: Triple classification accuracy in WN11 and FB13.

Commonsense Reasoning

Similar to the triple classification task [Socher et al.2013], in this work, we use the ConceptNet KB [Liu and Singh2004] to construct a new commonsense data set, named as CN14 hereafter. When building CN14, we first select all facts in ConceptNet related to 14 typical commonsense relations, e.g., UsedFor, CapableOf. (see Figure 5 for all 14 relations.) Then, we randomly divide the extracted facts into three sets, Train, Dev and Test. Finally, in order to create a test set for classification, we randomly switch entities (in the whole vocabulary) from correct triples and get a total of 2#Test triples (half are positive samples and half are negative examples). The statistics of CN14 are given in Table 5.

Dataset # R # Ent. # Train # Dev # Test
CN14 14 159,135 200,198 5,000 10,000
Table 5: The statistics for the CN14 dataset.

The CN14 dataset is designed for answering commonsense questions like Is a camel capable of journeying across desert? The proposed NAM models answer this question by calculating the association probability where and . In this paper, we compare two NAM methods with the popular NTN method in [Socher et al.2013] on this data set and the overall results are given in Table 6. We can see that both NAM methods outperform NTN in this task, and the DNN and RMNN models obtain similar performance.

Model Positive Negative total
NTN 82.7 86.5 84.6
DNN 84.5 86.9 85.7
RMNN 85.1 87.1 86.1
Table 6: Accuracy (in %) comparison on CN14.

Figure 5: Accuracy of different relations in CN14.

Furthermore, we show the classification accuracy of all 14 relations in CN14 for RMNN and NTN in Figure 5, which show that the accuracy of RMNN varies among different relations from 80.1% (Desires) to 93.5% (CreatedBy). We notice some commonsense relations (such as Desires, CapableOf) are harder than the others (like CreatedBy, CausesDesire). RMNN overtakes NTN in almost all relations.

Knowledge Transfer Learning

Figure 6: Accuracy (in %) on the test set of a new relation CausesDesire is shown as a function of used training samples from CausesDesire when updating the relation code only. (Accuracy on the original relations remains as 85.7%.)

Knowledge transfer between various domains is a characteristic feature and crucial cornerstone of human learning. In this section, we evaluate the proposed NAM models for a knowledge transfer learning scenario, where we adapt a pre-trained model to an unseen relation with only a few training samples from the new relation. Here we randomly select a relation, e.g., CausesDesire in CN14 for this experiment. This relation contains only 4800 training samples and 480 test samples. During the experiments, we use all of the other 13 relations in CN14 to train baseline NAM models (both DNN and RMNN). During the transfer learning, we freeze all NAM parameters, including all weights and entity representations, and only learn a new relation code for CausesDesire from the given samples. At last, the learned relation code (along with the original NAM models) is used to classify the new samples of CausesDesire in the test set. Obviously, this transfer learning does not affect the model performance in the original 13 relations because the models are not changed. Figure 6 shows the results of knowledge transfer learning for the relation CausesDesire as we increase the training samples gradually. The result shows that RMNN performs much better than DNN in this experiment, where we can significantly improve RMNN for the new relation with only 5-20% of the total training samples for CausesDesire. This demonstrates that the structure to connect the relation code to all hidden layers leads to more effective learning of new relation codes from a relatively small number of training samples.

Next, we also test a more aggressive learning strategy for this transfer learning setting, where we simultaneously update all the network parameters during the learning of the new relation code. The results are shown in Figure 7. This strategy can obviously improve performance more on the new relation, especially when we add more training samples. However, as expected, the performance on the original 13 relations deteriorates. The DNN improves the performance on the new relation as we use all training samples (up to 94.6%). However, the performance on the remaining 13 original relations drops dramatically from 85.6% to 75.5%. Once again, RMNN shows an advantage over DNN in this transfer learning setting, where the accuracy on the new relation increases from 77.9% to 90.8% but the accuracy on the original 13 relations only drop slightly from 85.9% to 82.0%.

Figure 7: Transfer learning results by updating all network parameters. The left figure shows results on the new relation while the right figure shows results on the original relations.

Extending NAMs for Winograd Schema

Data Collection

In the previous experiments sections, all the tasks already contained manually constructed training data for us. However, in many cases, if we want to realize flexible commonsense reasoning under the real world conditions, obtaining the training data can also be very challenging. More specifically, since the proposed neural association model is a typical deep learning technique, lack of training data would make it difficult for us to train a robust model. Therefore, in this paper, we make some efforts and try to mine useful data for model training. As a very first step, we are now working on collecting the cause-effect relationships between a set of common words and phrases. We believe this type of knowledge would be a key component for modeling the association relationships between discrete events.

This section describes the idea for automatic cause-effect pair collection as well as the data collection results. We will first introduce the common vocabulary we created for query generation. After that, the detailed algorithm for cause-effect pair collection will be presented. Finally, the following section will present the data collection results.

Common Vocabulary and Query Generation

To avoid the data sparsity problem, we start our work by constructing a vocabulary of very common words. In our current investigations, we construct a vocabulary which contains 7500 verbs and adjectives. As shown in Table 7, this vocabulary includes 3000 verb words, 2000 verb phrases and 2500 adjective words. The procedure for constructing this vocabulary is straightforward. We first extract all words and phrases (divided by part-of-speech tags) from WordNet [Miller1995]. After conducting part-of-speech tagging on a large corpus, we then get the occurrence frequencies for all those words and phrases by scanning over the tagged corpus. Finally, we sort those words and phrases by frequency and then select the top N results.

Set Category Size
1 Verb words 3000
2 Verb phrases 2000
3 Adjective words 2500
Table 7: Common vocabulary constructed for mining cause-effect event pairs.

Query Generation

Based on the common vocabulary, we generate search queries by pairing any two words (or phrases). Currently we only focus on extracting the association relationships between verbs and adjectives. Even for this small vocabulary, the search space is very large (7.5K by 7.5K leads to tens of millions pairs). In this work, we define several patterns for each word or phrase based on two popular semantic dimensions: 1) positive-negative, 2) active-passive [Osgood1952]. Using the verbs rob and arrest for example, each of them contains 4 patterns, i.e. (active, positive), (active, negative), (passive, positive) and (passive, negative). Therefore, the query formed by rob and arrest would contain 16 possible dimensions, as shown in Figure 8. The task of mining the cause-effect relationships for any two words or phrases then becomes the task of getting the number of occurrences for all the possible links.

Figure 8: Typical 16 dimensions for a typical query.

Automatic Cause-Effect Pair Collection

Based on the created queries, in this section, we present the procedures for extracting cause-effect pairs from large unstructured texts. The overall system framework is shown in Figure 9.

Figure 9: Automatic pair collection system framework.

Query Searching

The goal of query searching is to find all the possible sentences that may contain the input queries. Since the number of queries is very large, we structure all the queries as a hashmap and conduct string matching during text scanning. In detail, the searching program starts by conducting lemmatizing, part-of-speech tagging and dependency parsing on the source corpus. After it, we scan the corpus from the begining to end. When dealing with each sentence, we will try to find the matched words (or phrases) using the hashmap. This strategy help us to reduce the search complexity to be linear with the size of corpus, which has been proved to be very efficient in our experiments.

Subject-Object Matching

Based on the dependency parsing results, once we find one phrase of a query, we would check whether that phrase is associated with at least one subject or object in the corresponding sentence or not. At the same time, we record whether the phrase was positive or negative, active or passive. Moreover, for helping us to decide the cause-effect relationships, we would check whether the phrase is linked with some connective words or not. Typical connective words used in this work are because and if. To finally extract the cause-effect pairs, we design a simple subject-object matching rule, which is similar to the work of [Peng, Khashabi, and Roth2015]. 1) If the two phrases in one query share the same subject, the relationship between them is then straightforward; 2) If the subject of one phrase is the object of the other phrase, then we need to apply the passive pattern to the phrase related to the object. This subject-object matching idea is similar to the work proposed in [Peng, Khashabi, and Roth2015]. Using query (arrest, rob) as an example. Once we find sentence “Tom was arrested because Tom robbed the man”, we obtain its dependency parsing result as shown in Figure 10. The verb arrest and rob share a same subject, and the pattern for arrest is passive, we will add the occurrence of the corresponding association link, i.e. link from the (active,positive) pattern of rob to the (passive,positive) pattern of arrest, by 1.

Figure 10: Dependency parsing result of sentence “Tom was arrested because Tom robbed the man”.

Data Collection Results

Table 8 shows the corpora we used for collecting the cause-effect pairs and the corresponding data collection results. We extract approximately 240,000 pairs from different corpora.

Corpus Result pairs
Gigaword [Graff et al.2003] 117,938
Novels [Zhu et al.2015] 129,824
CBTest [Hill et al.2015] 4,167
BNC [Burnard1995] 2,128
Table 8: Data collection results on different corpora.

Winograd Schema Challenge

Based on all the experiments described in the previous sections, we could conclude that the neural association model has the potential to be effective in commonsense reasoning. To further evaluate the effectiveness of the proposed neural association model, in this paper, we conduct experiments on solving the complex Winograd Schema challenge problems [Levesque, Davis, and Morgenstern2011, Morgenstern, Davis, and Ortiz Jr2016]. Winograd Schema is a commonsense reasoning task proposed in recent years, which has been treated as an alternative to the Turing Test [Turing1950]. This is a new AI task and it would be very interesting to see whether neural network methods are suitable for solving this problem. This section then describes the progress we have made in attempting to meet the Winograd Schema Challenge. For making clear what is the main task of the Winograd Schema, we will firstly introduce it at a high level. Afterwards, we will introduce the system framework as well as all the corresponding modules we proposed to automatically solve the Winograd Schema problems. Finally, experiments and discussions on a human annotated cause-effect dataset and discussion will be presented.

Winograd Schema

The Winograd Schema

(WS) evaluates a system’s commonsense reasoning ability based on a traditional, very difficult natural language processing task: coreference resolution

[Levesque, Davis, and Morgenstern2011, Saba2015]. The Winograd Schema problems are carefully designed to be a task that cannot be easily solved without commonsense knowledge. In fact, even the solution of traditional coreference resolution problems relies on semantics or world knowledge [Rahman and Ng2011, Strube2016]. For describing the WS in detail, here we just copy some words from [Levesque, Davis, and Morgenstern2011]. A WS is a small reading comprehension test involving a single binary question. Here are two examples:

  • The trophy would not fit in the brown suitcase because it was too big. What was too big?

    • Answer 0: the trophy

    • Answer 1: the suitcase

  • Joan made sure to thank Susan for all the help she had given. Who had given the help?

    • Answer 0: Joan

    • Answer 1: Susan

The correct answers here are obvious for human beings. In each of the questions, the corresponding WS has the following four features:

  1. Two parties are mentioned in a sentence by noun phrases. They can be two males, two females, two inanimate objects or two groups of people or objects.

  2. A pronoun or possessive adjective is used in the sentence in reference to one of the parties, but is also of the right sort for the second party. In the case of males, it is “he/him/his”; for females, it is “she/her/her” for inanimate object it is “it/it/its,” and for groups it is “they/them/their.”

  3. The question involves determining the referent of the pronoun or possessive adjective. Answer 0 is always the first party mentioned in the sentence (but repeated from the sentence for clarity), and Answer 1 is the second party.

  4. There is a word (called the special word) that appears in the sentence and possibly the question. When it is replaced by another word (called the alternate word), everything still makes perfect sense, but the answer changes.

Solving WS problems is not easy since the required commonsense knowledge is quite difficult to collect. In the following sections, we are going to describe our work on solving the Winograd Schema problems via neural network methods.

Schema texts Verb/Adjective 1 Verb/Adjective 2 Verb/Adjective 3
The man couldn’t lift his son because he was so weak weak not lift not be lifted
The man couldn’t lift his son because he was so heavy heavy not lift not be lifted
The fish ate the worm. it was tasty tasty eat be eaten
The fish ate the worm. it was hungry hungry eat be eaten
Mary tucked her daughter Anne into bed, so that she could sleep tuck into bed be tucked into bed sleep
Mary tucked her daughter Anne into bed, so that she could work tuck into bed be tucked into bed work
Tom threw his schoolbag down to ray after he reached the top of the stairs reach top throw down be thrown down
Tom threw his schoolbag down to ray after he reached the bottom of the stairs reach bottom throw down be thrown down
Jackson was greatly influenced by Arnold, though he lived two centuries earlier live earlier influence be influenced
Jackson was greatly influenced by Arnold, though he lived two centuries later live later influence be influenced
Table 9: Examples of the Cause-Effect dataset labelled from the Winograd Schema Challenge.

System Framework

In this paper, we propose that the commonsense knowledge required in many Winograd Schema problems could be formulized as some association relationships between discrete events. Using sentence “Joan made sure to thank Susan for all the help she had given” as an example, the commonsense knowledge is that the man who receives help should thank to the man who gives help to him. We believe that by modeling the association between event receive help and thank, give help and thank, we can make the decision by comparing the association probability and . If the models are well trained, we should get the inequality . Following this idea, we propose to utilize the data constructed from the previous section and extend the NAM models for solving WS problems. Here we design two frameworks for training NAM models.

  • TransMat

    -NAM: We design to apply four linear transformation matrices, i.e., matrices of (active, positive), (active, negative), (passive, positive) and (passive, negative), for transforming both the cause event and the effect event. After it, we then use NAM for model the cause-effect association relationship between any cause and effect events.

    Figure 11: The model framework for TransMat-NAM.
  • RelationVec-NAM: On the other hand, in this configuration, we treat all the typical 16 dimensions shown in Figure 8 as distinct relations. So there are 16 relation vectors in the corresponding NAM models. Currently we use the RMNN structure for NAM.

    Figure 12: The model framework for RelationVec-NAM.

Training the NAM models based on these two configurations is straightforward. All the network parameters, including the relation vectors and the linear transformation matrices, are learned by the standard stochastic gradient descend algorithm.


In this section, we will introduce our current experiments on solving the Winograd Schema problems. We will first select a cause-effect dataset constructed from the standard WS dataset. Subsequently, experimental setup will be described in detail. After presenting the experimental results, discussions would be made at the end of this section.

Cause-Effect Dataset Labelling

In this paper, based on the WS dataset available at, we labelled 78 cause-effect problems among all 278 available WS questions for our experiments. Table 9 shows some typical examples. For each WS problem, we label three verb (or adjective) phrases for the corresponding two parities and the pronoun. In the labelled phrases, we also record the corresponding patterns for each word respectively. Using word lift for example, we will generate lift for its active and positive pattern, not lift for its active and negative pattern, be lifted for its passive and positive pattern, and not be lifted for its passive and negative pattern. For example, in sentence “The man couldn’t lift his son because he was so weak”, we identify weak, not lift and not be lifted for he, the man and son resspectively. The commonsense is that somebody who is weak would more likely to has the effect not lift rather than not be lifted. The main work of NAM for solving this problem is to calculate the association probability between these phrases.

Experimental setup

The setup for NAM on this cause-effect task is similar to the settings on the previous tasks. For representing the phrases in neural association models, we use the bag-of-word (BOW) approach for composing phrases from pre-trained word vectors. Since the vocabulary we use in this experiment contains only 7500 common verbs and adjectives, there are some out-of-vocabulary (OOV) words in some phrases. Based on the BOW method, a phrase would be useless if all the words it contains are OOV. In this paper, we remove all the testing samples with useless phrases which results in 70 testing cause-effect samples. For network settings, we set the embedding size to 50 and the dimension of relation vectors to 50. We set 2 hidden layers for the NAM models and all the hidden layer sizes are set to 100. The learning rate is set to 0.01 for all the experiments. At the same time, to better control the model training, we set the learning rates for learning all the embedding matrices and the relation vectors to 0.025.

Negative sampling is very important for model training for this task. In the TransMat-NAM system, we generate negative samples by randomly selecting different patterns with respect to the pattern of the effect event in the positive samples. For example, if the positive training sample is “hungry (active, positive) causes eat (active, positive)”, we may generate negative samples like “hungry (active, positive) causes eat (passive, positive)”, or “hungry (active, positive) causes eat (active, negative)”. In the RelationVec-NAM system, the negative sampling method is much more straightforward, i.e., we will randomly select a different effect event from the whole vocabulary. In the example shown here, the possible negative sample would be “hungry (active, positive) causes happy (active, positive)”, or “hungry (active, positive) causes talk (active, positive)” and so on.


The experimental results are shown in Table 10. From the results, we find that the proposed NAM models achieve about 60% accuracy on the cause-effect dataset constructed from Winograd Schemas. More specifically, the RelationVec-NAM system performs slightly better than the TransMat-NAM system.

Model Accuracy (%)
TransMat-NAM 58.6 (41 / 70)
RelationVec-NAM 61.4 (43 / 70)
Table 10: Results of NAMs on the Winograd Schema Cause-Effect dataset.

In the testing results, we find the NAM performs well on some testing examples. For instance, in the call phone scenario, the proposed NAM generates the corresponding association probabilities as follows.

  • Paul tried to call George on the phone, but he wasn’t successful. Who was not successful?

    • Paul: = 0.7299

    • George: = 0.5430

    • Answer: Paul

  • Paul tried to call George on the phone, but he wasn’t available. Who was not available?

    • Paul: = 0.6859

    • George: = 0.8306

    • Answer: George

For these testing examples, we find our model can answer the questions by correctly calculating the association probabilities. The probability is larger than while the probability is smaller than . Those simple inequality relationships between the association probabilities are very reasonable in our commonsense. Here are some more examples:

  • Jim yelled at Kevin because he was so upset. Who was upset?

    • Jim: = 0.9296

    • Kevin: = 0.8785

    • Answer: Jim

  • Jim comforted Kevin because he was so upset. Who was upset?

    • Jim: = 0.0282

    • Kevin: = 0.5657

    • Answer: Kevin

This example also conveys some commonsense knowledge in our daily life. We all know that somebody who is upset would be more likely to yell at other people. Meanwhile, it is also more likely that they would be be comforted by other people.


In this paper, we have proposed neural association models (NAM) for probabilistic reasoning. We use neural networks to model association probabilities between any two events in a domain. In this work, we have investigated two model structures, namely DNN and RMNN, for NAMs. Experimental results on several reasoning tasks have shown that both DNNs and RMNNs can outperform the existing methods. This paper also reports some preliminary results to use NAMs for knowledge transfer learning. We have found that the proposed RMNN model can be quickly adapted to a new relation without sacrificing the performance in the original relations. After proving the effectiveness of the NAM models, we apply it to solve more complex commonsense reasoning problems, i.e., the Winograd Schemas [Levesque, Davis, and Morgenstern2011]. To support model training in this task, we propose a straightforward method to collect associative phrase pairs from text corpora. Experiments conducted on a set of Winograd Schema problems have indicated the neural association model does solve some problems successfully. However, it is still a long way to finally achieving automatic commonsense reasoning.


We want to thank Prof. Gary Marcus of New York University for his useful comments on commonsense reasoning. We also want to thank Prof. Ernest Davis, Dr. Leora Morgenstern and Dr. Charles Ortiz for their wonderful organizations for making the first Winograd Schema Challenge happen. This paper was supported in part by the Science and Technology Development of Anhui Province, China (Grants No. 2014z02006), the Fundamental Research Funds for the Central Universities (Grant No. WK2350000001) and the Strategic Priority Research Program of the Chinese Academy of Sciences (Grant No. XDB02070006).


  • [Bengio et al.2003] Bengio, Y.; Ducharme, R.; Vincent, P.; and Janvin, C. 2003. A neural probabilistic language model.

    The Journal of Machine Learning Research

  • [Bordes et al.2012] Bordes, A.; Glorot, X.; Weston, J.; and Bengio, Y. 2012. Joint learning of words and meaning representations for open-text semantic parsing. In Proceedings of AISTATS, 127–135.
  • [Bordes et al.2013] Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; and Yakhnenko, O. 2013. Translating embeddings for modeling multi-relational data. In Proceedings of NIPS, 2787–2795.
  • [Bowman et al.2015] Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326.
  • [Bowman2013] Bowman, S. R. 2013. Can recursive neural tensor networks learn logical reasoning? arXiv preprint arXiv:1312.6192.
  • [Burnard1995] Burnard, L. 1995. Users reference guide british national corpus version 1.0.
  • [Collobert et al.2011] Collobert, R.; Weston, J.; Bottou, L.; Karlen, M.; Kavukcuoglu, K.; and Kuksa, P. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research 12:2493–2537.
  • [Getoor2007] Getoor, L. 2007. Introduction to statistical relational learning. MIT press.
  • [Glorot and Bengio2010] Glorot, X., and Bengio, Y. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of AISTATS, 249–256.
  • [Graff et al.2003] Graff, D.; Kong, J.; Chen, K.; and Maeda, K. 2003. English gigaword. Linguistic Data Consortium, Philadelphia.
  • [Hill et al.2015] Hill, F.; Bordes, A.; Chopra, S.; and Weston, J. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301.
  • [Hinton et al.2012] Hinton, G. E.; Srivastava, N.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. R. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580.
  • [Hornik, Stinchcombe, and White1990] Hornik, K.; Stinchcombe, M.; and White, H. 1990. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural networks 3(5):551–560.
  • [Jensen1996] Jensen, F. V. 1996. An introduction to Bayesian networks, volume 210. UCL press London.
  • [Koller and Friedman2009] Koller, D., and Friedman, N. 2009. Probabilistic graphical models: principles and techniques. MIT press.
  • [LeCun, Bengio, and Hinton2015] LeCun, Y.; Bengio, Y.; and Hinton, G. 2015. Deep learning. Nature 521(7553):436–444.
  • [Levesque, Davis, and Morgenstern2011] Levesque, H. J.; Davis, E.; and Morgenstern, L. 2011. The winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning.
  • [Lin et al.2015] Lin, Y.; Liu, Z.; Sun, M.; Liu, Y.; and Zhu, X. 2015. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of AAAI.
  • [Liu and Singh2004] Liu, H., and Singh, P. 2004. Conceptnet: a practical commonsense reasoning toolkit. BT technology journal 22(4):211–226.
  • [McCarthy1986] McCarthy, J. 1986. Applications of circumscription to formalizing common-sense knowledge. Artificial Intelligence 28(1):89–116.
  • [Mikolov et al.2013] Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
  • [Miller1995] Miller, G. A. 1995. Wordnet: a lexical database for english. Communications of the ACM 38(11):39–41.
  • [Minsky1988] Minsky, M. 1988. Society of mind. Simon and Schuster.
  • [Morgenstern, Davis, and Ortiz Jr2016] Morgenstern, L.; Davis, E.; and Ortiz Jr, C. L. 2016. Planning, executing, and evaluating the winograd schema challenge. AI Magazine 37(1):50–54.
  • [Mueller2014] Mueller, E. T. 2014. Commonsense Reasoning: An Event Calculus Based Approach. Morgan Kaufmann.
  • [Nair and Hinton2010] Nair, V., and Hinton, G. E. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of ICML, 807–814.
  • [Neapolitan2012] Neapolitan, R. E. 2012. Probabilistic reasoning in expert systems: theory and algorithms. CreateSpace Independent Publishing Platform.
  • [Nickel et al.2015] Nickel, M.; Murphy, K.; Tresp, V.; and Gabrilovich, E. 2015. A review of relational machine learning for knowledge graphs. arXiv preprint arXiv:1503.00759.
  • [Nickel, Tresp, and Kriegel2012] Nickel, M.; Tresp, V.; and Kriegel, H.-P. 2012. Factorizing YAGO: scalable machine learning for linked data. In Proceedings of WWW, 271–280. ACM.
  • [Osgood1952] Osgood, C. E. 1952. The nature and measurement of meaning. Psychological bulletin 49(3):197.
  • [Pearl1988] Pearl, J. 1988. Probabilistic reasoning in intelligent systems: Networks of plausible reasoning.
  • [Peng, Khashabi, and Roth2015] Peng, H.; Khashabi, D.; and Roth, D. 2015. Solving hard coreference problems. Urbana 51:61801.
  • [Rahman and Ng2011] Rahman, A., and Ng, V. 2011. Coreference resolution with world knowledge. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, 814–824. Association for Computational Linguistics.
  • [Richardson and Domingos2006] Richardson, M., and Domingos, P. 2006. Markov logic networks. Machine learning 62(1-2):107–136.
  • [Saba2015] Saba, W. 2015. On the winograd schema challenge.
  • [Socher et al.2013] Socher, R.; Chen, D.; Manning, C. D.; and Ng, A. 2013. Reasoning with neural tensor networks for knowledge base completion. In Proceedings of NIPS, 926–934.
  • [Strube2016] Strube, M. 2016. The (non) utility of semantics for coreference resolution (corbon remix). In NAACL 2016 workshop on Coreference Resolution Beyond OntoNotes.
  • [Turing1950] Turing, A. M. 1950. Computing machinery and intelligence. Mind 59(236):433–460.
  • [Wang et al.2014] Wang, Z.; Zhang, J.; Feng, J.; and Chen, Z. 2014.

    Knowledge graph embedding by translating on hyperplanes.

    In Proceedings of AAAI, 1112–1119. Citeseer.
  • [Xue et al.2014] Xue, S.; Abdel-Hamid, O.; Jiang, H.; Dai, L.; and Liu, Q. 2014. Fast adaptation of deep neural network based on discriminant codes for speech recognition. Audio, Speech, and Language Processing, IEEE/ACM Trans. on 22(12):1713–1725.
  • [Zhu et al.2015] Zhu, Y.; Kiros, R.; Zemel, R.; Salakhutdinov, R.; Urtasun, R.; Torralba, A.; and Fidler, S. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In

    Proceedings of the IEEE International Conference on Computer Vision

    , 19–27.