A Neural-Symbolic Approach to Natural Language Tasks

10/29/2017 ∙ by Qiuyuan Huang, et al. ∙ University of Florida IEEE 0

Deep learning (DL) has in recent years been widely used in natural language processing (NLP) applications due to its superior performance. However, while natural languages are rich in grammatical structure, DL has not been able to explicitly represent and enforce such structures. This paper proposes a new architecture to bridge this gap by exploiting tensor product representations (TPR), a structured neural-symbolic framework developed in cognitive science over the past 20 years, with the aim of integrating DL with explicit language structures and rules. We call it the Tensor Product Generation Network (TPGN), and apply it to 1) image captioning, 2) classification of the part of speech of a word, and 3) identification of the phrase structure of a sentence. The key ideas of TPGN are: 1) unsupervised learning of role-unbinding vectors of words via a TPR-based deep neural network, and 2) integration of TPR with typical DL architectures including Long Short-Term Memory (LSTM) models. The novelty of our approach lies in its ability to generate a sentence and extract partial grammatical structure of the sentence by using role-unbinding vectors, which are obtained in an unsupervised manner. Experimental results demonstrate the effectiveness of the proposed approach.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

CAPTCHA, which stands for “Completely Automated Public Turing test to tell Computers and Humans Apart”, is a type of challenge-response test used in computers to determine whether or not the user is a human [1]. Most CAPTCHA systems are based on reading text contained in an image as shown in Fig. 1. However, such systems can be easily cracked by deep learning since deep learning can achieve very high accuracy in recognizing text. To address this, this paper proposes a new CAPTCHA, which distinguishes a human from a computer by testing the capability of image captioning (which generates a caption for a given image) or video storytelling (which generates a story consisting of multiple sentences for a give video sequence). It is known that the performance of computerized image captioning or video storytelling is far from human performance [2]. Hence, image/visual captioning based CAPTCHAs can more reliably distinguish computers from humans.

Figure 1: Reading-text-based CAPTCHA.

Deep learning is an important tool in many current natural language processing (NLP) applications. However, language rules or structures cannot be explicitly represented in deep learning architectures. The tensor product representation developed in [3, 4] has the potential of integrating deep learning with explicit rules (such as logical rules, grammar rules, or rules that summarize real-world knowledge). This paper develops a TPR approach for image/visual-captioning-based CAPTCHAs, introducing the Tensor Product Generation Network (TPGN) architecture.

A TPGN model generates natural language descriptions via learned representations. The representations learned in a crucial layer of the TPGN can be interpreted as encoding grammatical roles for the words being generated. This layer corresponds to the role-encoding component of a general, independently-developed architecture for neural computation of symbolic functions, including the generation of linguistic structures. The key to this architecture is the notion of Tensor Product Representation (TPR), in which vectors embedding symbols (e.g., lives, frodo) are bound to vectors embedding structural roles (e.g., verb, subject) and combined to generate vectors embedding symbol structures ([frodo lives]). TPRs provide the representational foundations for a general computational architecture called Gradient Symbolic Computation (GSC)

, and applying GSC to the task of natural language generation yields the specialized architecture defining the model presented here. The generality of GSC means that the results reported here have implications well beyond the particular task we address here.

In our proposed image/visual captioning based CAPTCHA, a TPGN takes an image as input and generates a caption. Then, an evaluator will evaluate the TPGN’s captioning performance by calculating the metric of SPICE [5] by comparing to human-generated gold-standard captions. If the SPICE value is less than a threshold, this input image can be used as a challenge in our CAPTCHA. In this way, we can obtain a set of images to be used as challenges in CAPTCHA.

In our CAPTCHA shown in Fig. 2, an image is randomly selected from the set and rendered as a challenge; a tester is asked to give a description of the image as an answer. An evaluator calculates a SPICE value for the answer. If the SPICE value is greater than a threshold, the answer is considered to be correct and the tester is deemed to be a human; otherwise, the tester is deemed to be a computer.

Figure 2: Image-captioning-based CAPTCHA.

The paper is organized as follows. Section 2 presents the rationale for our proposed architecture. In Section 3, we present our experimental results. Finally, Section 4 concludes the paper.

Figure 3: Architecture of TPGN, a TPR-capable generation network. “” denotes the matrix-vector product.

2 Design of image-captioning-based CAPTCHA

2.1 A TPR-capable generation architecture

In this work we propose an approach to network architecture design we call the TPR-capable method. The architecture we use (see Fig. 3) is designed so that TPRs could, in theory, be used within the architecture to perform the target task — here, generating a caption one word at a time. Unlike previous work where TPRs are hand-crafted, in our work, end-to-end deep learning will induce representations which the architecture can use to generate captions effectively.

As shown in Fig. 3, our proposed system is denoted by . The input of is an image feature vector and the output of is a caption. The image feature vector is extracted from a given image by a pre-trained CNN. The first part of our system is a sentence-encoding subnetwork which maps to a representation which will drive the entire caption-generation process; contains all the image-specific information for producing the caption. (We will call a caption a “sentence” even though it may in fact be just a noun phrase.)

If were a TPR of the caption itself, it would be a matrix (or 2-index tensor) which is a sum of matrices, each of which encodes the binding of one word to its role in the sentence constituting the caption. To serially read out the words encoded in , in iteration 1 we would unbind the first word from , then in iteration 2 the second, and so on. As each word is generated, could update itself, for example, by subtracting out the contribution made to it by the word just generated; denotes the value of when word is generated. At time step we would unbind the role occupied by word of the caption. So the second part of our system — the unbinding subnetwork — would generate, at iteration , the unbinding vector . Once produces the unbinding vector , this vector would then be applied to to extract the symbol that occupies word ’s role; the symbol represented by would then be decoded into word by the third part of , i.e., the lexical decoding subnetwork , which outputs , the 1-hot-vector encoding of .

Unbinding in TPR is achieved by the matrix-vector product (see Appendix A). So the key operation in generating is thus the unbinding of within , which amounts to simply:


This matrix-vector product is denoted “” in Fig. 3.

Thus the system of Fig. 3 is TPR-capable. This is what we propose as the Tensor-Product Generation Network (TPGN) architecture. The learned representation will not be proven to literally be a TPR, but by analyzing the unbinding vectors the network learns, we will gain insight into the process by which the learned matrix gives rise to the generated caption.

What type of roles might the unbinding vectors be unbinding? A TPR for a caption could in principle be built upon positional roles, syntactic/semantic roles, or some combination of the two. In the caption a man standing in a room with a suitcase, the initial a and man might respectively occupy the positional roles of and ; standing might occupy the syntactic role of verb; in the role of Spatial-P(reposition); while a room with a suitcase might fill a 5-role schema . In fact, there is evidence that our network learns just this kind of hybrid role decomposition.

What form of information does the sentence-encoding subnetwork need to encode in ? Continuing with the example of the previous paragraph, needs to be some approximation to the TPR summing several filler/role binding matrices. In one of these bindings, a filler vector — which the lexical subnetwork will map to the article a — is bound (via the outer product) to a role vector which is the dual of the first unbinding vector produced by the unbinding subnetwork : . In the first iteration of generation the model computes , which then maps to a. Analogously, another binding approximately contained in is . There are corresponding bindings for the remaining words of the caption; these employ syntactic/semantic roles. One example is . At iteration 3, decides the next word should be a verb, so it generates the unbinding vector which when multiplied by the current output of , the matrix , yields a filler vector which maps to the output standing. decided the caption should deploy standing as a verb and included in the binding . It similarly decided the caption should deploy in as a spatial preposition, including in the binding ; and so on for the other words in their respective roles in the caption.

2.2 CAPTCHA generation method

We first describe how to obtain a set of images as challenges in our CAPTCHA system. In our proposed image/visual captioning based CAPTCHA, a TPGN takes an image as input and generates a caption. Then, an evaluator will evaluate the TPGN’s captioning performance by calculating the metric of SPICE [5] by comparing to human-generated gold-standard captions. If the SPICE value is less than a threshold , this input image can be used as a challenge in our CAPTCHA; otherwise, this input image will not be selected as a challenge and a new image will be examined. In this way, we can obtain a set of images used as challenges in CAPTCHA. All images in are difficult to be captioned by a computerized image captioning system due to their low SPICE values.

Now, we describe our CAPTCHA generation method. In our CAPTCHA, an image is randomly selected from the set and rendered as a challenge; a tester is asked to give a description of the image as an answer. An evaluator calculates a SPICE value for the answer. If the SPICE value is greater than a threshold , the answer is considered to be correct and the tester is deemed to be a human; otherwise, the answer is considered to be wrong and the tester is deemed to be a computer.

Due to space limitations, the detailed design of our system is described in Appendix B; and TPR is reviewed in Appendix A.

3 Experimental results

To evaluate the performance of our proposed architecture, we use the COCO dataset [2]. The COCO dataset contains 123,287 images, each of which is annotated with at least 5 captions. We use the same pre-defined splits as [6, 7]: 113,287 images for training, 5,000 images for validation, and 5,000 images for testing. We use the same vocabulary as that employed in [7], which consists of 8,791 words.

For the CNN of Fig. 3, we used ResNet-152 [8]

, pretrained on the ImageNet dataset. The feature vector

has 2048 dimensions. Word embedding vectors in are downloaded from the web [9]

. The model is implemented in TensorFlow


with the default settings for random initialization and optimization by backpropagation.

In our experiments, we choose (where is the dimension of vector ). The dimension of is ; the vocabulary size ; the dimension of and is .

NIC [11] 0.666 0.451 0.304 0.203
CNN-LSTM 0.238 0.698 0.525 0.390 0.292 0.889
TPGN 0.243 0.709 0.539 0.406 0.305 0.909 0.18
Table 1: Performance of the proposed TPGN model on the COCO dataset.

The main evaluation results on the MS COCO dataset are reported in Table 1. The widely-used BLEU [12], METEOR [13], CIDEr [14], and SPICE [5] metrics are reported in our quantitative evaluation of the performance of the proposed schemes. In evaluation, our baseline is the widely used CNN-LSTM captioning method originally proposed in [11]. For comparison, we include results in that paper in the first line of Table 1. We also re-implemented the model using the latest ResNet feature and report the results in the second line of Table 1. Our re-implementation of the CNN-LSTM method matches the performance reported in [7], showing that the baseline is a state-of-the-art implementation. As shown in Table 1, compared to the CNN-LSTM baseline, the proposed TPGN significantly outperforms the benchmark schemes in all metrics across the board. The improvement in BLEU- is greater for greater ; TPGN particularly improves generation of longer subsequences. The results clearly attest to the effectiveness of the TPGN architecture.

Now, we address the issue of how to determine the two thresholds and in our CAPTCHA system. We set , which is about 80% less than the SPICE metric obtained by TPGN. In this way, the image set only contains images that are most difficult to be captioned by a computer.

We run a trained TPGN with input images from COCO test dataset, and select ten images from COCO test dataset, whose SPICE metrics are less than . Then we use Amazon Mechanical Turk system to generate captions for these ten images by humans. We observe that the SPICE metric obtained by a caption generated by a human is always larger than 0.3; hence, we set , which is much larger than the SPICE metric achievable by any existing computerized image captioning system [2].

4 Conclusion

In this paper, we proposed a new Tensor Product Generation Network (TPGN) for image/visual-captioning-based CAPTCHAs. The model has a novel architecture based on a rationale derived from the use of Tensor Product Representations for encoding and processing symbolic structure through neural network computation. In evaluation, we tested the proposed model on captioning with the MS COCO dataset, a large-scale image-captioning benchmark. Compared to widely adopted LSTM-based models, the proposed TPGN gives significant improvements on all major metrics including METEOR, BLEU, CIDEr, and SPICE. Our findings in this paper show great promise of TPRs in CAPTCHA. In the future, we will explore extending TPR to a variety of other NLP tasks and spam email detection.


Appendix A Review of tensor product representation

Tensor product representation (TPR) is a general framework for embedding a space of symbol structures into a vector space. This embedding enables neural network operations to perform symbolic computation, including computations that provide considerable power to symbolic NLP systems [4, 15]. Motivated by these successful examples, we are inspired to extend the TPR to the challenging task of learning image captioning. And as a by-product, the symbolic character of TPRs makes them amenable to conceptual interpretation in a way that standard learned neural network representations are not.

A particular TPR embedding is based in a filler/role decomposition of . A relevant example is when is the set of strings over an alphabet . One filler/role decomposition deploys the positional roles , where the filler/role binding assigns the ‘filler’ (symbol) to the position in the string. A string such as is uniquely determined by its filler/role bindings, which comprise the (unordered) set . Reifying the notion role in this way is key to TPR’s ability to encode complex symbol structures.

Given a selected filler/role decomposition of the symbol space, a particular TPR is determined by an embedding that assigns to each filler a vector in a vector space , and a second embedding that assigns to each role a vector in a space . The vector embedding a symbol is denoted by and is called a filler vector; the vector embedding a role is and called a role vector. The TPR for is then the following 2-index tensor in :



denotes the tensor product. The tensor product is a generalization of the vector outer product that is recursive; recursion is exploited in TPRs for, e.g., the distributed representation of trees, the neural encoding of formal grammars in connection weights, and the theory of neural computation of recursive symbolic functions. Here, however, it suffices to use the outer product; using matrix notation we can write (

2) as:


Generally, the embedding of any symbol structure is ; here: [3, 4].

A key operation on TPRs, central to the work presented here, is unbinding, which undoes binding. Given the TPR in (3), for example, we can unbind to get ; this is achieved simply by . Here is the unbinding vector dual to the binding vector . To make such exact unbinding possible, the role vectors should be chosen to be linearly independent. (In that case the unbinding vectors are the rows of the inverse of the matrix containing the binding vectors as columns, so that while for all other role vectors ; this entails that , the filler vector bound to . Replacing the matrix inverse with the pseudo-inverse allows approximate unbinding when the role vectors are not linearly independent).

Appendix B System Description

Figure 4: The sentence-encoding subnet and the unbinding subnet are inter-connected LSTMs; encodes the visual input while the encode the words of the output caption.

The unbinding subnetwork and the sentence-encoding network of Fig. 3 are each implemented as (1-layer, 1-directional) LSTMs (see Fig. 4); the lexical subnetwork

is implemented as a linear transformation followed by a softmax operation. In the equations below, the LSTM variables internal to the

subnet are indexed by 1 (e.g., the forget-, input-, and output-gates are respectively ) while those of the unbinding subnet are indexed by 2.

Thus the state updating equations for are, for = caption length:


where , , , , , , ,

is the (element-wise) logistic sigmoid function;

is the hyperbolic tangent function; the operator denotes the Hadamard (element-wise) product; , , , , , , , , . For clarity, biases — included throughout the model — are omitted from all equations in this paper. The initial state is initialized by:



is the vector of visual features extracted from the current image by ResNet

[7] and is the mean of all such vectors; . On the output side, is a 1-hot vector with dimension equal to the size of the caption vocabulary, , and is a word embedding matrix, the -th column of which is the embedding vector of the -th word in the vocabulary; it is obtained by the Stanford GLoVe algorithm with zero mean [9]. is initialized as the one-hot vector corresponding to a “start-of-sentence” symbol.

For in Fig. 3, the state updating equations are:


where , , and . The initial state is the zero vector.

The dimensionality of the crucial vectors shown in Fig. 3, and , is increased from to as follows. A block-diagonal matrix is created by placing copies of the matrix as blocks along the principal diagonal. This matrix is the output of the sentence-encoding subnetwork . Now, following Eq. (1), the ‘filler vector’ — ‘unbound’ from the sentence representation with the ‘unbinding vector’ — is obtained by Eq. (17).


Here , the output of the unbinding subnetwork , is computed as in Eq. (18), where is ’s output weight matrix.


Finally, the lexical subnetwork produces a decoded word by


where is the softmax function and is the overall output weight matrix. Since plays the role of a word de-embedding matrix, we can set


where is the word-embedding matrix. Since is pre-defined, we directly set by Eq. (20) without training through Eq. (19). Note that and are learned jointly through the end-to-end training.

Figure 5: Pre-training of TPGN.

Fig. 5 shows a pre-training method for initializing TPGN. During the pre-training phase, there is no image input, i.e., image feature vector . In Fig. 5, at time , the LSTM module takes a sentence of length as input and outputs a vector () at time . That is, the LSTM converts a sentence into , which is the input of TPGN. We use end-to-end training to train the whole system shown in Fig. 5. After finishing pre-training, we let and use images as input to train the TPGN in Fig. 3, initialized by the pretrained parameter values.

Appendix C Related work

Most existing DL-based image captioning methods [16, 11, 17, 18, 19, 6, 20, 21]

involve two phases/modules: 1) image analysis, typically by a Convolutional Neural Network (CNN), and 2) a language model for caption generation (


). The CNN module takes an image as input and outputs an image feature vector or a list of detected words with their probabilities. The language model is used to create a sentence (caption) out of the detected words or the image feature vector produced by the CNN.

There are mainly two approaches to natural language generation in image captioning. The first approach takes the words detected by a CNN as input, and uses a probabilistic model, such as a maximum entropy (ME) language model, to arrange the detected words into a sentence. The second approach takes the penultimate activation layer of the CNN as input to a Recurrent Neural Network (RNN), which generates a sequence of words (the caption)


The work reported here follows the latter approach, adopting a CNN + RNN-generator architecture. Specifically, instead of using a conventional RNN, we propose a recurrent network that has substructure derived from the general GSC architecture: one recurrent subnetwork holds an encoding — which is treated as an approximation of a TPR — of the words yet to be produced, while another recurrent subnetwork generates a sequence of vectors that is treated as a sequence of roles to be unbound from , in effect, reading out a word at a time from . Examining how the model deploys these roles allows us to interpret them in terms of grammatical categories; roughly speaking, a sequence of categories is generated and the words stored in are retrieved and spelled out via their categories.


  • [1] “Wikipedia link for captcha,” https://en.wikipedia.org/wiki/CAPTCHA, 2017.
  • [2] “Coco dataset for image captioning,” http://mscoco.org/dataset/#download, 2017.
  • [3] P. Smolensky, “Tensor product variable binding and the representation of symbolic structures in connectionist systems,” Artificial intelligence, vol. 46, no. 1-2, pp. 159–216, 1990.
  • [4] P. Smolensky and G. Legendre, The harmonic mind: From neural computation to optimality-theoretic grammar. Volume 1: Cognitive architecture.   MIT Press, 2006.
  • [5] P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in

    European Conference on Computer Vision

    .   Springer, 2016, pp. 382–398.
  • [6] A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” in

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    , 2015, pp. 3128–3137.
  • [7] Z. Gan, C. Gan, X. He, Y. Pu, K. Tran, J. Gao, L. Carin, and L. Deng, “Semantic compositional networks for visual captioning,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
  • [8] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
  • [9] J. Pennington, R. Socher, and C. Manning, “Stanford glove: Global vectors for word representation,” https://nlp.stanford.edu/projects/glove/, 2017.
  • [10] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015, software available from tensorflow.org. [Online]. Available: https://www.tensorflow.org/
  • [11] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3156–3164.
  • [12] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting on association for computational linguistics.   Association for Computational Linguistics, 2002, pp. 311–318.
  • [13] S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization.   Association for Computational Linguistics, 2005, pp. 65–72.
  • [14] R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 4566–4575.
  • [15] P. Smolensky, “Symbolic functions from neural computation,” Philosophical Transactions of the Royal Society — A: Mathematical, Physical and Engineering Sciences, vol. 370, pp. 3543 – 3569, 2012.
  • [16] J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, and A. Yuille, “Deep captioning with multimodal recurrent neural networks (m-rnn),” in Proceedings of International Conference on Learning Representations, 2015.
  • [17] J. Devlin, H. Cheng, H. Fang, S. Gupta, L. Deng, X. He, G. Zweig, and M. Mitchell, “Language models for image captioning: The quirks and what works,” arXiv preprint arXiv:1505.01809, 2015.
  • [18] X. Chen and C. Lawrence Zitnick, “Mind’s eye: A recurrent visual representation for image caption generation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 2422–2431.
  • [19] J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, “Long-term recurrent convolutional networks for visual recognition and description,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 2625–2634.
  • [20] R. Kiros, R. Salakhutdinov, and R. Zemel, “Multimodal neural language models,” in Proceedings of the 31st International Conference on Machine Learning (ICML-14), 2014, pp. 595–603.
  • [21] R. Kiros, R. Salakhutdinov, and R. S. Zemel, “Unifying visual-semantic embeddings with multimodal neural language models,” arXiv preprint arXiv:1411.2539, 2014.
  • [22] H. Fang, S. Gupta, F. Iandola, R. K. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. C. Platt et al., “From captions to visual concepts and back,” in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2015, pp. 1473–1482.