1 Overview: S-Lang, S-Net and S-Rep
Embedding words and sentences in vector spaces has brought many symbolic tasks (especially in NLP) within the scope of deep neural network (DNN) models(Hinton, 1988; Palangi et al., 2016; Pollack, 1990; Socher et al., 2010; Weston et al., 2015)
. In general, DNNs may be expected to benefit if they can incorporate some of the power of symbolic computation without compromising the power of deep learning. The problem of embedding general symbol structures in vector spaces, and performing symbolic computation with these vectors, has been addressed theoretically, but these methods can require very large embedding spaces — e.g., Tensor Product Representation, TPR(Lee et al., 2016; Smolensky, 1990; Smolensky & Legendre, 2006) — or major error-correction/clean-up processes — e.g., Holographic Reduced Representation, HRR (Crawford et al., 2016; Plate, 1993, 2002, 2003) (Sec. 4; see also Kanerva (2009); Touretzky (1990)). We show here (Sec. 3) that deep learning can itself discover satisfactory methods of embedding general symbol structures, methods that operate in relatively small vector spaces without need for clean-up processing.
We define a general formal language scheme in which expressions denote symbol structures. Such a language will be called an S-Lang (Sec. 2). Information within these structures is accessed by evaluating query expressions within the language. The model that learns to encode structure-denoting expressions and to evaluate queries over these structures (Zaremba & Sutskever, 2014) is a simple bidirectional encoder-decoder model that operates on symbols in the formal language one at a time (Cho et al., 2014). We call such a model an S-Net (Sec. 3), and call the vector embedding of an S-Lang learned by an S-Net an S-Rep.
2 The task: Embedding general symbolic structures in vector spaces and accessing their contents
Symbol-structure-denoting expressions: Structural-role binding. In general, a symbol structure can be characterized as a set of symbols each bound to a role that it plays in the structure (Newell, 1980, 141). The method is applicable to any type of symbol structure, but we focus on binary trees here. The simple binary tree consists of the symbols respectively bound to the roles , where is the role of left-child (‘0’) of right-child (‘1’) of root, etc. Symbolic structural roles are typically recursive. The recursive character of binary tree roles can be seen by viewing as the symbol bound to the role . In the simple formal language we develop here, S-Lang, the tree will be denoted by the expression , where respectively abbreviate the roles . The grammar of S-Lang is shown in Fig. 1.
Query-denoting expressions: Structural-role unbinding. A minimal requirement for a vector embedding a symbol structure is that it be possible to extract, with vector computation, (the embedding of) the symbol that is bound to any specified role. In S-Lang (see Fig. 1), the query denoted asks for the structure (possibly a single symbol), bound to the role denoted ; this is role-unbinding. Thus the expression asks for the structure filling the role in , i.e., the structure forming the left child of the root, which in happens to be a single symbol, . Similarly, the expression asks for the left child of the right child of the root, and so has the value . An S-Lang query can return a structure rather than a single symbol. The expression asks for the right child of the root of , which is the structure , denoting ’s right sub-tree, .
Expressions combining querying and structure-building. The general expression in S-Lang allows structure that is returned by queries to be used to build new structures. Table 1 provides examples of expressions correctly evaluated by S-Net.
|unbind (not found)|
|3-bind, unbind, rebind|
3 The S-Net model and experimental results
S-Net is a standard bidirectional encoder-decoder network where the output of the bidirectional LSTM encoder is the S-Rep embedding of the input S-Lang expression. The S-Rep vector is then fed as input to an LSTM decoder. Some implementation details are given in Table 2, which also gives the results of training S-Net on randomly-selected input/output pairs.
Performance and hyperparameters of the trainedS-Net model
4 Analysis of S-Rep: The Superposition Principle
The Superposition Principle in theoretical structure-embedding schemes. Theoretical solutions to performing the task defined in Sec. 2 were proposed in the previous generation of neural network modeling. Two general solutions, TPR and HRR, were introduced in Sec. 1. The TPR embedding of a symbol structure with symbols respectively bound to roles is , where denotes the tensor (generalized outer) product and and are embeddings of the symbols and roles, with respective dimensions and ; the dimension of the TPR itself is then .
If the role-embedding vectors
are linearly independent, when collected together they form an invertible matrix; the rows of are the “unbinding” vectors : so these vectors can be used to unbind the roles in a TPR. The symbol that fills role in structure is exactly the symbol with embedding .
A crucial property of TPR is that the embedding of a structure is the sum over embeddings of its symbols. This is TPR’s Superposition Principle. This is what enables extraction of symbols from any binding: since .
HRRs are essentially contracted TPRs (Smolensky & Legendre, 2006, 260). The equation defining TPR() also defines HRR(), provided is reinterpreted to denote circular convolution: . Assuming the elements of the
are randomly (typically, normally) distributed, each role-embedding vectorcan be used as its own unbinding vector. However the HRR unbinding equation holds only approximately: . This noise must be eliminated by ‘clean-up’ processes. Note that, like TPR, HRR obeys the Superposition Principle.
Testing the Superposition Principle in the learned representation. As a test of whether the Superposition Principle holds of S-Rep, let denote the S-Rep vector embedding of S-Lang expression , and consider expressions containing two symbol/role bindings, such as . Then if the Superposition Principle holds, we have111The simpler equation does not hold in S-Rep; it appears that the manifolds of one- and two-binding embeddings are distinct. Eq. 1 is designed to stay within the latter. Eq. 1 is analogous to the famous equation of Mikolov et al. (2013). To make the analogy exact, let the roles ‘gender’, ‘status’ be denoted and let be denoted . Then , i.e., Eq. 1. :
A standard bidirectional encoder-decoder model can generate vector embeddings of expressions denoting complex symbol structures and can successfully query the content of such representations. Like theoretical techniques for accomplishing this, the learned representation obeys the Superposition Principle (approximately; at least within the manifold of embeddings of two-binding expressions).
- Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. Empirical Methods in Natural Language Processing-2014, abs/1406.1078, 2014.
- Crawford et al. (2016) Eric Crawford, Matthew Gingerich, and Chris Eliasmith. Biologically plausible, human-scale knowledge representation. Cognitive Science, 40(4):782–821, 2016.
- Hinton (1988) Geoffrey E. Hinton. Representing part-whole hierarchies in connectionist networks. In Proceedings of the Tenth Annual Conference of the Cognitive Science Society, pp. 48–54. 1988.
Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors.Cognitive Computing, 1:139–159, 2009.
- Lee et al. (2016) Moontae Lee, Xiaodong He, Wen-tau Yih, Jianfeng Gao, Li Deng, and Paul Smolensky. Reasoning in vector space: An exploratory study of question answering. In Proceedings of the International Conference on Learning Representations-2016, 2016.
- Mikolov et al. (2013) Tomas Mikolov, Scott Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT-2013). May 2013.
- Newell (1980) Allen Newell. Physical symbol systems. Cognitive Science, 4(1):135–183, 1980.
Palangi et al. (2016)
Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen,
Xinying Song, and Rabab Ward.
Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval.IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 24(4):694–707, 2016.
- Plate (1993) Tony Plate. Holographic Recurrent Networks. In Stephen José Hanson and C Lee Giles (eds.), Advances in Neural Information Processing Systems 5. Morgan Kaufmann, San Mateo, CA, 1993.
- Plate (2002) Tony Plate. Distributed representations. Encyclopedia of Cognitive Science, 2002.
- Plate (2003) Tony Plate. Holographic Reduced Representation: Distributed Representation for Cognitive Structures. CSLI Publications, Stanford, CA, 2003.
- Pollack (1990) Jordan B. Pollack. Recursive distributed representations. Artificial Intelligence, 46(1):77–105, 1990.
- Smolensky (1990) Paul Smolensky. Tensor product variable binding and the representation of symbolic structures in connectionist networks. Artificial Intelligence, 46:159–216, 1990.
- Smolensky & Legendre (2006) Paul Smolensky and Géraldine Legendre. The harmonic mind: From neural computation to Optimality-Theoretic grammar. 2 vols. MIT Press, Cambridge, MA, 2006.
- Socher et al. (2010) Richard Socher, Christopher D Manning, and Andrew Y Ng. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Proceedings of the NIPS-2010 Deep Learning and Unsupervised Feature Learning Workshop, pp. 1–9, 2010.
- Touretzky (1990) David S. Touretzky. Boltzcons: Dynamic symbol structures in a connectionist network. Artificial Intelligence, 46:5–46, 1990.
- Weston et al. (2015) Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merrienboer, Armand Joulin, and Tomas Mikolov. Towards AI-complete question answering: A set of prerequisite toy tasks. arXiv:1502.05698, 2015.
- Zaremba & Sutskever (2014) Wojciech Zaremba and Ilya Sutskever. Learning to execute. arXiv:1410.4615, 2014.