Grammar-Aware Question-Answering on Quantum Computers

12/07/2020 ∙ by Konstantinos Meichanetzidis, et al. ∙ 0

Natural language processing (NLP) is at the forefront of great advances in contemporary AI, and it is arguably one of the most challenging areas of the field. At the same time, with the steady growth of quantum hardware and notable improvements towards implementations of quantum algorithms, we are approaching an era when quantum computers perform tasks that cannot be done on classical computers with a reasonable amount of resources. This provides a new range of opportunities for AI, and for NLP specifically. Earlier work has already demonstrated a potential quantum advantage for NLP in a number of manners: (i) algorithmic speedups for search-related or classification tasks, which are the most dominant tasks within NLP, (ii) exponentially large quantum state spaces allow for accommodating complex linguistic structures, (iii) novel models of meaning employing density matrices naturally model linguistic phenomena such as hyponymy and linguistic ambiguity, among others. In this work, we perform the first implementation of an NLP task on noisy intermediate-scale quantum (NISQ) hardware. Sentences are instantiated as parameterised quantum circuits. We encode word-meanings in quantum states and we explicitly account for grammatical structure, which even in mainstream NLP is not commonplace, by faithfully hard-wiring it as entangling operations. This makes our approach to quantum natural language processing (QNLP) particularly NISQ-friendly. Our novel QNLP model shows concrete promise for scalability as the quality of the quantum hardware improves in the near future.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.


Appendix A Pregroup Grammar

Pregroup grammars where introduced by Lambek as an algebraic model for grammar Lambek .

A pregroup grammar is freely generated by the basic types in a finite set . Basic types are decorated by an integer , which signifies their adjoint order. Negative integers , with , are called left adjoints of order and positive integers are called right adjoints. We shall refer to a basic type to some adjoint order (include the zeroth order) simply as ‘type’. The zeroth order signifies no adjoint action on the basic type and so we often omit it in notation, .

The pregroup algebra is such that the two kinds of adjoint (left and right) act as left and right inverses under multiplication of basic types

where is the trivial or unit type. The left hand side of this reduction is called a contraction and the right hand side an expansion. Pregroup grammar also accommodates induced steps for . The symbol ‘’ is to be read as ‘type-reduction’ and the pregroup grammar sets the rules for which reductions are valid.

Now, to go from word to sentence, we consider a finite set of words called the vocabulary . We call the dictionary

(or lexicon) the finite set of entries

. The star symbol denotes the set of finite strings that can be generated by the elements of the set . Each dictionary entry assigns a product (or string) of types to a word , .

Finally, a pregroup grammar G generates a language as follows. A sentence is a sequence (or list) of words . The type of a sentence is the product of types of its words , where and . A sentence is grammatical, i.e. it belongs to the language generated by the grammar , if and only if there exists a sequence of reductions so that the type of the sentence reduces to the special sentence-type as . Note that it is in fact possible to type-reduce grammatical sentences only using contractions.

Appendix B String Diagrams

String diagrams describing process theories are generated by states, effects, and processes. In Fig.6 we comprehensively show these generators along with constraining equations on them. String diagrams for process theories formally describe process networks where only connectivity matters, i.e. which outputs are connected to which inputs. In other words, the length of the wires carries no meaning and the wires are freely deformable as long as the topology of the network is respected.

Figure 6: Diagrams are read from top to bottom. States have only outputs, effects have only inputs, processes (boxes) have both input and output wires. All wires carry types. Placing boxes side by side is allowed by the monoidal structure and signifies parallel processes. Sequential process composition is represented by composing outputs of a box with inputs of another box. A process transforms a state into a new state. There are special kinds of states called caps and effects called cups, which satisfy the snake equation which relates them to the identity wire (trivial process). Process networks freely generated by these generators need not be planar, and so there exists a special process that swaps wires and acts trivially on caps and cups.

It is beyond the purposes of this work to provide a comprehensive exposition on diagrammatic languages. We provide the necessary elements which are used for the implementation of our QNLP experiments.

Appendix C Random Sentence Generation with CFG

A context-free grammar generates a language from a set of production (or rewrite) rules applied on symbols. Symbols belong to a finite set and There is a special type called initial. Production rules belong to a finite set and are of the form , where . The application of a production rule results in substituting a symbol with a product (or string) of symbols. Randomly generating a sentence amounts to starting from and randomly applying production rules uniformly sampled from the set . The production ends when all types produced are terminal types, which are non other than words in the finite vocabulary .

From a process theory point of view, we represent symbols as types carried by wires. Production rules are represented as boxes with input and output wires labelled by the appropriate types. The process network (or string diagram) describing the production of a sentence ends with a production rule whose output is the -type. Then we randomly pick boxes and compose them backwards, always respecting type-matching when inputs of production rules are fed into outputs of other production rules. The generation terminates when production rules are applied which have no inputs (i.e. they are states), and they correspond to the words in the finite vocabulary.

In Fig.7 (on the left hand side of the arrows) we show the string-diagram generators we use to randomly produce sentences from a vocabulary of words composed of nouns, transitive verbs, intransitive verbs, and relative pronouns. The corresponding types of these parts of speech are . The vocabulary is the union of the words of each type, .

Having randomly generated a sentence from the CFG, its string diagram can be translated into a pregroup sentence diagram. To do so we use the translation rules shown in Fig.7. Note that a cup labeled by the basic type is used to represent a contraction . Pregroup grammars are weakly equivalent to context-free grammars, in the sense that they generate the same language Buszkowski and Moroz ; Pentus (1993).

Figure 7: CFG generation rules used to produce the corpora used in this work, represented as string-diagram generators, where , , , . They are mapped to pregroup reductions by mapping CFG symbols to pregroup types, and so CFG-states are mapped to DisCoCat word-states and production boxes are mapped to products of cups and identities. Note that the pregroup unit is the empty wire and so it is never drawn. Pregroup type contractions correspond to cups and expansions to caps. Since grammatical reduction are achievable only with contractions, only cups are required for the construction of sentence diagrams.

Appendix D Corpora

Here we present the sentences and their labels used in the experiments presented in the main text.

The types assigned to the words of this sentence are as follows. Nouns get typed as , transitive verbs are given type , intransitive verbs are typed , and the relative pronoun is typed .

Corpus of labeled sentences from the vocabulary , , , :
[(’Dude who loves Walter bowls’, 1),
(’Dude bowls’, 1),
(’Dude annoys Walter’, 0),
(’Walter who abides bowls’, 0),
(’Walter loves Walter’, 1),
(’Walter annoys Dude’, 1),
(’Walter bowls’, 1),
(’Walter abides’, 0),
(’Dude loves Walter’, 1),
(’Dude who bowls abides’, 1),
(’Walter who bowls annoys Dude’, 1),
(’Dude who bowls bowls’, 1),
(’Dude who abides abides’, 1),
(’Dude annoys Dude who bowls’, 0),
(’Walter annoys Walter’, 0),
(’Dude who abides bowls’, 1),
(’Walter who abides loves Walter’, 0),
(’Walter who bowls bowls’, 1),
(’Walter loves Walter who abides’, 0),
(’Walter annoys Walter who bowls’, 0),
(’Dude abides’, 1),
(’Dude loves Walter who bowls’, 1),
(’Walter who loves Dude bowls’, 1),
(’Dude loves Dude who abides’, 1),
(’Walter who abides loves Dude’, 0),
(’Dude annoys Dude’, 0),
(’Walter who annoys Dude bowls’, 1),
(’Walter who annoys Dude abides’, 0),
(’Walter loves Dude’, 1),
(’Dude who bowls loves Walter’, 1)]

Corpus of labeled sentences from the vocabulary , , , :
[(’Romeo dies’, 1.0),
(’Romeo loves Juliet’, 0.0),
(’Juliet who dies dies’, 1.0),
(’Romeo loves Romeo’, 0.0),
(’Juliet loves Romeo’, 0.0),
(’Juliet dies’, 1.0)]

Corpus of labeled sentences from the vocabulary , , , :
[(’Juliet kills Romeo who dies’, 0),
(’Juliet dies’, 1),
(’Romeo who loves Juliet dies’, 1),
(’Romeo dies’, 1),
(’Juliet who dies dies’, 1),
(’Romeo loves Juliet’, 1),
(’Juliet who dies loves Juliet’, 0),
(’Romeo kills Juliet who dies’, 0),
(’Romeo who kills Romeo dies’, 1),
(’Romeo who dies dies’, 1),
(’Romeo who loves Romeo dies’, 0),
(’Romeo kills Juliet’, 0),
(’Romeo who dies kills Romeo’, 1),
(’Juliet who dies kills Romeo’, 0),
(’Romeo loves Romeo’, 0),
(’Romeo who dies kills Juliet’, 0)]

Appendix E Sentence to Circuit mapping

Quantum theory has formally been shown to be a process theory. Therefore it enjoys a diagrammatic language in terms of string diagrams. Specifically, in the context of the quantum circuits we construct in our experiments, we use pure quantum theory. In the case of pure quantum theory, processes are unitary operations, or quantum gates in the context of circuits. The monoidal structure allowing for parallel processes is instantiated by the tensor product and sequential composition is instantiated by sequential composition of quantum gates.

In Fig.8 we show the generic construction of the mapping from sentence diagrams to parameterised quantum circuits for the hyperparameters and parameterised word-circuits we use in this work.

A wire carrying basic pregroup type is given qubits. A word-state with only one output wire becomes a one-qubit-state prepared from . For the preparation of such unary states we choose the sequence of gates defining an Euler decomposition of one-qubit unitaries . Word-states with more than one output wires become multiqubit states on qubits prepared by an IQP-style circuit from . Such a word-circuit is composed of -many layers. Each layer is composed of a layer of Hadamard gates followed by a layer in which every neighbouring pair of qubit wires is connected by a gate, . Since all gates commute with each other it is justified to consider this as a single layer, at least abstractly. The Kronecker tensor with -many output wires of type is mapped to a GHZ state on qubits. Specifically, GHZ is a circuit that prepares the state , where is the binary expression of an integer. The cup of pregroup type is mapped to -many nested Bell effects, each of which is implemented as a CNOT followed by a Hadamard gate on the control qubit and postselection on .

Figure 8: Mapping from sentence diagrams to parameterised quantum circuits. Here we show how the generators of sentence diagrams are mapped to generators of circuits, for the hyperparameters we consider in this work.

Appendix F Optimisation Method

The gradient-free otpimisation method we use, Simultaneous Perturbation Stochastic Approximation (SPSA), works as follows. Start from a random point in parameter space. At every iteration pick randomly a direction and estimate the derivative by finite difference with step-size depending on towards that direction. This requires two cost function evaluations. This provides a significant speed up the evaluation of . Then take a step of size depending on towards (opposite) that direction if the derivative has negative (positive) sign. In our experiments we use minimizeSPSA from the Python package noisyopt noi , and we set and , except for the experiment on ibmq for for which we set and .

Note that for classical simulations, we use just-in-time compilation of the cost function by invoking jit from jax Bradbury et al. (2018). In addition, the choice of the squares-of-differences cost we defined in Eq.2 is not unique. One can as well use the binary cross entropy

and the cost function can be minimised as well, as shown in Fig.9.

Figure 9: Minimisation of binary cross entropy cost function with SPSA for the question answering task for corpus .

In our classical simulation of the experiment we also used basinhopping Olson et al. (2012) in combination with Nelder-Mead Gao and Han (2010) from the Python package SciPy sci . Nelder-Mead is a gradient-free local optimisation method. basinhopping hops (or jumps) between basins (or local minima) and then returns the minimum over local minima of the cost function, where each minimum is found by Nelder-Mead. The hop direction is random. The hop is accepted according to a Metropolis criterion depending on the the cost function to be minimised and a temperature. We used the default temperature value () and the default number of basin hops ().

f.1 Error Decay

In Fig.10 we show the decay of mean training and test errors for the question answering task for corpus simulated classically, which is shown as inset in Fig.4. Plotting in log-log scale we reveal, at least initially, an algebraic decay of the errors with the depth of the word-circuits.

Figure 10: Algebraic decay of mean training and testing error for the data displayed in Fig.4 (bottom) obtained by basinhopping. Increasing the depth of the word-circuits results in algebraic decay of the mean training and testing errors. The slopes are and . We attribute the existence of the plateau for at large depths is due the small scale of our experiment and the small values for our hyperparameters determining the size of the quantum-enhanced feature space.

f.2 On the influence of noise to the cost function landscape

Regarding optimisation on a quantum computer, we comment on the effect of noise on the optimal parameters. Consider a successful optimisation of performed on a NISQ device, returning . However, if we instantiate the circuits and evaluate them on a classical computer to obtain the predicted labels , we observe that these can in general differ from the labels predicted by evaluating the circuits on the quantum computer. In the context of a fault-tolerant quantum computer, this should not be the case. However, since there is a non-trivial coherent-noise channel that our circuits undergo, it is expected that the optimiser’s result are affected in this way.

Appendix G Quantum Compilation

In order to perform quantum compilation we use Sivarajah et al. (2020). It is a Python module for interfacing with CQC’s , a toolset for quantum programming. From this toolbox, we need to make use of compilation passes.

At a high level, quantum compilation can be described as follows. Given a circuit and a device, quantum operations are decomposed in terms of the devices native gateset. Furthermore, the quantum circuit is reshaped in order to make it compatible with the device’s topology Cowtan et al. (2019). Specifically, the compilation pass that we use is default_compilation_pass(2). The integer option is set to for maximum optimisation under compilation pyt .

Circuits written in can be run on other devices by simply changing the backend being called, regardless whether the hardware might be fundamentally different in terms of what physical systems are used as qubits. This makes it platform agnostic. We stress that on IBMQ machines specifically, the native gates are arbitrary single-qubit unitaries (‘U3’ gate) and entangling controlled-not gates (‘CNOT’ or ‘CX’). Importantly, CNOT gates show error rates which are one or even two orders of magnitude larger than error rates of U3 gates. Therefore, we measure the depth of or circuits in terms of the CNOT-depth. Using pytket this can be obtained by invoking the command depth_by_type(OpType.CX).

For both backends used in this work, ibmq_montreal and ibmq_toronto, the reported quantum volume is 32 and the maximum allowed number of shots is .

Figure 11: Circuit for the Hadamard test. Measuring the control qubit in the computational basis allows one to estimate if , and if . The state can be a multiqubit state, and in this work we are interested in the case .
Figure 12: Use of Hadamard test to estimate the amplitude represented by the postselected circuit of Fig.3.

Appendix H Hadamard Test

In our binary classification NLP task, the predicted label is the norm squared of zero-to-zero transition amplitude , where the unitary includes the word-circuits and the circuits that implement the Bell effects as dictated by the grammatical structure. Estimating these amplitudes can be done by postselecting on . However, postselection costs exponential time in the number of postselected qubits; in our case needs to discard all bitstring sampled from the quantum computer that have Hamming weight other than zero. This is the procedure we follow in this proof of concept experiment, as we can afford doing so due to the small circuit sizes.

In such a setting, postselection can be avoided by using the Hadamard test Aharonov et al. (2008). See Fig.11 for the circuit allowing for the estimation of the real and imaginary part of an amplitude. In Fig.12 we show how the Hadamard test can be used to estimate the amplitude represented by the postselected quantum circuit of Fig.3.