Speeding-up the decision making of a learning agent using an ion trap quantum processor

Introduction

The past decade has seen the parallel advance of two research areas — quantum computationNielsen and Chuang (2000)

Russell and Norvig (2003) — from abstract theory to practical applications and commercial use. Quantum computers, operating on the basis of information coherently encoded in superpositions of states that could be considered classical bit values, hold the promise of exploiting quantum advantages to outperform classical algorithms, e.g., for searching databasesGrover (1996), factoring numbersShor (1994)

, or even for precise parameter estimation

Giovannetti et al. (2004); Friis et al. (2017). At the same time, artificial intelligence and machine learning have become integral parts of modern automated devices using classical processorsLim et al. (2017); Silver et al. (2016); Mnih et al. (2015); Schaeffer et al. (2007). Despite this seemingly simultaneous emergence and promise to shape future technological developments, the overlap between these areas still offers a number of unexplored problemsBiamonte et al. (2016). It is hence of fundamental and practical interest to determine how quantum information processing and autonomously learning machines can mutually benefit from each other.

Within the area of artificial intelligence, a central component of modern applications is the learning paradigm of an agent interacting with an environmentSutton and Barto (1998); Russell and Norvig (2003); Briegel and De las Cuevas (2012) illustrated in Fig. 1

(a), which is usually formalized as so-called reinforcement learning. This entails receiving perceptual input and being able to react to it in different ways. The learning aspect is manifest in the reinforcement of the connections between the inputs and actions, where the correct association is (often implicitly) specified by a reward mechanism, which may be external to the agent. In this very general context, an approach to explore the intersection of quantum computing and artificial intelligence is to equip autonomous learning agents with quantum processors for their deliberation procedure

¹¹1Other approaches that we will not further discuss here concern models where the environment, and the agent’s interaction with it may be of quantum mechanical nature as wellDunjko et al. (2016).. That is, an agent chooses its reactions to perceptual input by way of quantum algorithms or quantum random walks. The agent’s learning speed can then be quantified in terms of the average number of interactions with the environment until targeted behavior (reactions triggering a reward) is reproduced by the agent with a desired efficiency. This learning speed cannot generically be improved by incorporating quantum technologies into the agent’s designDunjko et al. (2016).

However, a recent modelPaparo et al. (2014) for learning agents based on projective simulation (PS)Briegel and De las Cuevas (2012) allows for a generic speed-up in the agent’s deliberation time during each individual interaction. This quantum improvement in the reaction speed has been established within the reflecting projective simulation (RPS) variant of PSPaparo et al. (2014)

. There, the desired actions of the agent are chosen according to a specific probability distribution that can be modified during the learning process. This is of particular relevance to adapt to rapidly changing environments

Paparo et al. (2014), as we shall elaborate on in the next section. For this task, the deliberation time of classical RPS agents is proportional to the quantities

1 / δ

and

1 / ϵ

. These characterize the time needed to generate the specified distribution in the agent’s internal memory and the time to sample a suitable (e.g., rewarded rather than an unrewarded) action from it, respectively. A quantum RPS (Q-RPS) agent, in contrast, is able to obtain such an action quadratically faster, i.e., within a time of the order

1 / \sqrt{δ ϵ}

(see Methods).

Figure 1: Learning agent & quantum reflecting projective simulation (Q-RPS)
. (a) Learning agents receive perceptual input (“percepts”) from and act on the environment. The projective simulation (PS) decision-making process draws from the agent’s memory and can be modeled as a random walk in a clip network, which, in turn, is represented by a stochastic matrix
$P$ . (b) Q-RPS agents enhance the relative probability of (desired) actions (green columns) compared to other clips (grey) that may include undesired actions or percepts (blue) within the stationary distribution of $P$ before sampling, achieving a quadratic speed-up w.r.t. to classical RPS agents.

Here, we report on the first proof-of-principle experimental demonstration of a quantum-enhanced reinforcement learning system, complementing recent experimental work in the context of (un)supervised learning

Ristè et al. (2017); Li et al. (2015); Cai et al. (2015). We implement the deliberation process of an RPS learning agent in a system of two qubits that are encoded in the energy levels of one trapped atomic ion each. Within experimental uncertainties, our results confirm the agent’s action output according to the desired distributions and within deliberation times that are quadratically improved with respect to comparable classical agents. This laboratory demonstration of speeding up a learning agent’s deliberation process can be seen as the first experiment combining novel concepts from machine learning with the potential of ion trap quantum computers where complete quantum algorithms have been demonstrated Hanneke et al. (2010); Monz et al. (2016); Piltz et al. (2016); Debnath et al. (2016) and feasible concepts for scaling up Kielpinski et al. (2002); Monroe et al. (2014); Lekitsch et al. (2017) are vigorously pursued.

Experimental Implementation of Rank-One RPS

The proof-of-principle experiment that we report in this paper experimentally demonstrates the quantum speed-up of quantum-enhanced learning agents. That is, we are able to empirically confirm both the quadratically improved scaling of $O (1 / \sqrt{ϵ})$ , and the correct output according to the tail of the stationary distribution. Here, $ϵ$ denotes the initial probability of finding a flagged action within the stationary distribution $α = {a_{i}}$ ) for the average number of calls of the diffusion operator before sampling one of the desired actions (see Methods). The tail is defined as the first $n$ components of $α$ . The latter means that $a_{j} / a_{k} = b_{j} / b_{k} \forall j, k \in {1, \dots, n}$ , where $b_{j}$ denotes the final probability that the agent obtains the flagged action labeled $j$ . Note that the Q-RPS algorithm enhances the overall probability of obtaining a flagged action such that

~ ϵ \equiv n \sum i = 1 b_{i}

> n \sum i = 1 a_{j} = ϵ,

(1)

whilst maintaining the relative probabilities of the flagged actions according to the tail of $α$ , as illustrated in Fig. 1 (b).

For the implementation we hence need at least a three-dimensional Hilbert space that we realize in our experiment using two qubits encoded in the energy levels of two trapped ions (see the experimental setup section): Two states to represent two different flagged actions (represented in our experiment by $| 00 ⟩$ and $| 01 ⟩$ ), and at least one additional state for all non-flagged actions ( $| 10 ⟩$ and $| 11 ⟩$ in our experiment). The preparation of the stationary state is implemented by

| α ⟩

= U_{P} (θ_{1}, θ_{2}) | 0 ⟩ = R_{1} (θ_{1}, \frac{π}{2}) R_{2} (θ_{2}, \frac{π}{2}) | 0 ⟩,

(2)

where $R_{j} (θ, ϕ)$ is a single-qubit rotation on qubit $j$ , i.e.,

R_{j} (θ, ϕ)

= exp [i \frac{θ}{2} (X_{j} cos ϕ - Y_{j} sin ϕ)] .

(3)

Here, $X_{j}$ , $Y_{j}$ , and $Z_{j}$ denote the usual Pauli operators of qubit $j$ . The total probability $ϵ = a_{00} + a_{01}$ for a flagged action within the stationary distribution is then determined by $θ_{1}$ via

ϵ = {cos}^{2} (θ_{1} / 2),

(4)

whereas $θ_{2}$ determines the relative probabilities of obtaining one of the flagged actions via

a_{00} / ϵ = {cos}^{2} (θ_{2} / 2) .

(5)

Figure 2: Quantum circuit for Q-RPS. A rank-one Q-RPS is implemented using two qubits. The diffusion step consisting of reflections over the flagged actions and the stationary distribution (shown once each) is repeated $k$ times, where $k$ is given by Eq. (11). The specific pulse sequence implementing this circuit is explained in the Supplementary Material.

The reflection over the flagged actions ${ref}_{A}$ is here given by a $Z$ rotation, defined by $R_{j, z} (θ) = exp [- i \frac{θ}{2} Z_{j}]$ , with rotation angle $- π$ for the first qubit,

{ref}_{A} = R_{1, z} (- π) = exp [i \frac{π}{2} Z_{1}] .

(6)

The reflection over the stationary distribution can be performed by a combination of single-qubit rotations determined by $θ_{1}$ and $θ_{2}$ and a CNOT gate given by

	${ref}_{α}$	$=R1(θ1−π,π2)R2(θ2+π2,π2)U\raisebox0.0pt$CNOT$$
		$\times R_{1} (- θ_{1} - π, \frac{π}{2}) R_{2} (- θ_{2} - \frac{π}{2}, \frac{π}{2}),$		(7)

which can be understood as two calls to $U_{P}$ (once in terms of $U_{P}^{†}$ ) supplemented by fixed single-qubit operationsDunjko et al. (2015). The total gate sequence for a single diffusion step (consisting of a reflection over the flagged actions followed by a reflection over the stationary distribution) can hence be decomposed into single-qubit rotations and CNOT gates and is shown in Fig. 2. The speed-up of the rank-one Q-RPS algorithm w.r.t. a classical RPS agent manifests in terms of a quadratically smaller average number of calls to $U_{P}$ (or, equivalently, to the diffusion operator $D = {ref}_{α} {ref}_{A}$ ) until a flagged action is sampled. Since the final probability of obtaining a desired action is $~ ϵ \equiv \sum_{i = 1, \dots, n} b_{i}$ , we require $1 / ~ ϵ$ samples on average, each of which is preceded by the initial preparation of $| α ⟩$ and $k$ diffusion steps. The average number of uses of $U_{P}$ to sample correctly is hence $C = (2 k (ϵ) + 1) / ~ ϵ$ , which we refer to as ’cost’ in this paper. In the following, it is this functional relationship between $C$ and $ϵ$ that we put to the test, along with the predicted ratio $a_{00} / a_{01}$ of occurrence of the two flagged actions.

The experimental setup

Two $^{171}$ Yb $^{+}$ ions are confined in a linear Paul trap with axial and radial trap frequencies of 2 $π \times 117$ kHz and $2 π \times 590$ kHz, respectively. After Doppler cooling, the two ions form a linear Coulomb crystal, which is exposed to a static magnetic field gradient of $19$ T/m, generated by a pair of permanent magnets. The ion-ion spacing in this configuration is approximately $10 μ$ m. Magnetic gradient induced coupling (MAGIC) between ions results in an adjustable qubit interaction mediated by the common vibrational modes of the Coulomb crystalKhromova et al. (2012). In addition, qubit resonances are individually shifted as a result of this gradient and become position dependent. This makes the qubits distinguishable and addressable by their frequency of resonant excitation. The addressing frequency separation for this two-ion system is about $3.7$ MHz. All coherent operations are performed using radio frequency (RF) radiation near $12.6$ GHz, matching the respective qubit resonancesPiltz et al. (2014). A more detailed description of the experimental setup can be found in Refs.Khromova et al. (2012); Wölk et al. (2015); Piltz et al. (2016).

The qubits are encoded in the hyperfine manifold of each ion’s ground state, representing an effective spin $1 / 2$ system. The qubit states $| 0 ⟩$ and $| 1 ⟩$ are represented by the energy levels $|^{2} S_{1 / 2}, F = 0 ⟩$ and $|^{2} S_{1 / 2}, F = 1, m_{F} = + 1 ⟩$ , respectively. The ions are Doppler cooled on the resonance $|^{2} S_{1 / 2}, F = 1 ⟩$ $\leftrightarrow$ $|^{2} P_{1 / 2}, F = 0 ⟩$ with laser light near $369$ nm. Optical pumping into long-lived meta-stable states is prevented using laser light near $935$ nm and $638$ nm. The vibrational excitation of the Doppler cooled ions is further reduced by employing RF sideband cooling for both the center of mass mode and the stretch mode. This leads to a mean vibrational quantum number of $⟨ n ⟩ \leq 5$ for both modes. The ions are then initialized in the qubit state $| 0 ⟩$ by state selective optical pumping with a $2.1$ GHz red-shifted Doppler-cooling laser on the $|^{2} S_{1 / 2}, F = 1 ⟩$ $\leftrightarrow$ $|^{2} P_{1 / 2}, F = 1 ⟩$ resonance.

The desired qubit states are prepared by applying an RF pulse resulting in a coherent qubit rotation with precisely defined rotation angle and phase (Eq. (3)). The required number of diffusion steps is then applied to both qubits, using appropriate single-qubit rotations and a two-qubit ZZ-interaction given by

U_{Z Z} (θ) = exp [i \frac{θ}{2} Z_{1} Z_{2}],

(8)

which is directly realizable with MAGICKhromova et al. (2012). A CNOT gate ( $U_{C N O T}$ ) can then be performed via

U\raisebox0.0pt$CNOT$

= e^{- i \frac{π}{4}} R_{2} (\frac{π}{2}, \frac{3 π}{2}) U_{Z Z} (\frac{π}{2}) R_{2} (\frac{π}{2}, 0) R_{2, z} (\frac{π}{2}) R_{1, z} (- \frac{π}{2}) .

The required number of single qubit gates is optimized by combining appropriate single qubit rotations together from ${ref}_{A}$ and ${ref}_{α}$ (see Fig. 2). Thus, we can simplify the algorithm to

	$D =$	$R_{2} (θ_{2}, \frac{π}{2}) R_{1} (θ_{1}, \frac{π}{2}) R_{2, z} (- \frac{π}{2}) R_{1, z} (\frac{π}{2})$
		$U_{Z Z} (\frac{π}{2}) R_{2} (- θ_{2}, \frac{π}{2}) R_{1} (θ_{1}, \frac{π}{2}),$		(9)

as shown in Fig. 5 of Methods.

During the evolution time of $4.24$ ms for each diffusion step both qubits are protected from decoherence by applying universally robust (UR) dynamical decoupling (DD) pulsesGenov et al. (2017). The complete pulse sequence for the experiment reported here can be found in Fig. 5 of Methods.

Finally, projective measurements on both qubits are performed in the computational basis ${| 0 ⟩, | 1 ⟩}$ by scattering laser light near $369$ nm on the $|^{2} S_{1 / 2}, F = 1 ⟩$ $\leftrightarrow$ $|^{2} P_{1 / 2}, F = 0 ⟩$ transition, and detecting spatially resolved resonance fluorescence using an electron multiplying charge coupled device (EMCCD) to determine the relative frequencies $b_{00}, b_{01}, b_{10}, b_{11}$ for obtaining the states $| 00 ⟩$ , $| 01 ⟩$ , $| 10 ⟩$ , and $| 11 ⟩$ , respectively.

Results

Figure 3: Scaling behavior of the learning agent’s cost employing Q-RPS and RPS. After the preparation of $| α ⟩$ , $k$ diffusion steps are applied before an action is sampled. This procedure is repeated until a flagged action is obtained, accumulating a certain cost $C$ , whose average is shown on the vertical axis. Measurements are performed for different values of $ϵ$ corresponding to $k = 1$ to $k = 7$ diffusion steps. The dashed black line and the solid blue line represent the behavior expected for ideal Q-RPS (1/ $\sqrt{ϵ}$ ) and ideal classical RPS (1/ $ϵ$ ), respectively. The fit to the experimental data confirms that the scaling behavior follows a $1 / \sqrt{ϵ}$ behavior, and thus is consistent with Q-RPS. The data show that the experimental Q-RPS outperforms the classical RPS within the range of $ϵ$ chosen in the experiment. The error bars on the x-axis represent the uncertainties in preparing the quantum states. The error bars on the y-axis represent the statistical errors.

As discussed above, our goal is to test the two characteristic features of rank-one Q-RPS: (i) the scaling of the average cost $C$ with $ϵ$ , and (ii) the sampling ratio for the different flagged actions. For the former, we expect a scaling of $1 / \sqrt{ϵ}$ , while we expect the ratio of the number of occurrences of the two actions to be maintained with respect to the relative probabilities given by the stationary distribution. Therefore, our first set of measurements studies the behavior of the cost $C$ as a function of the total initial probability $ϵ$ . The second set of measurements studies the behavior of the output probability ratio $r_{f} = b_{00} / b_{01}$ as a function of input probability ratio $r_{i} = a_{00} / a_{01}$ .

For the former, a series of measurements is performed for different values of $ϵ$ corresponding to $k = 1$ to $k = 7$ diffusion steps after the initial state preparation. To obtain the cost $C = (2 k (ϵ) + 1) / ~ ϵ$ , where $~ ϵ = b_{00} + b_{01}$ , we measure the probabilities $b_{00}$ and $b_{01}$ after $k$ diffusion steps and repeat the experiment 1600 times for fixed $ϵ$ . The average cost is then plotted against $ϵ$ as shown in Fig. 3. The experimental data shows that the cost decreases with $ϵ$ as $1 / \sqrt{ϵ}$ , as desired. This is in good agreement with the behavior expected for the ideal Q-RPS algorithm. In the range of chosen probabilities $ϵ$ , the experimental result of Q-RPS outperforms the classical RPS, as shown in Fig. 3. Therefore, we demonstrate that the experimental efficiency is already good enough not only to obtain improved scaling, but also to outperform the classical algorithm, despite the offset in the cost function and the finite precision of the quantum algorithm. The deviation from the ideal behavior is attributed to a small detuning of the RF pulses implementing coherent operations, as we discuss in the Supplementary Materials.

Figure 4: Output distribution. A comparison of the output ratio of two flagged actions at the end of the algorithm with the corresponding input ratio is shown. Measurements are performed with $k = 1$ (red square) and $k = 3$ (blue circle) diffusion steps. The black dashed line shows the behavior of the ideal Q-RPS. The red and blue dashed lines, each representing a linear fit to the corresponding set of data, confirm that the initial probability ratio is maintained. Error bars represent statistical errors.

For the second set of measurements, we select a few calculated probabilities $a_{00}$ and $a_{01}$ in order to obtain different values of the input ratio $r_{i} = a_{00} / a_{01}$ between 0 and 2, whilst keeping $k (ϵ)$ in a range between $k = 1$ and $k = 3$ . For these probabilities $a_{00}$ and $a_{01}$ , the corresponding rotation angles $θ_{1}$ and $θ_{2}$ of RF pulses intended for preparation are extracted using Eq. (4) and Eq. (5). We then perform the Q-RPS algorithm for the specific choices of $k$ and repeat it 1600 times to estimate the probabilities $b_{00}$ and $b_{01}$ . We finally obtain the output ratio $r_{f} = b_{00} / b_{01}$ , which is plotted against the input ratio in Fig. 4. The experimental data follows a straight line with an offset from the behavior $r_{f} / r_{i} = 1$ expected for an ideal Q-RPS agent. The slopes of the two fitted linear functions agree within their respective error showing that the deviation of the output ratio from the ideal result is independent of the number of diffusion steps. In addition, this indicates that this deviation is not caused by the quantum algorithm itself, but by the initial state preparation and/or by the final measurement process where such a deviation can be caused by an asymmetry in the detection fidelity. Indeed, the observed deviation is well explained by a typical asymmetry in the detection fidelity of 3% as encountered in the measurements presented here. This implies reliability of the quantum algorithm also for a larger number of diffusion steps. A detailed discussion of experimental sources of error is given in the Supplementary Materials.

Conclusion

We have investigated a quantum-enhanced deliberation process of a learning agent implemented in an ion trap quantum processor. Our approach is centered on the projective simulationBriegel and De las Cuevas (2012) model for reinforcement learning. Within this paradigm, the decision-making procedure is cast as a stochastic diffusion process, that is, a (classical or quantum) random walk in a representation of the agent’s memory.

The classical PS framework can be used to solve standard textbook problems in reinforcement learningMautner et al. (2015); Melnikov et al. (2014); Makmal et al. (2016), and has recently been applied in advanced roboticsHangl et al. (2016), adaptive quantum computationTiersch et al. (2015), as well as in the machine-generated design of quantum experimentsMelnikov et al. (2017). We have focused on reflecting projective simulationPaparo et al. (2014), an advanced variant of the PS model based on ”mixing” (see Methods), where the deliberation process allows for a quantum speed-up of Q-RPS agents w.r.t. to their classical counterparts. In particular, we have considered the interesting special case of rank-one Q-RPS. This provides the advantage of the speed-up offered by the mixing-based approach, but is also in one-to-one correspondence with the hitting-based basic PS using two-layered networks, which has been applied in classical task environmentsMautner et al. (2015); Melnikov et al. (2014); Makmal et al. (2016); Hangl et al. (2016); Tiersch et al. (2015); Melnikov et al. (2017).

In a proof-of-principle experimental demonstration, we verify that the deliberation process of the quantum learning agent is quadratically faster compared to that of a classical learning agent. The experimental uncertainties in the reported results, which are in excellent agreement with a detailed model, do not interfere with this genuine quantum advantage in the agent’s deliberation time. We achieve results for the cost $C$ for up to 7 diffusion steps corresponding to an initial probability $ϵ$ = 0.01 to choose a flagged action. The systematic variation of the the ratio $r_{i}$ between the input probabilities, $a_{00}$ and $a_{01}$ for flagged actions and the measurement of the ratio $r_{f}$ between the learning agent’s output probabilities, $b_{00}$ and $b_{01}$ as a function of $r_{i}$ shows that the quantum algorithm is reliable independent of the number of diffusion steps.

This experiment highlights the potential of a quantum computer in the field of quantum enhanced learning and artificial intelligence. A practical advantage, of course, will become evident once larger percept spaces and general rank- $N$ Q-RPS are employed. Such extensions are, from the theory side, unproblematic given that the modularized nature of the algorithm makes it scalable. An experimental realization of such large-scale quantum enhanced learning will be feasible with the implementation of scalable quantum computer architectures. Meanwhile, all essential elements of Q-RPS have been successfully demonstrated in the proof-of-principle experiment reported here.

Acknowledgments

T. S., S. W., G. S. G. and C. W. acknowledge funding from Deutsche Forschungsgemeinschaft and from Bundesministerium für Bildung und Forschung (FK 16KIS0128). G. S. G. also acknowledges support from the European Commission’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement number 657261. H. J. B. and N. F. acknowledge support from the Austrian Science Fund (FWF) through Grants No. SFB FoQuS F4012 and the START project Y879-N27, respectively.

References

Nielsen and Chuang (2000) M. A. Nielsen and I. L. Chuang, Quantum Computation and Quantum Information (Cambridge University Press, Cambridge, U.K., 2000).
Russell and Norvig (2003) S. J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, 2nd ed. (Pearson Education, 2003).
Grover (1996) L. K. Grover, in Proceedings of the Twenty-eighth Annual ACM Symposium on Theory of Computing, STOC ’96 (ACM, New York, NY, USA, 1996) pp. 212–219.
Shor (1994) P. W. Shor, in Proceedings 35th Annual Symposium on Foundations of Computer Science (1994) pp. 124–134.
Giovannetti et al. (2004) V. Giovannetti, S. Lloyd, and L. Maccone, Science 306, 1330 (2004).
Friis et al. (2017) N. Friis, D. Orsucci, M. Skotiniotis, P. Sekatski, V. Dunjko, H. J. Briegel, and W. Dür, New J. Phys. 19, 063044 (2017).
Lim et al. (2017) K. Lim, Y. Hong, Y. Choi, and H. Byun, PLOS ONE 12, e0173317 (2017).
Silver et al. (2016) D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, Nature 529, 484 (2016).
Mnih et al. (2015) V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, Nature 518, 529 (2015).
Schaeffer et al. (2007) J. Schaeffer, N. Burch, Y. Björnsson, A. Kishimoto, M. Müller, R. Lake, P. Lu, and S. Sutphen, Science 317, 1518 (2007).
Biamonte et al. (2016) J. Biamonte, P. Wittek, N. Pancotti, P. Rebentrost, N. Wiebe, and S. Lloyd, “Quantum machine learning,” (2016), arXiv:1611.09347 .
Sutton and Barto (1998) R. Sutton and A. Barto, Reinforcement learning (The MIT Press, 1998).
Briegel and De las Cuevas (2012) H. J. Briegel and G. De las Cuevas, Sci. Rep. 2, 400 (2012).
Dunjko et al. (2016) V. Dunjko, J. M. Taylor, and H. J. Briegel, Phys. Rev. Lett. 117, 130501 (2016).
Paparo et al. (2014) G. D. Paparo, V. Dunjko, A. Makmal, M. A. Martin-Delgado, and H. J. Briegel, Phys. Rev. X 4, 031002 (2014).
Ristè et al. (2017) D. Ristè, M. P. da Silva, C. A. Ryan, A. W. Cross, A. D. Córcoles, J. A. Smolin, J. M. Gambetta, J. M. Chow, and B. R. Johnson, npj Quantum Information 3, 16 (2017).
Li et al. (2015) Z. Li, X. Liu, N. Xu, and J. Du, Phys. Rev. Lett. 114, 140504 (2015).
Cai et al. (2015) X.-D. Cai, D. Wu, Z.-E. Su, M.-C. Chen, X.-L. Wang, L. Li, N.-L. Liu, C.-Y. Lu, and J.-W. Pan, Phys. Rev. Lett. 114, 110504 (2015).
Hanneke et al. (2010) D. Hanneke, J. P. Home, J. D. Jost, J. M. Amini, D. Leibfried, and D. J. Wineland, Nat. Phys. 6, 13 (2010).
Monz et al. (2016) T. Monz, D. Nigg, E. A. Martinez, M. F. Brandl, P. Schindler, R. Rines, S. X. Wang, I. L. Chuang, and R. Blatt, Science 351, 1068 (2016).
Piltz et al. (2016) C. Piltz, T. Sriarunothai, S. S. Ivanov, S. Wölk, and C. Wunderlich, Sci. Adv. 2, e1600093 (2016).
Debnath et al. (2016) S. Debnath, N. M. Linke, C. Figgatt, K. A. Landsman, K. Wright, and C. Monroe, Nature 536, 63 (2016).
Kielpinski et al. (2002) D. Kielpinski, C. Monroe, and D. J. Wineland, Nature 417, 709 (2002).
Monroe et al. (2014) C. Monroe, R. Raussendorf, A. Ruthven, K. R. Brown, P. Maunz, L.-M. Duan, and J. Kim, Phys. Rev. A 89, 022317 (2014).
Lekitsch et al. (2017) B. Lekitsch, S. Weidt, A. G. Fowler, K. Mølmer, S. J. Devitt, C. Wunderlich, and W. K. Hensinger, Sci. Adv. 3, e1601540 (2017).
Dunjko et al. (2015) V. Dunjko, N. Friis, and H. J. Briegel, New J. Phys. 17, 023006 (2015).
Khromova et al. (2012) A. Khromova, C. Piltz, B. Scharfenberger, T. F. Gloger, M. Johanning, A. F. Varón, and C. Wunderlich, Phys. Rev. Lett. 108, 220502 (2012).
Piltz et al. (2014) C. Piltz, T. Sriarunothai, A. Varón, and C. Wunderlich, Nat. Commun. 5, 4679 (2014).
Wölk et al. (2015) S. Wölk, C. Piltz, T. Sriarunothai, and C. Wunderlich, J. Phys. B: At. Mol. Opt. Phys. 48, 075101 (2015).
Genov et al. (2017) G. T. Genov, D. Schraft, N. V. Vitanov, and T. Halfmann, Phys. Rev. Lett. 118, 133202 (2017).
Mautner et al. (2015) J. Mautner, A. Makmal, D. Manzano, M. Tiersch, and H. J. Briegel, New Generation Computing 33, 69 (2015).
Melnikov et al. (2014) A. A. Melnikov, A. Makmal, and H. J. Briegel, “Projective simulation applied to the grid-world and the mountain-car problem,” (2014), arXiv:1405.5459 .
Makmal et al. (2016) A. Makmal, A. A. Melnikov, V. Dunjko, and H. J. Briegel, IEEE Access 4, 2110 (2016).
Hangl et al. (2016) S. Hangl, E. Ugur, S. Szedmak, and J. Piater, in Proceedings 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE, 2016) p. 2799.
Tiersch et al. (2015) M. Tiersch, E. J. Ganahl, and H. J. Briegel, Sci. Rep. 5, 12874 (2015).
Melnikov et al. (2017) A. A. Melnikov, H. Poulsen Nautrup, M. Krenn, V. Dunjko, M. Tiersch, A. Zeilinger, and H. J. Briegel, “Active learning machine learns to create new quantum experiments,” (2017), arXiv: 1706.00868 .
Szegedy (2004) M. Szegedy, in 45th Annual IEEE Symposium on Foundations of Computer Science (IEEE, 2004).
Magniez et al. (2011) F. Magniez, A. Nayak, J. Roland, and M. Santha, SIAM Journal on Computing 40, 142 (2011).
Friis et al. (2014) N. Friis, V. Dunjko, W. Dür, and H. J. Briegel, Phys. Rev. A 89, 030303 (2014).
Friis et al. (2015) N. Friis, A. A. Melnikov, G. Kirchmair, and H. J. Briegel, Sci. Rep. 5, 18036 (2015).
Vitanov et al. (2015) N. V. Vitanov, T. F. Gloger, P. Kaufmann, D. Kaufmann, T. Collath, M. Tanveer Baig, M. Johanning, and C. Wunderlich, Phys. Rev. A 91, 033406 (2015).

Methods

Theoretical Framework of RPS

A generic picture for modeling autonomous learning scenarios is that of repeated rounds of interaction between an agent and its environment. In each round the agent receives perceptual input (“percepts”) from the environment, processes the input using an internal deliberation mechanism, and finally acts upon (or reacts to) the environment, i.e., performs an “action”Briegel and De las Cuevas (2012). Depending on the reward system in place and the given percept, such actions may be rewarded or not, which leads the agent to update its deliberation process, the agent learns.

Within the projective simulation (PS)Briegel and De las Cuevas (2012)

paradigm for learning agents, the decision-making procedure is cast as a (physically motivated) stochastic diffusion process within an episodic compositional memory (ECM), i.e., a (classical or quantum) random walk in a representation of the agent’s memory containing the interaction history. One may think of the ECM as a network of clips that can correspond to remembered percepts, remembered actions, or combinations thereof. Mathematically, this clip network is described by a stochastic matrix (defining a Markov chain)

P = (p_{i j})

, where the

p_{i j}

with

0 \leq p_{i j} \leq 1

and

\sum_{i} p_{i j} = 1

represent transition probabilities between the clips labeled

i

and

j

with

i, j \in {1, 2, \dots, N}

. The learning process is implemented through an update of the

N \times N

matrix

P

, which, in turn, serves as a basis for the random walks in the clip network. Different types of PS agents vary in their deliberation mechanisms, update rules, and other specifications.

In particular, one may distinguish between PS agents based on “hitting” and “mixing”. For the former type of PS agent, a random walk could, for instance, start from a clip $c_{1}$ called by the initially received percept. The first “step” of the random walk then corresponds to a transition to clips $c_{j}$ with probabilities $p_{1 j}$ . The agent then samples from the resulting distribution ${p_{1 j}}_{j}$ . If such a sample provides an action, e.g., if the clip $c_{k}$ is “hit”, this action is selected as output, otherwise the walk continues on from the clip $c_{k}$ . An advanced variant of the PS model based on “mixing” is reflecting projective simulation (RPS)Paparo et al. (2014). There, the Markov chain is first “mixed”, i.e., an appropriate number²²2The mixing time depends on the spectral gap $δ$ of the Markov chain $P$

, i.e., the difference between the two largest eigenvalues of

P

Paparo et al. (2014). of steps are applied until the stationary distribution is attained (approximately), before a sample is taken. This, or other implementations of random walks in the clip network provide the basis for the PS framework for learning. The classical PS framework can be used to solve standard textbook problems in reinforcement learningMautner et al. (2015); Melnikov et al. (2014); Makmal et al. (2016), and has recently been applied in advanced roboticsHangl et al. (2016), adaptive quantum computationTiersch et al. (2015), as well as in the machine-generated design of quantum experimentsMelnikov et al. (2017).

Here, we focus on RPS agents, where the deliberation process based on mixing allows for a speed-up of Q-RPS agents w.r.t. to their classical counterpartsPaparo et al. (2014). In contrast to basic hitting-based PS agents, the clip network of RPS agents is structured into several sub-networks, one for each percept clip, and each with its own stochastic matrix $P$ . In addition to being stochastic, these matrices specify Markov chains which are ergodicPaparo et al. (2014)

, which ensures that the Markov chain in question has a unique stationary distribution, i.e., a unique eigenvector

α

with eigenvalue

+ 1

P α = α

. Starting from any initial state, continued application of

P

(or its equivalent in the quantized version) mixes the Markov chain, leaving the system in the stationary state.

As part of their deliberation process, RPS agents generate stationary distributions over their clip space, as specified by $P$ , which is updated as the agent learns. These distributions have support over the whole sub-network clip space, and additional specifiers – flags – are used to ensure an output from a desired sub-set of clips. For instance, standard agents are presumed to output actions only, in which case only the actions are “flagged”³³3Such flags are rudimentary emoticons defined inBriegel and De las Cuevas (2012). This ensures that an action will be output, while maintaining the relative probabilities of the actions. The same mechanism of flags, which can be thought of as short-term memory, is also used to eliminate iterated attempts of actions which did not yield rewards in recent time-steps. This leads to a more efficient exploration of correct behavior.

In the quantum version of RPS, each clip $c_{i}$

is represented by a basis vector

| i ⟩

in a Hilbert space

H

. The mixing process is then realized by a diffusion process on two copies of the original Hilbert space. On the doubled space

H \otimes H

a unitary operator

W (P)

(called the Szegedy walk operatorSzegedy (2004); Magniez et al. (2011)) and a quantum state

| α^{'} ⟩

with

W (P) | α^{'} ⟩ = | α^{'} ⟩

take the roles of the classical objects

P

and

α

. Both

W (P)

and

| α^{'} ⟩

depend on a set of unitaries

U_{i}

H

that act as

U_{i} | 0 ⟩ = \sum_{j} \sqrt{p_{i j}} | j ⟩

for some reference state

| 0 ⟩ \in H

, i.e.,

| α^{'} ⟩ = \sum_{i} \sqrt{a_{i}} | i ⟩ \otimes U_{i} | 0 ⟩

. The more intricate construction of

W (P)

is given in detail inDunjko et al. (2015). The feature of the quantum implementation of RPS that is crucial for us here is an amplitude amplification similar to Grover’s algorithmGrover (1996), which incorporates the mixing of the Markov chain and allows outputting flagged actions after an average of

O (1 / \sqrt{ϵ})

calls to

W (P)

, where

ϵ

is the probability of sampling an action from the stationary distribution. The algorithm achieving this is structured as follows. After an initialization stage where

| α^{'} ⟩

is prepared, a number of diffusion steps are carried out. Each such step consists of two parts. The first part is a reflection over the states corresponding to actions in the first copy of

H

, i.e., an operation

{ref}_{A}

= 2 n \sum i = 1 | i ⟩ ⟨ i | \otimes 1 - 1 \otimes 1,

(10)

where $A = span {| 1 ⟩, \dots, | n ⟩}$ denotes the subspace of the clip network corresponding to actions. In the second part, an approximate reflection over the state $| α^{'} ⟩$ , the mixing, is carried out, i.e., an operation designed to approximate ${ref}_{α^{'}} = 2 | α^{'} ⟩ ⟨ α^{'} | - 1$ Paparo et al. (2014). This second step involves $O (1 / \sqrt{δ})$ calls to $W (P)$ . The two-part diffusion steps are repeated $O (1 / \sqrt{ϵ})$ times before a sample is taken from the resulting state by measuring in the basis ${| i ⟩}_{i = 1, \dots, N}$ . If an action is sampled, the algorithm concludes and that action is chosen as output. Otherwise, all steps are repeated. Since the algorithm amplifies the probability of sampling an action (almost) to unity, carrying out the deliberation procedure with the help of such a Szegedy walk hence requires an average of $O (1 / \sqrt{δ ϵ})$ calls to $W (P)$ . In comparison, a classical RPS agent would require an average of $O (1 / δ)$ applications of $P$ to mix the Markov chain, and an average of $O (1 / ϵ)$ samples to find an action. Q-RPS agents can hence achieve a quadratic speed-up in their reaction time.

Here, it should be noted that, its elegance not withstanding, the construction of the approximate reflection for general RPS networks is extremely demanding for current quantum computational architectures. Most notably, this is due to the requirement of two copies of $H$ , on which frequently updated coherent conditional operations need to be carried outDunjko et al. (2015); Friis et al. (2014, 2015). However, as we shall explain now, these requirements can be circumvented for the interesting class of rank-one Markov chains. In this special case, the entire Markov chain $P$ can be represented on one copy of $H$ by a single unitary $U_{P} = U_{i} \forall i$ , since all columns of $P$ are identical. Conceptually, this simplification corresponds to a situation where each percept-specific clip network contains only actions and the Markov chain is mixed in one step ( $δ = 1$ ). In such a case one uses flags to mark desired actions. Interestingly, these minor alterations also allow to establish a one-to-one correspondence with the hitting-based basic PS using two-layered networks, which has been applied in classical task environmentsMautner et al. (2015); Melnikov et al. (2014); Makmal et al. (2016); Hangl et al. (2016); Tiersch et al. (2015); Melnikov et al. (2017).

Let us now discuss how the algorithm above is modified for the rank-one case with the flagging mechanism in place. First, we restrict $A$ to be the subspace of the flagged actions only, assuming that there are $n ≪ N$ of these, and we denote the corresponding probabilities within the stationary distribution by $a_{1}, \dots, a_{n}$ . In the initialization stage, the state $| α ⟩ = \sum_{i = 1, \dots, N} \sqrt{a_{i}} | i ⟩$ is prepared. Then, an optimal number of $k$ diffusion stepsGrover (1996) is carried out, where

k

= round (\frac{π}{4 \sqrt{ϵ}} - \frac{1}{2}),

(11)

and $ϵ = \sum_{i = 1, \dots, n} a_{i}$ is the probability to sample a flagged action from the stationary distribution. Within the diffusion steps, the reflections over all actions of Eq. (10) are replaced by reflections over flagged actions, i.e.,

{ref}_{A} = 2 n \sum i = 1 | i ⟩ ⟨ i | - 1 .

(12)

In the rank-one case, the reflections over the stationary distribution $α$ becomes an exact reflection ${ref}_{α}$ over the state $| α ⟩$ and can be carried out on one copy of $H$ Dunjko et al. (2015). After the diffusion steps, a sample is taken and the agent checks if the obtained action is marked with a flag. If this is the case, the action is chosen as output, otherwise the algorithm starts anew.

While a classical RPS agents requires an average of $O (1 / ϵ)$ samples until obtaining a flagged action, this number reduces to $O (1 / \sqrt{ϵ})$ for Q-RPS agents. This quantum advantage is particularly pronounced when the overall number of actions is very large compared to $n$ and the environment is unfamiliar to the agent or has recently changed its rewarding pattern, in which case $ϵ$ may be very small. Given some time, both agents learn to associate rewarded actions with a given percept, suitably add or remove flags, and adapt $P$ (and by extension $α$ ). In the short run, however, classical agents may be slow to respond and the advantage of a Q-RPS agent becomes apparent. Despite the remarkable simplification of the algorithm for the rank-one case with flags, the quadratic speed-up is hence preserved.

Experimental Details

We discuss some details of the experimental implementation. The detailed pulse sequence for the reported experiments is shown in Fig. 5. Radio frequency (RF) pulses tuned to the respective qubit resonance frequencies near 12.6 GHz are applied for all coherent manipulations. The RF power is carefully adjusted for each ion in order to achieve an equal Rabi frequency of $20.92 (3)$ kHz. For each single-qubit rotation, $R_{j} (θ, ϕ)$ during the preparation and diffusion steps, $θ_{1}$ and $θ_{2}$ represent the rotation angle according to Eq. (3 - 5) (main text). We replace $R_{z} (\frac{π}{2})$ by $R (\frac{π}{2}, \frac{π}{2}) R (\frac{π}{2}, 0) R (\frac{π}{2}, \frac{3 π}{2})$ and $R_{z} (- \frac{π}{2})$ by $R (\frac{π}{2}, \frac{π}{2}) R (\frac{π}{2}, π) R (\frac{π}{2}, \frac{3 π}{2})$ .

During the conditional evolution time, a set of ten UR14 RF $π$ -pulses (equaling a total of 140 pulses) is appliedGenov et al. (2017) to protect the qubits from decoherence by dynamical decoupling (DD). Each set is comprised of 14 error correcting pulses (Fig. 5) with appropriately chosen phase $ϕ$ :

(0, \frac{6 π}{7}, \frac{4 π}{7}, \frac{8 π}{7}, \frac{4 π}{7}, \frac{6 π}{7}, 0, 0, \frac{6 π}{7}, \frac{4 π}{7}, \frac{8 π}{7}, \frac{4 π}{} 7, \frac{6 π}{7}, 0) .

Since the phases of the $π$ -pulses are symmetrically arranged in time, only the first seven pulses are shown in Fig. 5. The last pulse is also shown to visualize the spacing of these pulses with respect to the start and end of evolution time, compared to the intermediate pulses. The maximum interaction time of 30 ms required to realize the deliberation algorithm (corresponding to 7 diffusion steps) presented in the main text is 60 times longer than the qubit coherence time. Such a long coherent interaction time is accomplished by the DD pulses applied to each qubit simultaneously.

Laser light near 369 nm is applied to the ions for cooling, state preparation, and state selective read-out as described in the main text. The durations are: 30 ms for Doppler cooling, 100 ms for sideband cooling on the center-of-mass mode, 100 ms for sideband cooling on the stretch mode, 0.25 ms for initialization in state $| 0 ⟩$ of the ions, and 2 ms for detection.

Projective measurements on both qubits are performed to determine the states $| 00 ⟩$ , $| 01 ⟩$ , $| 10 ⟩$ , and $| 11 ⟩$ . Two thresholds are used to distinguish between dark and bright states of the ions, thus discarding 10% of all measurements as ambiguous events with a photon count that lies in the region of two partially overlapping Poissonian distributions representing the dark and bright states of the ionsVitanov et al. (2015); Wölk et al. (2015).

Figure 5: Experimental sequence for Q-RPS. RF1 and RF2 each indicate a time axis for a qubit. The qubits are prepared in the desired input states using single-qubit rotations implemented by applying RF pulses. For each RF pulse, the two parameters within the parentheses represent the specific rotation angle and phase according to Eq. (3 - 5) in the main text. Also, dynamical decoupling (DD) during conditional evolution, ( $U_{z z} (π / 2)$ ) (indicated by a green box) is implemented using RF pulses (indicated in yellow). Ten sets of 14 pulses each (UR14) are appliedGenov *et al.* (2017) during the evolution time $τ = 4.24$ ms with a J-coupling between the two ions of $2 π \times 59$ Hz. The diffusion step is repeated $k$ times according to Eq. (11) in Methods. Laser light near 369 nm is used for cooling and to initialize the ions in the qubit state $| 0 ⟩ \equiv |^{2} S_{1 / 2}, F = 0 ⟩$ . At the end of the coherent manipulation, laser light is used again for state selective detection and also for Doppler cooling.

Supplementary Materials

In this section, we discuss deviations of the experimental data from idealized theory predictions. In particular, for the chosen values of $ϵ$ and the corresponding optimal $k (ϵ)$ , it is expected that the probability of obtaining a flagged action is close to $100 %$ . However, the success probability in our experiment lies between $66 %$ (for $k = 7$ ) and $88 %$ (for $k = 1$ ). In what follows, we discuss several reasons for this.

Scaling error

		Theory			Theory
$k$	$a_{00}$	$a_{01}$	$ϵ$	$b_{00}$	$b_{01}$	$~ ϵ$
1	0.1371	0.1371	0.2742	0.4966	0.4966	0.9932
2	0.0493	0.0493	0.0987	0.4996	0.4996	0.9993
3	0.0252	0.0252	0.0504	0.4999	0.4999	0.9998
4	0.0152	0.0152	0.0305	0.5000	0.5000	1.0000
5	0.0102	0.0102	0.0204	0.5000	0.5000	1.0000
6	0.0073	0.0073	0.0146	0.5000	0.5000	1.0000
7	0.0055	0.0055	0.0110	0.5000	0.5000	1.0000
					Experiment
$k$				$b_{00}$	$b_{01}$	$~ ϵ$
1				0.449(15)	0.440(15)	0.89(2)
2				0.347(15)	0.353(15)	0.70(2)
3				0.438(16)	0.334(15)	0.77(2)
4				0.422(15)	0.336(15)	0.76(2)
5				0.407(17)	0.331(16)	0.74(2)
6				0.431(17)	0.324(16)	0.76(2)
7				0.365(15)	0.299(14)	0.66(2)

Table 1: Experimentally realized success probabilities. Initial theoretical probabilities,

ϵ

, of finding a flagged action within the stationary distribution for various diffusion steps are shown. Success probabilities (

~ ϵ

), that are theoretically calculated and experimentally measured, for diffusion steps

k

= 1-7 are also shown.

Even in an ideal scenario without noise or experimental imperfections the success probability $~ ε$ , as defined in Eq.(4) of the main text, after $k$ diffusion steps is usually not equal to unity, and depends on the specific value of $ϵ$ . This behavior originates from the step-wise increase of the number of diffusion steps $k = round (\frac{π}{4 \sqrt{ϵ}} - \frac{1}{2})$ in the algorithm. The success probability is hence only $100 %$ if $k$ is an integer without rounding. The change of the ideal success probability with deviations of $ϵ$ from such specific values is largest for small numbers of diffusion steps (e.g., $k = 1$ ) and can drop down to $82 %$ (neglecting the cases where it is not advantageous to use a quantum algorithm at all). For larger numbers of diffusion steps, the exact value of $ϵ$ does not play an important role any more for the ideal success probability provided that the correct number of diffusion steps is chosen. For example, for $k = 6$ , the ideal success probability is larger than $98 %$ independently of the exact value of $ϵ$ . Throughout this paper, we have chosen $ϵ$ in such a way, that $(\frac{π}{4 \sqrt{ϵ}} - \frac{1}{2})$ is always close to an integer (see Tab. 1), such that the deviation from a $100 %$ success probability due to the theoretically chosen $ϵ$ is negligible compared to other error sources.

However, in a real experiment, the initial state, and therefore $ϵ$ , can only be prepared with a certain accuracy. This can lead to an inaccurate estimation of the optimal number of diffusion steps. As opposed to the ideal case, an assumed accuracy of $ϵ \pm 1 %$ for the preparation only has a small effect on the success probability $~ ϵ$ (drop of less than $5 %$ ) for $ϵ ≫ 0.01$ , corresponding to $k \leq 3$ . However, when $ε$ does not fulfil the aforementioned condition and approaches $\approx 0.01$ from above, corresponding to $k = 6$ , then the success probability drops down to $~ ϵ = 70 %$ due to a non-optimal choice of $k$ .

Figure 6: Detuning errors. The influence of the detuning of the RF pulses on the fidelity of the Q-RPS algorithm is shown for three different values of the relative detuning $Δ ω / Ω$ . Black lines indicate the results of numerical simulations of the complete Q-RPS algorithm taking different values of the relative detuning $Δ ω / Ω$ into account. Most of the experimental data (red circles) lie close to the $Δ ω / Ω = 0.03$ line.

The preparation accuracy depends on the detuning $Δ ω$ of the RF pulses for single-qubit rotations as well as on the uncertainty $Δ Ω$ in the determination of the Rabi frequency $Ω$ . The calibration of our experiment revealed $Δ ω / Ω < 0.05$ and $Δ Ω / Ω = 0.0015$ leading to an error in $ϵ$ of $\pm 2.5 \cdot 10^{- 3}$ and a decrease of the success probability $~ ϵ$ of less than $0.04$ . The detuning $Δ ω$ and the uncertainty of the Rabi frequency $Δ Ω$ not only influence the state preparation at the beginning of the quantum algorithm, but also its fidelity, as is detailed in the next paragraph.

To prevent decoherence during conditional evolution, we use $140$ MW $π$ -pulses per diffusion step and ion. Therefore, already a small detuning influences the fidelity of the algorithm. Consequently, the error induced by detuning is identified as the main error source leading, for example, to $~ ε \approx 0.70$ for $k = 6$ and $Δ ω / Ω = 0.03$ . This error is much larger than the error caused by dephasing (that is still present after DD is applied), or the detection error. To estimate the error by dephasing, we assume an exponential decay with $γ τ \approx 1 / 14$ for a single diffusion step of duration $τ \approx 4$ ms which would lead to $~ ε \approx 0.90$ for $k = 6$ . Here, $γ$ indicates the experimentally diagnosed rate of dephasing, and $τ$ is the time of coherent evolution. The influence of the detuning on the cost of our algorithm is shown in Fig. 6 for different detunings. Here, we simulated the complete quantum algorithm including the experimentally determined dephasing and detection errors for $Δ ω / Ω \in {0.01, 0.03, 0.05}$ . The experimental data is consistent with an average relative detuning of $Δ ω / Ω = 0.03$ . Note that the detuning not only influences the single-qubit rotations that are an integral part of the quantum algorithm, but also leads to errors during the conditional evolution when dynamical decoupling pulses are applied.

Ratio error

In the ideal algorithm, the output ratio $r_{f} = b_{00} / b_{01}$ of the two flagged actions represented by the states $| 00 ⟩$ and $| 01 ⟩$ at the end of the algorithm equals the input ratio $r_{i}$ . However, in the experiment we have observed deviations from $r_{f} / r_{i} = 1$ . During the measurements for the investigation of the scaling behavior (Fig. 3 in the main text), we fixed $r_{i} = 1$ . The observed output ratios are varying by $0.98 \leq r_{f} / r_{i} \leq 1.33$ . That is, the probability $b_{00}$ to obtain the state $| 00 ⟩$ is increased w.r.t. $b_{01}$ . Also during the measurement testing the output ratio, we observe that the output ratios are larger than the input ratios.

		Theory			Experiment
$k$	$a_{00}$	$a_{01}$	$r_{i}$	$b_{00}$	$b_{01}$	$r_{f}$
1	0.00271	0.27144	0.01	0.061(7)	0.809(12)	0.075(9)
1	0.07257	0.20159	0.36	0.290(14)	0.583(15)	0.50(3)
1	0.11383	0.16032	0.71	0.415(15)	0.466(15)	0.89(4)
1	0.14107	0.13309	1.06	0.488(15)	0.389(15)	1.25(6)
1	0.16040	0.11376	1.41	0.519(13)	0.351(12)	1.48(6)
1	0.17482	0.09933	1.76	0.566(15)	0.305(14)	1.85(10)
1	0.13708	0.13708	1.00	0.468(16)	0.401(16)	1.17(6)
3	0.00458	0.04578	0.10	0.127(10)	0.718(14)	0.176(14)
3	0.01633	0.03402	0.48	0.301(15)	0.518(16)	0.58(3)
3	0.02328	0.02707	0.86	0.442(16)	0.451(16)	0.98(5)
3	0.02788	0.02248	1.24	0.510(16)	0.354(15)	1.44(8)
3	0.03114	0.01922	1.62	0.551(16)	0.305(14)	1.81(10)
3	0.03357	0.01679	2.00	0.586(15)	0.268(13)	2.19(12)

Table 2: Input and output distributions. Input and output ratios,

r_{i}

and

r_{f}

respectively, of the two flagged actions represented by the states

| 00 ⟩

and

| 01 ⟩

for diffusion steps

k

= 1 and

k

= 3 are shown.

An asymmetric detection error could be the cause for this observation. Typical errors in our experiment are given by the probability to detect a bright ion ( $| 1 ⟩$ ) with a probability of $d_{B} = 0.06$ as dark, and a dark ion ( $| 0 ⟩$ ) with a probability of $d_{D} = 0.02$ as bright. In Fig. 7 we compare the measured output ratios with the calculated output ratios assuming the above mentioned detection errors only.

Fig. 7 shows that the experimental data, both for one step and for three steps, are well approximated by the simulation when the experimentally determined detection error is taken into account. Thus, the deviation of the measured ratios from the ideal result can be traced back mainly to the unbalanced detection error. In addition, also errors in the preparation of the input states play a role, especially when preparing very large or very small ratios leading to either $a_{00}$ or $a_{01}$ being close the the preparation accuracy of $\leq 2.5 \cdot 10^{- 3}$ . At the same time, the detuning plays a less prominent role in these measurements because fewer dynamical decoupling pulses where required due to the small number of diffusion steps. Moreover, the detuning during these measurements could be kept below $Δ ω / Ω = 0.03$ leading to an average success probability of $~ ϵ = 0.85 %$ also for $k = 3$ diffusion steps compared to $~ ϵ = 0.77 %$ for $k = 3$ during the measurements investigating the scaling (see Tab. 1).

Speeding-up the decision making of a learning agent using an ion trap quantum processor

Authors

Scheme of thinking quantum systems

Quantum Stream Learning

Machine learning for long-distance quantum communication

Machine-learning-enhanced quantum sensors for accurate magnetic field imaging

Order Effects for Queries in Intelligent Systems

Framework for learning agents in quantum environments

Predicting human-generated bitstreams using classical and quantum models

Introduction