Understanding Quantum Software Engineering Challenges An Empirical Study on Stack Exchange Forums and GitHub Issues

With the advance in quantum computing, quantum software becomes critical for exploring the full potential of quantum computing systems. Recently, quantum software engineering (QSE) becomes an emerging area attracting more and more attention. However, it is not clear what are the challenges and opportunities of quantum computing facing the software engineering community. This work aims to understand the QSE-related challenges perceived by developers. We perform an empirical study on Stack Exchange forums where developers post-QSE-related questions answers and Github issue reports where developers raise QSE-related issues in practical quantum computing projects. Based on an existing taxonomy of question types on Stack Overflow, we first perform a qualitative analysis of the types of QSE-related questions asked on Stack Exchange forums. We then use automated topic modeling to uncover the topics in QSE-related Stack Exchange posts and GitHub issue reports. Our study highlights some particularly challenging areas of QSE that are different from that of traditional software engineering, such as explaining the theory behind quantum computing code, interpreting quantum program outputs, and bridging the knowledge gap between quantum computing and classical computing, as well as their associated opportunities.

READ FULL TEXT VIEW PDF

Authors

page 1

page 2

page 3

page 4

04/25/2022

Bug Characteristics in Quantum Software Ecosystem

With the advance in quantum computing in recent years, quantum software ...
03/31/2022

Software Engineering for Quantum Programming: How Far Are We?

Quantum computing is no longer only a scientific interest but is rapidly...
10/01/2021

An analysis of open source software licensing questions in Stack Exchange sites

Free and open source software is widely used in the creation of software...
10/23/2020

Exploring Research Interest in Stack Overflow – A Systematic Mapping Study and Quality Evaluation

Platforms such as Stack Overflow are available for software practitioner...
05/28/2019

Analyzing and Supporting Adaptation of Online Code Examples

Developers often resort to online Q&A forums such as Stack Overflow (SO)...
06/16/2021

OpenSSLNTRU: Faster post-quantum TLS key exchange

Google's CECPQ1 experiment in 2016 integrated a post-quantum key-exchang...
06/27/2019

What Do Developers Ask About ML Libraries? A Large-scale Study Using Stack Overflow

Modern software systems are increasingly including machine learning (ML)...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Over the past decades, quantum computing has made steady and remarkable progress [33, 40, 81]. For example, IBM Quantum[30] now supports developers to develop quantum applications using its programming framework and execute them on its cloud-based quantum computers. Based on the quantum mechanics principles of superposition (quantum objects can be in different states at the same time) [20] and entanglement (quantum objects can be deeply connected without direct physical interaction) [61], quantum computers are expected to make revolutionary computation improvement over today’s classical computers [46]. In particular, quantum computing is expected to help solve the computational problems that are difficult for today’s classical computers, including problems in cryptography, chemistry, financial services, medicine, and national security [53].

The success of quantum computing will not be accomplished without quantum software. Several quantum programming languages (e.g., QCL [51]) and development tools (e.g., Qiskit[1] have been developed since the first quantum computers. Large software companies like Google[25], IBM[30], and Microsoft[43]

have developed their technologies for quantum software development. Quantum software developers have also achieved some preliminary success in applying quantum software to certain computational areas (e.g, machine learning 

[11], optimization [26], cryptography [38], and chemistry [56]). However, there still lacks large-scale quantum software. Much like Software Engineering is needed for developing large-scale traditional software, the concept of Quantum Software Engineering (QSE) has been proposed to support and guide the development of large-scale, industrial-level quantum software applications. This concept has been gaining more and more attention recently [53, 81, 54]. QSE aims to apply or adapt existing software engineering processes, methods, techniques, practices, and principles to the development of quantum software applications, or create new ones [53]. Pioneering work sheds light on new directions for QSE, such as quantum software processes & methodologies [45], quantum software modeling [7], and design of quantum hybrid systems [55]. In the meanwhile, we observe an exponential increase of discussions related to quantum software development on technical Q&A forums such as Stack Overflow(e.g. from 8 in 2010 to 1434 in 2020). We also notice an increasing number of quantum software projects hosted on GitHub, where developers use issue reports to track their development and issue fixing processes. Such technical Q&As and issue reports may communicate developers’ faced challenges when developing quantum software applications.

In this paper, we aim to understand the challenges perceived by quantum software developers and seek opportunities for future QSE research and practice. In particular, we examine technical Q&A forums where developers ask QSE-related questions, and GitHub issue reports where developers raise QSE-related issues. We apply a series of heuristics to search and filter Q&A posts that are related to QSE and to search and filter GitHub projects that are related to quantum software. In total, we extract and analyze 3,117 Q&A posts and 43,979 Github issues that are related to QSE. We combine manual analysis and automated topic modeling to examine these Q&A posts and Github issues, to understand the QSE challenges developers are facing. In particular, our study aims to answer the three following research questions (RQs):

RQ1:

What types of QSE questions are asked on technical forums?

To understand the intention behind developers’ questions on technical forums and the types of information that they are seeking, we manually examined a statistically representative sample of questions. We extended a previous taxonomy from prior work [9] and found nine categories of questions. Our results highlight the need for future efforts to support developers’ quantum program development, in particular, to develop learning resources, to help developers fix errors, and to explain the theory behind quantum computing code.

RQ2:

What QSE topics are raised in technical forums? The QSE-related posts may reflect developers’ challenges when learning or developing quantum programs. To understand their faced challenges, we use topic models to extract the semantic topics in their posts. We derived nine topics including traditional software engineering topics (e.g., environment management and dependency management) and QSE-specific topics (e.g., quantum execution results and quantum vs. classical computing). We highlighted some particularly challenging areas for QSE, such as interpreting quantum program outputs, understanding quantum algorithm complexity, and bridging the knowledge gap between quantum computing and classical computing.

RQ3:

What QSE topics are raised in the issue reports of quantum-computing projects? Issue reports of quantum computing projects record developers’ concerns and discussions when developing these projects. Thus, we analyze the topics in the issue reports to understand the challenges are developers facing in practical quantum computing projects. We observe that the QSE-related challenges that we derived from forum posts indeed impact practical quantum program development in these GitHub projects, while GitHub issues bring new perspectives on developers’ faced challenges (e.g., on specific quantum computing applications such as machine learning). We also observe that such challenges are general among quantum computing projects.

Paper organization. The rest of the paper is organized as follows. In Section II we discuss the background about quantum software engineering and the related work. Then, in Section III we describe the design of our study. In Section IV we present our results. Section V discusses threats to the validity of our findings. Finally, Section VI concludes the paper.

Ii Background and Related Work

This study aims to understand the quantum software engineering challenges through examining technical forum posts and GitHub issue reports. In this section, we present the background and prior work related to our study. First, we describe the background and related work of quantum computing, quantum programming, and quantum software engineering. Then, we discuss prior work that performs topic analysis on technical forum posts and issue reports.

Ii-a Quantum Computing

Quantum computers aim to leverage the principles of quantum mechanics such as superposition and entanglement to provide computing speed faster than today’s classical computers. While classical computers use bits in the form of electrical pulses to represent 1s and 0s, quantum computers use quantum bits or Qubits

in the form of subatomic particles such as electrons or photons to represent 1s and 0s. A Qubit, unlike a classical bit, can be 0 or 1 with a certain probability, which is known as the

superposition principle [31]. In other words, a quantum computer consisting of Qubits is in many different states at the same time. When a Qubit is measured, it collapses into a deterministic classical state. The status of two or more of Qubits can be correlated (or entangled) in the sense that changing the status of one Qubit will change the status of the others in a predictable way, which is known as the entanglement phenomenon [31]. The superposition and entanglement phenomenons give quantum computers advantages over classical computers in performing large-scale parallel computation [31].

Similar to classical logic gates (e.g., AND, OR, NOT), quantum logic gates (or quantum gates) alter the states (the probability of being 0 or 1) of the input Qubits. Like classical digit circuits, quantum circuits are collections of quantum logic gates interconnected by quantum wires. Figure 1 illustrates the architecture of a quantum computer [62, 81]. The architecture contains two layers: a quantum computing layer where the quantum physics and circuits reside, and a classical computing layer where the quantum programming environment and software applications reside.

  • Physical building blocks: physical realization of Qubits and their coupling/interconnect circuitry.

  • Quantum logic gates: physical circuitry for quantum logic gates.

  • Quantum-classical computer interface: the hardware and software that provides the boundary between classical computers and the quantum computing layer.

  • Quantum programming environment: quantum programming languages and development environment.

  • Business applications: quantum software applications (based on quantum programming languages) that meet specific business requirements

Fig. 1: The architecture of a quantum computer [62, 81]

.

Ii-B Quantum Programming

Quantum computing as a new general paradigm can massively influence how software is developed [53, 81, 22]. Quantum programming is the process to design an executable quantum program to accomplish a specific task [22]

. Quantum programming uses syntax-based notations to represent and operate quantum circuits and gates. Early efforts of quantum programming language development focused on the quantum Turing machine 

[19] but did not produce practical quantum programming languages. Later efforts have turned to the quantum circuits model where the quantum system is controlled by a classical computer [34]. This concept has given birth to many new quantum programming languages such as qGCL [60], LanQ [44], Q# [68] and Qiskit [1]. Prior work conducted extensive exploration along the lines of quantum programming [22] and quantum software development environments [35]. The survey [81] also provides a comprehensive overview of research works along these lines.

Ii-C Quantum Software Engineering

Quantum software engineering (QSE) is still in its infancy. As the result of the first International Workshop on Quantum Software Engineering & Programming (QANSWER), researchers and practitioners proposed the “Talavera Manifesto” for quantum software engineering and programming, which defines a set of principles about QSE [53], including: (1) QSE is agnostic regarding quantum programming languages and technologies; (2) QSE embraces the coexistence of classical and quantum computing; (3) QSE supports the management of quantum software development projects; (4) QSE considers the evolution of quantum software; (5) QSE aims at delivering quantum programs with desirable zero defects; (6) QSE assures the quality of quantum software; (7) QSE promotes quantum software reuse; (8) QSE addresses security and privacy by design; and (9) QSE covers the governance and management of software.

Zhao [81] performed a comprehensive survey of the existing technology in various phases of quantum software life cycle, including requirement analysis, design, implementation, testing, and maintenance. Prior work [45, 54, 7, 55] also discussed challenges and potential directions in QSE research, such as modeling [7] and quantum software processes & methodologies [45], and design of quantum hybrid systems [55]. Different from prior work, this work makes the first attempt to understand the challenges of QSE perceived by practitioners.

Ii-D Topic Analysis of Technical Q&As

Prior work performs rich studies on technical Q&A data, especially on Stack Exchange data[71]. Here we focus on prior work that performs topic analysis on technical Q&A data. Topic models are used extensively in prior work to understand the topics of general Stack Overflow posts and the topic trends [73, 16, 7, 4]. Prior work also leverages topic models to understand the topics of Stack Overflow posts related to specific application development domains, such as mobile application development [37, 58], client application development [72], machine learning application development [5], as well as concurrency [3] and security [77] related development. In addition, prior work leverages topic models to understand non-functional requirements communicated in Stack Overflow posts [84, 82]. Zhang et al. [80] use topic models to detect duplicate questions in Stack Overflow. Finally, Treude et al. [69] proposes an automated approach to suggest configurations of topic models for Stack Overflow data. Most of these studies use the Latent Dirichlet Allocation (LDA) algorithm or its variants to extract topics from the technical Q&A data. In this work, we also leverage the widely used LDA algorithm to extract topics from the technical Q&A data related to quantum software enginering.

Ii-E Topic Analysis of Issue Reports

Issue reports have been widely explored in prior work. Here we focus on studies that apply topic analysis on issue report data. Prior work leverages topic models to automatically assign issue reports to developers (a.k.a. bug triage) [79, 76, 47, 75]. These studies first uses topic models to categorize the textual information in the issue reports, then learn mappings between the categorized textual information and developers. Prior work also leverages topic models to automatically detect duplicate issue reports based on the similarity of their topics [29, 49, 83]. Nguyen et al. [48] use topic models to associate issue reports and source code based on their similarities, in order to help developers narrow down the searched source code space when resolving an issue. Finally, prior work also studies the trends of topics in issue reports [39, 2]. LDA and its variants are the most popular topic modeling approaches used in these studies. Therefore, we also leverage LDA to extract topics from GitHub issue reports related to QSE.

Iii Experiment Setup

This section describes the design of our empirical study.

Iii-a Overview

Figure 2 provides an overview of our empirical study. We study QSE-related posts on Stack Exchange (SE) forums and the issue reports of quantum computing GitHub projects. From Stack Exchange forums, we first use tags to filter QSE-related posts. In RQ1, we manually analyze a statistically representative sample of these posts to understand the type of information sought by developers. In RQ2, we use automated topic models to analyze the topics of these posts and their characteristics. From GitHub repositories, we first apply a set of heuristic rules to filter the quantum computing projects. Then we extract the issue reports of these quantum computing projects. Finally, we perform topics modeling on these issue reports to analyze the topics in the textual information of the issue reports (RQ3). We describe the details of our data collection and analysis approaches in the rest of this section.

Fig. 2: Overview of our empirical study

Iii-B Stack Exchange forums data collection

We follow three steps to collect QSE related data from Stack Exchange forums. First, we collect Q&A data from four Stack Exchange forums. Second, we identify a set of tags that are related to QSE. Finally, we use the identified tags to select the posts that are related to QSE. We explain the steps below.

Step 1: Collecting technical Q&A data. We extract technical Q&A data from four Stack Exchange forums: Stack Overflow[67], Quantum Computing Stack Exchange[66], Computer Science Stack Exchange[64]

, and Artificial Intelligence Stack Exchange

[63]. We consider the Stack Overflow forum as it contains posts related to quantum programming and it is widely used for studying various software engineering topics (e.g., mobile app development [37, 58], machine learning application development [5], etc.). We consider the other three forums because they contain posts that discuss topics related to quantum computing and quantum programming. We extracted the post data from these forums with the help of the Stack Exchange Data Explorer[65]. Stack Exchange data explorer holds an up to date data for these forum between 08-2008 and 03-2021.

Step 2: Identifying tags related to QSE. The studied Stack Exchange forums use user defined tags to categorize questions. We follow two sub-steps to select the tags that are related to QSE. We started by searching for questions with the tag “quantum-computing” in the entire Stack Exchange dataset through the data exchange explorer. We obtained 254 questions tagged with “quantum-computing” from the studied forums. After manually inspecting the 30 most voted questions, we selected an initial tag set consisting of ten tags including “quantum-computing”, “qiskit”, “qsharp”, “q#”, “quantum-development”, “quantum-circuit”, “ibmq”, “quantum-ai”, “qubit” and “qutip”. Then we extracted the questions related to from the initial dataset and obtained a new set of questions . In order to expand the initial tag set, we extracted the frequently co-occurring tags with from and build a new tag set .

Not all the tags in are related to quantum computing. To determine the final tag set , following previous work [70] [52], we filter the tags in based on their relationships when the initial tag set . For each tag t in , we calculate:

(1)
(2)

To select a tag t, the value of significance-relevance , need to be higher than a threshold we set. To select the optimal threshold values for and , we experimented with a set of values respectively between 0.05, 0.35 and 0.001, 0.03. For each and and for each tag above the threshold, we inspected the top 10 most voted posts and verified if the tag is related to QSE, we ended up with the optimal threshold respectively equal to 0.005 and 0.2 which are consistent with previous work [15] [52]. The final tag set is formed of 37 tags in total. Since quantum computing is a wide topic and our focus is QSE, we further manually inspected the description of each tag t in and the top 10 questions of each tag in each studied forum to remove tags that are not related to QSE. Finally our tag set was reduced from 37 to 18 tags (14 unique tags as different forums have tags with the same names). Table I lists our final set of tags.

Stack Ex. forum Tag set #Q #A
Stack overflow
post-quantum-cryptography, q#,
quantum-computing, qiskit, qcl,

qutip, qubit, tensorflow-quantum

250 183
Quantum computing
programming, classicalcomputing,
q#, qiskit, cirq, ibm-q-experience,
machine-learning, qutip
1534 778
Computer science
quantum-computing 238 117
Artificial intelligence
quantum-computing 13 4
TABLE I: Our selected tags and the number of questions and answers

Step 3: Selecting questions and answers. We extract the final sets of questions and answers using the final tag sets shown in Table I. We select all the posts that are tagged with at least one of the tags. We ended up with a total of 3,117 questions and answers from the four considered forums in our data set . 35% of the final data are answers where 65% are questions. The number of posts (questions and answers) extracted from each forum is shown in Table I.

Iii-C GitHub issues data collection

In this work, we study the issue reports of quantum computing projects on GitHub. We downloaded the GitHub selected quantum computing projects issues in March 2021. We follow three steps described below to extract the issue reports of quantum computing projects from GitHub.

Step 1: Searching candidate projects. We search for quantum computing related projects using three criteria: 1) The description of the project must be in English (i.e., for us to better understand the content). 2) The project name or description must contain the word “quantum” (the word quantum is case sensitive in the project name or description). 3) The project is in a mainline repository (i.e., not a fork of another repository). We end up with a total of 1,364 repositories.

Step 2: Filtering quantum computing projects. We filter the searching result and identify quantum computing related projects with three criteria: 1) To avoid selecting student assignments, following previous work [14] [13], we select repositories that were forked at least two times. 2) The projects must have a sufficient history of development for us to analyze the issue reports. Therefore, we select the projects that were created at least 10 months earlier than the data extraction date. Moreover, only the projects that have at least 100 commits and 10 issues are selected. 3) To ensure the quality of the project selected, we manually inspect the projects’ descriptions and remove projects that are not related to quantum computing, projects that are created for hosting quantum computing related documentation, as well as lecture notes related to quantum computing. Finally, we obtain a total of 122 projects directly related to quantum computing applications.

Step 3: Extracting issue reports. We use the GitHub Rest API [24] to extract all the issue reports of the final 122 projects on GitHub. In total, we obtain 43,979 issue reports.

Iii-D Data pre-processing for topic modeling

We build one topic model on the Stack Exchange forum data and another topic model on the GitHub issue data. Below we describe how we pre-process these two types of data before feeding them into topic models.

Pre-processing Q&A post data. We treat each post (i.e., a question or an answer) as an individual document in the topic model. For each question, we join the title and the body of the question to create a single document. As Q&A posts contain code snippets between <code> and </code> which may bring noise to our topic models, we remove all text between <code> and </code>. We also remove HTML tags (e.g., <p></p>), URLs and images from each post. In addition, we remove stop words (e.g., “like”, “this”, “the”), punctuation, and non-alphabetical characters using the Mallet and NLTK stop words set. Finally, we apply the Porter stemming [74] to normalize the words into their base forms (e.g., “computing” is transformed to “comput”), which can reduce the dimensionality of the word space and improve the performance of topic models [27]

Pre-processing issue report data. We treat each issue report as an individual document in the topic model. We join the title and the body of each issue as a single document. Similarly, we remove code snippets, URLs and images from the issue body. Since there are no tags in GitHub issues that identify code snippets, we look for backquote ” ” or triple backticks “‘ in the content of the issues and remove the code enclosed between this punctuation. We also remove stop words, non-alphabetical characters, and punctuation. Finally, we apply Porter stemming to normalize the words into their base forms.

Iii-E Topic modeling

We use automated topic modeling to analyze the topics in the Q&A posts and issue reports. Specifically, we use the Latent Dirichlet Allocation (LDA) algorithm [12]

to extract the topics from both of our datasets. LDA is a probabilistic topic modeling technique that derives the probability distribution of frequently co-occurred word sets (i.e., topics) in a text corpus. A topic is represented by a probability distribution of a set of words, while a document is represented as a probability distribution of a set of topics. LDA is widely used for modeling topics in software repositories 

[17], including technical Q&A posts (e.g, [8]) and issue reports (e.g., [29]). We use two separate topic models to extract the topics from the Q&A post data and the issue report data. For a better performance of the topic modeling and a good classification quality, following previous work [52] [59], we consider both uni-gram and bi-gram of words in our topic models.

LDA Implementation. We use the Python implementation of the Mallet topic modeling package [41] to perform our topic modeling. The Mallet package implements the Gibbs sampling LDA algorithm and uses efficient hyper-parameter optimization to improve the quality of the derived topics [41].

Determining topic modeling parameters. The number of topics () is usually manually set by the user as it controls the granularity of the topics [52]. The parameter controls the topic distribution in the documents (i.e., Q&A posts or issue reports), while the parameter controls the word distribution in the topics. In this work, we use the topic coherence score [57] to evaluate the quality of the resulting topics and determine the appropriate parameters (, , and ), similar to prior work [52, 68]. The coherence score measures the quality of a topic by measuring the semantic similarity between the top words in the topic. Thus, this score distinguishes between topics that are semantically interpretable and topics that are coincidences of statistical inference [57]. Specifically, we use the Gensim Python package’s CoherenceModel [23] module to calculate the coherence scores of the resulting topics. To capture a wide range of parameters and keep the topics distinct from each other, we experiment with different combination of the parameters, by varying the values of from 5 to 30 incremented by 1 each time, the values of document-topic distribution from 0.01 to 1 incremented by 0.01 [28], and the values of word-topic distribution from 0.01 to 1 incremented by 0.01 [28]. We retain the resulting topics with the highest average coherence score.

After getting the automatically derived topics, we manually analyze the resulting topics and assign meaningful labels to the topics. We elaborate more on this process in RQ2 and RQ3 for the Q&A post topics and the issue report topics, respectively.

Iv Experiment Results

In this section we report and discuss the results of our three research questions. For each research question, we first present the motivation and approach, then discuss the results for answering the research question.

RQ1: What types of QSE questions are asked on technical forums?

Iv-1 Motivation

In order to understand QSE challenges developers are facing, we first want to understand what types of questions they are asking (e.g., whether they are asking questions about using APIs or fixing errors). This is important to identify the areas in which QSE developers should be supported and the type of resources that they need. Similar to prior work [10], we focus on the intent behind the questions asked by QSE developers instead of the topics of the questions.

Iv-2 Approach

To identify the type of questions that users are asking in technical forums, we performed a manual analysis of a statistically representative sample from our studied QSE questions. We sampled 323 questions with a confidence level of 95% and a confidence interval of 5%. For each question, we examined its title and body, to understand the intent of the user who posted the question. We used a hybrid card sorting approach to perform the manual analysis and assign labels (i.e., types of questions) to each sampled question. Specifically, we based our manual analysis on an existing taxonomy of the types of questions asked on Stack Overflow 

[10] and added new types when needed. For each question we assigned one label; in case a question is associated with two or more labels, which we found only in a few cases, we chose the most relevant one.

Hybrid card sorting process. Two authors of the paper (i.e., coders) jointly performed the hybrid card sorting. We split the sampled data into two equal subsets and performed the sorting in two rounds, similar to prior work [36]. Our process guaranteed that each question is labelled by both coders.

  1. First-round labeling. Each coder labels a different half of the questions independently.

  2. First-round discussion. In order to have a consistent labeling strategy, we had a meeting to discuss the labeling results in the first round and reached an agreed-upon set of labels. A third author of the paper is involved in the discussion.

  3. Revising first-round labels. Each coder updated the first round labeling results based on the discussion.

  4. Second-round labeling. Each coder labeled the other half of the questions independently based on the agreed-upon labels in the first round. New labels are allowed in this round.

  5. Second-round discussion. We had a meeting to discuss the second-round labeling results, validate newly added labels and verify the consistency of our labels. A third author is also involved in the discussion.

  6. Revising second-round labels. Based on the second-round discussion, each coder revised the labels and finalized its individual labeling of the questions. We calculate the inter-coder agreement after this step.

  7. Resolving disagreement. We had a final meeting to resolve the disagreement in our labeling results and reached the final label for each question. For each difference in our labels, the two coders and a third author discussed the conflict and reached a consensus.

Inter-coder agreement. We measured the inter-coder agreement between the coders and obtained a Cohen’s kappa value of 0.73 which indicates a substantial agreement [42]. Therefore our manual labeling results are reliable.

Iv-3 Results

Category Description Freq
API usage
Questions of this category are usually identified by “how to”, i.e., how to use an API or how to implement a functionality.
85
Theoretical
This category of questioners ask about theoretical explanations of quantum programs, algorithms, and concepts.
54
Errors
This category of questions search for explanations and solutions of errors and exceptions when developing
or executing quantum programs.
49
Conceptual
Questions in this category are related to the limitation, background and the underlying concept of an API.
45
Discrepancy
Question of this category usually ask for explanations or solutions for unexpected results
(e.g., “what is the problem”, “why not work”.
31
Learning
Questions in this category are searching for learning resources such as documentation, research papers, tutorials, or websites.
22
Review
This category describes questions like: “How/Why this is working?” or “Is there a better solution?”.
Generally, the questions in this category look for a better solution to a problem or for help reviewing the current solution.
17
Tooling
This category describes questions like “I am looking for …”, “Is there a tool for …”.
These questions search for tools to solve a specific problem or check the features of a tool.
16
API change
This category of questions concern about changes of an API and the associated compatibility issues and other implications.
2
Categories newly identified in QSE-related questions.
TABLE II: A taxonomy of Question Categories which bases on and extends  [10]

Table II shows the result of our qualitative analysis for identifying the categories of questions in technical forums. Among the 323 questions we analyzed, we could not assign a label to only one question. In the table, we provide the description of each category and how frequent it appears in our qualitative analysis.

All seven categories of Stack Overflow questions identified in prior work appear in QSE-related posts. Prior work [10] identified seven categories of questions on Stack Overflow by studying Android-related questions, including API usage, Conceptual, Discrepancy, Errors, Review, API change, and Learning, ordered by their occurrence frequency. Although quantum computing is still a new area, people start to ask all these different categories of questions, indicating that quantum computing face similar software engineering challenges (e.g., API usage and API change) as other software engineering domains. Similar to prior work, we find that API usage is the most frequent category with 26.3% instances. The questions of this category are usually identified by “how to”; e.g., “How to return measurement probabilities from the QDK Full-state simulator?

The categories of Errors and Learning are relatively more frequent in QSE-related questions than in the prior taxonomy of question categories [10]. Compare to prior work [10]

on classifying Android-related questions, we find that

Errors and Learning questions are relatively more frequent. As quantum computing is still an emerging domain, people practicing it face many errors when developing quantum computing applications and they find it challenging to find learning resource for quantum computing. An example of the Errors category is “I have Qiskit installed via Anaconda and a virtual environment set up in Python 3.8. … I get an error. I’m not sure what the problem is. How do I fix it?”. Another example for the Learning category is “How do I learn Q#? What languages should I know prior to learning Q#? How do I get started with quantum computing?”. These findings suggest the need to develop tools or resources to help developers avoid or address such errors, as well as developing tutorials, books, and other learning resources to help beginners get acquainted with quantum computing.

Two new categories of questions (i.e., Theoretical and Tooling) emerge in QSE-related posts. In fact, the category of Theoretical is the second most frequent among all categories. This category is usually associated with keywords such as “can someone explain”, “what is”, and “does quantum”. An example question of this category is “What is the analysis of the Bell Inequality protocol in Cirq’s ‘examples’?” where Cirq[18] is a Python library for developing quantum computing applications. This category of questions indicates that people have challenges understanding the theoretical concepts behind quantum computing code. Future efforts are needed to explain such theoretical concepts for developers. The category of tooling represents questions that are looking for tools, frameworks, or libraries that can help solve a QSE-related problem or verifying whether a tool, framework, or library can help solve a problem. For example, “I want to use Blender and Blender Python Scripts working with Qiskit. How can I do this? How to make communication between Blender and Qiskit installed with Anaconda Python?”. This category indicates the lack of established tools for supporting quantum program development.

We identified nine categories of QSE-related questions in Stack Exchange forums. The categories Theoretical, Errors, Learning, and Tooling are new or become more frequent in QSE-related questions. Our results highlight the need for future efforts to support developers’ quantum program development, in particular, to develop learning resources, to help developers fix errors, and to explain theory behind quantum computing code.

RQ2: What QSE topics are raised in technical forums?

Iv-1 Motivation

Developers post QSE-related questions and answers on technical forums. Their posts may reflect their faced challenges when learning or developing quantum programs. To understand their faced challenges, we use topic models to extract the semantic topics in their posts and analyze the characteristics of these topics.

Iv-2 Approach

Topic assignment and frequency. The automated topic modeling generated nine topics and distribution of co-occurring words in each topic. We then manually assigned a meaningful label to each topic. Following prior work[78, 6, 52], to assign a meaningful label to a topic, the first author first proposed labels using two pieces of information: (1) the topic’s top 20 keywords, and (2) the top 10-15 most relevant questions associated with the topic. Then, three authors of the paper reviewed the labels in meetings and reassigned the labels when needed. We obtained a meaningful label for each of the nine topics at the end. For each topic, we measure the percentage of the posts (i.e., frequency) that have it as the dominant topic (i.e., with the highest probability).

Topic popularity. To understand developers’ attention towards each topic, following previous work [78, 6, 52], we measured three metrics for each topic: (1) the median number of views of the associated posts, (2) the median number of associated posts marked as favorite, and (3) the median score of the associated posts. For each topic, the associated posts refer to the posts that have it as the dominant topic.

Topic difficulty. In order to better understand the most challenging aspects for developers, we measure the difficulty of each topic in terms of how difficult it is for the associated posts to get accepted answers. Following prior work [78, 6, 52], for each topic, we measure two metrics: (1) the percentage of the associated questions with no accepted answer, and (2) the median time required by the associated questions to get an accepted answer (only considering the ones with an accepted answer). For each topic, the associated questions refer to the questions that have the topic as the dominant topic.

Iv-3 Results

Topic (manual label) Keywords Description % Freq
Environment management
quot, error, python, build, code
Development environment and build problems
15.03
Dependency management
qiskit, import, ibmq, operator, provider
Library installation, use, and versioning issues
14.82
Algorithm complexity
time, problem, algorithm, number, function
Quantum algorithm complexity and optimization
14.06
Quantum execution results
circuit, result, back-end, simulator, measure
Quantum program execution results on quantum
backends (e.g., simulators)
13.22
Learning resources
question, paper, work, understand, answer
Searching for learning resources such as research
papers and tutorials
9.05
Data structures and operations
matrix, return, array, datum, list
Data structures (e.g., matrix, arrays and list) and
their operations in quantum programs
8.81
Quantum circuits
qubit, gate, control, operation, cirq
Elements of quantum circuits (e.g., Qubits, gates)
and their operations
8.66
Quantum vs. classical computing
quantum, computer, classical, computing,
algorithm
Comparisons between quantum and classical computing
or migrating classic algorithms to quantum computing
8.30
Quantum algorithms understanding state, rangle, frac, theta, sqrt
Quantum algorithm explanation and interpretation
7.51
TABLE III: Topics extracted from QSE related posts on Stack Exchange forums

We derived nine topics that are discussed in QSE-related posts, including traditional software engineering topics (e.g., environment management and dependency management) and QSE-specific topics (e.g., quantum execution results and Quantum circuits). Table III describes the nine topics and their frequency in the analyzed posts. Table IV shows the median views, scores, and favorites of the posts associated with these topics. The three most dominant topics are environment management, dependency management, and algorithm complexity.

Environment management is the most dominant topic representing 15.03% of posts. For example, the most viewed question of this topic is “I downloaded the Quipper package but I have not been able to get haskell to recognize where all of the modules and files are and how to properly link everything” which gained 2772 views. Other examples include “How can I run QCL (quantum programming language) on Windows?” and “Visualization of Quantum Circuits when using IBM QISKit”. We can observe that users are new to quantum computing and facing problem while setting up their environment and installing their tools. This topic is also linked to the question category tooling that we derived in RQ1.

Dependency management is the second most discussed topic representing 14.82% of the posts. For example, the most viewed question (with 2,239 views) of this topic is “When trying the above code, I am receiving the following error: ModuleNotFoundError: No module named qiskit” where qiskit

is an open source framework for quantum program development

[1]. We noticed that a large number of questions are directly related to qiskit. This can be explained by the lack of documentation or tutorials in using this framework.

Algorithm complexity is the third most dominant topic. This topic is about understanding the complexity of quantum algorithms and how to optimize quantum algorithms. For example, the most viewed question of this topic is “For the other algorithms, I was able to find specific equations to calculate the number of instructions of the algorithm for a given input size (from which I could calculate the time required to calculate on a machine with a given speed). However, for Shor’s algorithm, the most I can find is its complexity: O( (log N) )”, which receives 4,718 views. This topic is linked to the questions category theoretical derived from RQ1. This topic indicates developers’ challenge in understanding the complexity of quantum algorithms.

As quantum programming is oriented to searching solutions in a probabilistic space, which is counter-intuitive from the classical computing perspective, understanding quantum execution results is particularly challenging for developers. As a Qubit can be 0 or 1 with a certain probability, a quantum program that has Qubits as its basic units can have many different states at the same time. The results of a quantum program are certain only when the results are observed (or “measured”). Therefore, it is more challenging for developers to understand the results of quantum programs than that of classical programs. For example “How to plot histogram or Bloch sphere for multiple circuits? I have tried to plot a histogram for the multiple circuits for the code given below. I think I have done it correctly but don’t know why it’s not working. Please help me out. If possible please solve it for the Bloch sphere” Future efforts are needed to interpret quantum program outputs.

Topic name
Quantum vs. classical computing 147.5 3 1.5
Quantum circuits 107.0 2 1.0
Environment management 106.0 1 1.0
Learning resources 102.0 2 1.0
Quantum execution results 98.0 1 1.0
Quantum algorithms understanding 97.5 2 1.0
Dependency management 93.0 2 1.0
Algorithm compolexity 87.5 1 1.0
Data structures and operations 82.0 1 1.0
TABLE IV: Popularity of QSE-related topics on Stack Exchange forums

Posts related to quantum vs. classical computing are gaining the most attention from developers. Since quantum computing is based on a new philosophy different from classical computing, developers often ask questions about the differences and look to understand the new doors quantum computing is opening. According to Table IV, posts with this topic receive the highest median number of views. This may show that software engineers are eager to contribute to QSE by starting from the differences between the two paradigms. However, as shown in Figure 3, posts on this topic are least likely to receive accepted answers. Our results indicate the need for resources and tools for bridging the knowledge gap between quantum computing and classical computing.

Fig. 3: The difficulty aspect of QSE-related topics on Stack Exchange forums

Questions of some topics (e.g., environment management) are much more difficult than others to receive accepted answers. According to Figure 3, the topic environment management is the most difficult topic to answer, with 61% of posts not receiving an accepted answer and a median time of 12 hours to receive an accepted answer. Learning resources, quantum vs. classical computing and data structures and operations are also among the most difficult topics in terms of the ratio of posts getting accepted answers and the time to get one. The results indicate the lack of community support in aspects such as setting up a development environment, searching for learning resources, and understanding differences between quantum computing and classical computing, which could impair the advancement of quantum software development practices.

From Q&A forums, we derived nine topics discussed in QSE-related posts, including traditional software engineering topics (e.g., environment management) and QSE-specific ones (e.g., quantum execution results). We highlighted some particularly challenging areas for QSE, such as interpreting quantum program outputs and bridging the knowledge gap between quantum computing and classical computing.

RQ3: What QSE topics are raised in the issue reports of quantum-computing projects?

Manual label Keywords Description % Freq # Projects
Learning resources
summary, remove, tutorial, link, documentation
Search for documentation, tutorials, websites, etc.
14.94 109 (90.08%)
Environment management
build, include, library, release, variable
Development environment and build problems
13.72 107 (88.43%)
API change
version, qiskit, code, issue, update
API update or deprecation issues
11.65 98 (80.99%)
Quantum circuits
gate, circuit, qubit, operation, control
Elements of quantum circuits (e.g., Qubits, gates)
and their operations
8.30 70 (57.85%)
Quantum chemistry
input, calculation, basis, energy, pyscf
Issues with quantum chemistry libraries (e.g., PySCF)
6.72 76 (62.80%)
Quantum execution errors
error, artiq, follow, experiment, device
Errors in the execution of quantum programs
6.38 80 (66.12%)
Unit testing
test, check, fail, unit, script
Unit testing failures
6.32 87 (71.90%)
API usage
function, method, class, parameter, call
How to use an API
5.81 80 (66.11%)
Quantum execution results
state, number, result, time, measurement
Quantum program execution results (i.e., measured state)
5.76 89 (73.55%)
Data structures and operations
implement, operator, matrix, problem, array
Data structures (e.g., matrix, arrays and list)
and their operations
5.73 88 (72.73%)
Machine learning
model, datum, dataset, layer, benchmark
Quantum computing application in machine learning
5.26 75 (61.9%)
Dependency management
file, python, import, package, install
Library installation, use and versioning issues
5.23 93 (76.86%)
Algorithm optimization
case, time, optimization, long, performance
Program performance and algorithm optimization
4.19 83 (68.6%)
TABLE V: QSE-related topics derived from issue reports of quantum computing projects on GitHub

Iv-1 Motivation

Issue reports of quantum computing projects record developers’ concerns and discussions when adding features or resolving issues in these projects. The textual information in the issue reports may communicate developers’ challenges when developing quantum computing applications. Therefore, we analyze the topics in the issue reports to understand the challenges developers are facing as well as the prevalence of these challenges. While the questions on technical forums can provide information about developers’ general challenges, the issue reports may communicate developers’ challenges for specific problems (i.e., issues).

Iv-2 Approach

Topic assignment and frequency. Our topic model on the issue report data generated 17 topics. We follow the same process as described in RQ2 to manually assign meaningful labels to the automated topics, based on the top words in the topics and the content of the associated issue reports. During the manual assignment process, we found that some topics are similar to each other even though such similarity is not detected by the probabilistic topic model. Therefore, we follow prior work [50, 58] and merged similar topics. We also discarded one topic as we could not derive a meaningful label from the top words and the associated issue reports. In the end, we obtained 13 meaningful topics. For each topic, we measure the percentage of the issue reports (i.e., frequency) that have it as the dominant topic (i.e., with the highest probability). We also measure the number and percentage of the studied projects that have at least one issue report of each topic.

Topic difficulty. To further understand developers’ challenges in developing quantum computing applications, we measure the “difficulty” of the issue reports associated with each topic. As we cannot directly measure the “difficulty” of issue reports, we measure three indirect metrics for each topic: (1) the percentage of issue reports associated with the topic that is closed, (2) the median time required to close an issue (since its creation) associated with the topic, and (3) the median number of comments in an associated topic (intuitively, an issue report with more comments may be more difficult [32]). For each topic, the associated issue reports refer to the issue reports that have it as the dominant topic.

Iv-3 Results

We derived 13 topics from GitHub issue reports, bringing new perspectives to the challenges faced by QSE developers. Table V shows the list of our derived topics, their descriptions, their percentage frequency in the studied issue reports, and the number of projects that have at least one issue report of the topic. Among the 13 topics, 6 of them (learning resources, environment management, quantum circuits, quantum execution results, data structures and operations, and dependency management) are overlapping with the topics derived from Stack Exchange posts (RQ2), and another 2 of them (API change and API usage) are overlapping with the categories of Stack Exchange questions derived in RQ1. This result indicates that the QSE-related challenges that we derived from forum posts indeed impact practical quantum program development in GitHub projects.

Among the other five topics, two of them (i.e., machine learning and quantum chemistry) are related to the most popular and promising quantum computing application areas: machine learning and chemistry. For example, an issue report associated with quantum chemistry raises an issue when using a molecular optimizer Python library: “it leads to PyBerny optimizing an unconverged ground state energy, which generally leads to the geometry optimization never converging”. The other three topics (i.e., quantum execution errors, unit testing, algorithm optimization) are related to applications of traditional software engineering processes in quantum program development.

All derived topics are general among the quantum computing projects, as each topic is present in the issue reports of 58% to 90% of the projects. We observe that learning resources and environment management are the two most frequent topics and appear in 90% and 88% of all the studied projects, respectively, which once again highlights the need of efforts for developing learning resources and supporting developers in setting up their quantum program development environment.

Some topics are particularly challenging for developers, such as data structures and operations, quantum circuits, and quantum execution results. Table VI shows the median time it takes to close an issue report and the number of comments in an issue report associated with each topic. All the issues are closed at the time when we analyzed their status. The issues associated with each topic only have a median of one to two comments, indicating that developers’ interactions on these issue reports are not intense. Data structures and operations is also among the most difficult topics on forum posts (as discussed in RQ2). However, the Quantum circuits and quantum execution results topics are not among the most difficult topics on forum posts, while they are two of the most difficult ones on GitHub issues, which indicates that quantum circuit issues and the interpretation of quantum program execution results are more difficult in specific problem contexts.

Topic name
Data structures and operations 151.40 1
Quantum circuits 114.98 1
Quantum execution results 98.02 1
Machine learning 94.68 2
API usage 80.70 1
Quantum chemistry 62.89 2
Quantum execution errors 59.44 2
API change 47.34 1
Algorithm optimization 39.83 1
Dependency management 32.40 2
Unit testing 28.26 1
Learning resources 27.02 1
Environment management 21.57 1
All the issues are closed at the time we analyzed their status.
TABLE VI: The difficulty aspect of QSE-related topics on GitHub issues

QSE-related challenges that we derived from forum posts indeed impact practical quantum program development in GitHub projects, while GitHub issues bring new perspectives on developers’ faced challenges (e.g., on specific quantum computing applications such as machine learning). In particular, we observe that the challenges are generally among the quantum computing projects.

V Threats to Validity

External validity. In this work, we analyze four Stack Exchange forums and 122 GitHub repositories to understand the challenges of QSE. Our studied forum posts and GitHub issues may not cover all the ones that are related to QSE. Developers may also communicate their discussions in other media (e.g., mailing lists). Future work considering other data sources may complement our study. In addition, we identify and collect the posts from Q&A forums using a selected set of tags. Our analysis may miss some QSE tags. However, to alleviate this threat, we follow prior work [70, 52] and use an iterative method to identify the relevant tags.

Internal validity. In this work, we use topic models to cluster the forum posts and GitHub issue reports, based on the intuition that the same clusters would have similar textual information. However, different clusters of posts and issue reports may exist when a different approach is used. To ensure the quality of the clusters, we manually reviewed the resulting topics, merged similar topics when needed, and assigned meaningful labels to them.

Construct validity. In RQ1, we manually analyze the categories of QSE-related questions on technical Q&A forums. Our results may be subjective and depend on the judgment of the researchers who conducted the manual analysis. To mitigate the bias, two authors of the paper collectively conducted an manual analysis and reached a substantial agreement, indicating the reliability of the analysis results. A third author also participated in the discussions during the manual analysis, to ensure the quality of the results. In RQ2 and RQ3, the parameters of the topic models (e.g., the number of topics ) may impact our findings. To mitigate this threat, following previous work  [70] [52], we did multiple experiments and use the topic coherence score to select the most suitable parameters. The manual labeling of topics can be subjective. To reduce this threat, the authors read each topic’s top 20 keywords and the top 15 highest contributed posts to the topic. We followed a clear-cut approach adapted in previous works [70] [52]. In addition, in our analysis of the QSE posts, we did not filter posts using the number of comments, votes or answers (as done in prior work [21]), which may lead to noise in the analyzed posts (e.g., low-quality posts). We made this decision since QSE is a new topic and the number of posts in the Q&A forums is relatively small.

Vi Conclusions

This paper examines challenges quantum program developers are facing by analyzing Stack Exchange forums posts related to QSE and the GitHub issue reports of quantum computing projects. Results indicate that QSE developers face traditional software engineering challenges (e.g., dependency management) as well as QSE-specific challenges (e.g., interpreting quantum program execution results). In particular, some QSE-related areas (e.g., bridging the knowledge gap between quantum and classical computing) are gaining the highest attention from developers while being very challenging to them. As the initial effort for understanding QSE-related challenges perceived by developers, our work shed light on future opportunities in QSE (e.g., supporting explanations of theory behind quantum program code and the interpretations of quantum program execution results).

References

  • [1] H. Abraham, AduOffei, R. Agarwal, I. Y. Akhalwaya, and et al. (2019) Qiskit: an open-source framework for quantum computing. External Links: Document Cited by: §I, §II-B, §IV-3.
  • [2] A. Aggarwal, G. Waghmare, and A. Sureka (2014) Mining issue tracking systems using topic models for trend analysis, corpus exploration, and understanding evolution. In Proceedings of the 3rd International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering, pp. 52–58. Cited by: §II-E.
  • [3] S. Ahmed and M. Bagherzadeh (2018) What do concurrency developers ask about? a large-scale study using stack overflow. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 1–10. Cited by: §II-D.
  • [4] M. Allamanis and C. Sutton (2013) Why, when, and what: analyzing stack overflow questions by topic, type, and code. In 2013 10th Working Conference on Mining Software Repositories (MSR), pp. 53–56. Cited by: §II-D.
  • [5] M. Alshangiti, H. Sapkota, P. K. Murukannaiah, X. Liu, and Q. Yu (2019) Why is developing machine learning applications challenging? a study on stack overflow posts. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 1–11. Cited by: §II-D, §III-B.
  • [6] M. Bagherzadeh and R. Khatchadourian (2019) Going big: a large-scale study on what big data developers ask. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, New York, NY, USA, pp. 432–442. External Links: ISBN 9781450355728, Link, Document Cited by: §IV-2, §IV-2, §IV-2.
  • [7] L. S. Barbosa (2020) Software engineering for’quantum advantage’. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, pp. 427–429. Cited by: §I, §II-C, §II-D.
  • [8] A. Barua, S. W. Thomas, and A. E. Hassan (2014) What are developers talking about? an analysis of topics and trends in stack overflow. Empirical Software Engineering 19 (3), pp. 619–654. Cited by: §III-E.
  • [9] S. Beyer, C. Macho, M. D. Penta, and M. Pinzger (2021) What kind of questions do developers ask on stack overflow? a comparison of automated approaches to classify posts into question categories. In Software Engineering, Cited by: item RQ1:.
  • [10] S. Beyer, C. Macho, M. Di Penta, and M. Pinzger (2020) What kind of questions do developers ask on stack overflow? a comparison of automated approaches to classify posts into question categories. Empirical Software Engineering 25 (3), pp. 2258–2301. Cited by: §IV-1, §IV-2, §IV-3, §IV-3, TABLE II.
  • [11] J. Biamonte, P. Wittek, N. Pancotti, P. Rebentrost, N. Wiebe, and S. Lloyd (2017) Quantum machine learning. Nature 549 (7671), pp. 195–202. Cited by: §I.
  • [12] D. M. Blei, A. Y. Ng, and M. I. Jordan (2003) Latent dirichlet allocation. the Journal of machine Learning research 3, pp. 993–1022. Cited by: §III-E.
  • [13] J. Businge, M. Openja, D. Kavaler, E. Bainomugisha, F. Khomh, and V. Filkov (2019) Studying android app popularity by cross-linking github and google play store. 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 287–297. Cited by: §III-C.
  • [14] J. Businge, M. Openja, S. Nadi, E. Bainomugisha, and T. Berger (2018) Clone-based variability management in the android ecosystem. 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 625–634. Cited by: §III-C.
  • [15] B. Cartaxo, G. Pinto, D. Ribeiro, F. K. Kamei, R. Santos, F. Silva, and S. Soares (2017-05) Using q&a websites as a method for assessing systematic reviews. pp. . External Links: Document Cited by: §III-B.
  • [16] H. Chen, J. Coogle, and K. Damevski (2019) Modeling stack overflow tags and topics as a hierarchy of concepts. Journal of Systems and Software 156, pp. 283–299. Cited by: §II-D.
  • [17] T. Chen, S. W. Thomas, and A. E. Hassan (2016) A survey on the use of topic models when mining software repositories. Empirical Software Engineering 21 (5), pp. 1843–1919. Cited by: §III-E.
  • [18] Cirq Note: See full list of authors on Github: https://github .com/quantumlib/Cirq/graphs/contributors External Links: Document, Link Cited by: §IV-3.
  • [19] D. Deutsch and R. Penrose (1985) Quantum theory, the church&#x2013;turing principle and the universal quantum computer. Proceedings of the Royal Society of London. A. Mathematical and Physical Sciences 400 (1818), pp. 97–117. External Links: Document, Link, https://royalsocietypublishing.org/doi/pdf/10.1098/rspa.1985.0070 Cited by: §II-B.
  • [20] P. A. M. Dirac (1981) The principles of quantum mechanics. Oxford university press. Cited by: §I.
  • [21] E. Farhana, N. Imtiaz, and A. Rahman (2019) Synthesizing program execution time discrepancies in julia used for scientific software. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 496–500. Cited by: §V.
  • [22] S. Garhwal, M. Ghorani, and A. Ahmad (2019) Quantum programming language: a systematic review of research topic and top cited languages. Archives of Computational Methods in Engineering, pp. 1–22. Cited by: §II-B.
  • [23] Gensim coherencemodel implimentation. Note: https://radimrehurek.com/gensim/models/coherencemodel.htmlLast accessed 05/04/2021 Cited by: §III-E.
  • [24] () GitHub REST API. Note: https://developer.github.com/v3/Last accessed 05/04/2021 Cited by: §III-C.
  • [25] Google Google QuantumAI. Note: https://quantumai.googleLast accessed 05/04/2021 Cited by: §I.
  • [26] G. G. Guerreschi and M. Smelyanskiy (2017) Practical optimization for hybrid quantum-classical algorithms. arXiv preprint arXiv:1701.01450. Cited by: §I.
  • [27] V. Gurusamy and S. Kannan (2017-08) Performance analysis: stemming algorithm for the english language. International Journal for Scientific Research and Development 5, pp. 2321–613. Cited by: §III-D.
  • [28] J. Han, E. Shihab, Z. Wan, S. Deng, and X. Xia (2020)

    What do programmers discuss about deep learning frameworks

    .
    Empirical Software Engineering 25, pp. 2694–2747. Cited by: §III-E.
  • [29] A. Hindle, A. Alipour, and E. Stroulia (2016) A contextual approach towards more accurate duplicate bug report detection and ranking. Empirical Software Engineering 21 (2), pp. 368–410. Cited by: §II-E, §III-E.
  • [30] IBM IBM Quantum Computing. Note: https://www.ibm.com/quantum-computing/Last accessed 05/04/2021 Cited by: §I, §I.
  • [31] P. Kaye, R. Laflamme, M. Mosca, et al. (2007) An introduction to quantum computing. Oxford University Press on Demand. Cited by: §II-A.
  • [32] F. Khomh, B. Chan, Y. Zou, and A. E. Hassan (2011) An entropy evaluation approach for triaging field crashes: a case study of mozilla firefox. In 2011 18th Working Conference on Reverse Engineering, Vol. , pp. 261–270. External Links: Document Cited by: §IV-2.
  • [33] W. Knight (2018) Serious quantum computers are finally here. what are we going to do with them. MIT Technology Review 30. Cited by: §I.
  • [34] E. Knill (1996-06) Conventions for quantum pseudocode. . External Links: Document, Link Cited by: §II-B.
  • [35] R. LaRose (2019) Overview and comparison of gate level quantum software platforms. Quantum 3, pp. 130. Cited by: §II-B.
  • [36] H. Li, W. Shang, B. Adams, M. Sayagh, and A. E. Hassan (2020) A qualitative study of the benefits and costs of logging from developers’ perspectives. IEEE Transactions on Software Engineering. Cited by: §IV-2.
  • [37] M. Linares-Vásquez, B. Dit, and D. Poshyvanyk (2013) An exploratory analysis of mobile development issues using stack overflow. In 2013 10th Working Conference on Mining Software Repositories (MSR), pp. 93–96. Cited by: §II-D, §III-B.
  • [38] L. O. Mailloux, C. D. Lewis II, C. Riggs, and M. R. Grimaila (2016) Post-quantum cryptography: what advancements in quantum computing mean for it professionals. IT Professional 18 (5), pp. 42–47. Cited by: §I.
  • [39] L. Martie, V. K. Palepu, H. Sajnani, and C. Lopes (2012) Trendy bugs: topic trends in the android bug reports. In 2012 9th IEEE Working Conference on Mining Software Repositories (MSR), pp. 120–123. Cited by: §II-E.
  • [40] D. Maslov, Y. Nam, and J. Kim (2018) An outlook for quantum computing [point of view]. Proceedings of the IEEE 107 (1), pp. 5–10. Cited by: §I.
  • [41] A. K. McCallum (2002) MALLET: a machine learning for language toolkit. Note: http://mallet.cs.umass.edu Cited by: §III-E.
  • [42] M. McHugh (2012-10) Interrater reliability: the kappa statistic. Biochemia medica : časopis Hrvatskoga društva medicinskih biokemičara / HDMB 22, pp. 276–82. External Links: Document Cited by: §IV-2.
  • [43] Microsoft Microsoft Azure Quantum Service. Note: https://azure.microsoft.com/en-ca/services/quantum/Last accessed 05/04/2021 Cited by: §I.
  • [44] H. MLNAŘÍK (2008) SEMANTICS of quantum programming language lanq. International Journal of Quantum Information 06 (supp01), pp. 733–738. External Links: Document, Link, https://doi.org/10.1142/S0219749908004031 Cited by: §II-B.
  • [45] E. Moguel, J. Berrocal, J. García-Alonso, and J. M. Murillo (2020) A roadmap for quantum software engineering: applying the lessons learned from the classics. In Proceedings of the 1st International Workshop on Software Engineering ++& Technology, Q-SET ’20. Cited by: §I, §II-C.
  • [46] L. Mueck (2017) Quantum software. Nature 549 (171). Cited by: §I.
  • [47] H. Naguib, N. Narayan, B. Brügge, and D. Helal (2013) Bug report assignee recommendation using activity profiles. In 2013 10th Working Conference on Mining Software Repositories (MSR), pp. 22–30. Cited by: §II-E.
  • [48] A. T. Nguyen, T. T. Nguyen, J. Al-Kofahi, H. V. Nguyen, and T. N. Nguyen (2011) A topic-based approach for narrowing the search space of buggy files from a bug report. In 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011), pp. 263–272. Cited by: §II-E.
  • [49] A. T. Nguyen, T. T. Nguyen, T. N. Nguyen, D. Lo, and C. Sun (2012) Duplicate bug report detection with a combination of information retrieval and topic modeling. In 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, pp. 70–79. Cited by: §II-E.
  • [50] E. Noei, F. Zhang, and Y. Zou (2019) Too many user-reviews, what should app developers look at first?. IEEE Transactions on Software Engineering. Cited by: §IV-2.
  • [51] B. Ömer (2003) QCL-a programming language for quantum computers. Software available on-line at http://tph. tuwien. ac. at/~ oemer/qcl. html. Cited by: §I.
  • [52] M. Openja, B. Adams, and F. Khomh (2020) Analysis of modern release engineering topics : – a large-scale study using stackoverflow –. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), Vol. , pp. 104–114. External Links: Document Cited by: §III-B, §III-E, §III-E, §IV-2, §IV-2, §IV-2, §V, §V.
  • [53] M. Piattini, G. Peterssen, R. Pérez-Castillo, J. L. Hevia, M. A. Serrano, G. Hernández, I. G. R. de Guzmán, C. A. Paradela, M. Polo, E. Murina, et al. (2020) The talavera manifesto for quantum software engineering and programming.. In The First International Workshop on the Quantum Software Engineering & Programming, QANSWER ’20, pp. 1–5. Cited by: §I, §I, §II-B, §II-C.
  • [54] M. Piattini, G. Peterssen, and R. Pérez-Castillo (2020) Quantum computing: a new software engineering golden age. ACM SIGSOFT Software Engineering Notes 45 (3), pp. 12–14. Cited by: §I, §II-C.
  • [55] M. Piattini, M. Serrano, R. Perez-Castillo, G. Petersen, and J. L. Hevia (2021) Toward a quantum software engineering. IT Professional 23 (1), pp. 62–66. Cited by: §I, §II-C.
  • [56] M. Reiher, N. Wiebe, K. M. Svore, D. Wecker, and M. Troyer (2017) Elucidating reaction mechanisms on quantum computers. Proceedings of the National Academy of Sciences 114 (29), pp. 7555–7560. Cited by: §I.
  • [57] M. Röder, A. Both, and A. Hinneburg (2015) Exploring the space of topic coherence measures. Proceedings of the Eighth ACM International Conference on Web Search and Data Mining. Cited by: §III-E.
  • [58] C. Rosen and E. Shihab (2016) What are mobile developers asking about? a large scale study using stack overflow. Empirical Software Engineering 21 (3), pp. 1192–1223. Cited by: §II-D, §III-B, §IV-2.
  • [59] R. Rossi and S. Rezende (2011-01) Generating features from textual documents through association rules. pp. . Cited by: §III-E.
  • [60] J. W. Sanders and P. Zuliani (2000) Quantum programming. In Mathematics of Program Construction, Lecture Notes in Computer Science, Vol. 1837, pp. 80–99. Cited by: §II-B.
  • [61] E. Schrödinger (1935) Discussion of probability relations between separated systems. In Mathematical Proceedings of the Cambridge Philosophical Society, Vol. 31, pp. 555–563. Cited by: §I.
  • [62] B. Sodhi (2018) Quality attributes on quantum computing platforms. CoRR abs/1803.07407. External Links: Link, 1803.07407 Cited by: Fig. 1, §II-A.
  • [63] Stack Exchange Artificial Intelligence forum. Note: https://ai.stackexchange.comLast accessed 05/04/2021 Cited by: §III-B.
  • [64] Stack Exchange Computer Science forum. Note: https://cs.stackexchange.comLast accessed 05/04/2021 Cited by: §III-B.
  • [65] stack exchange data explorer. Cited by: §III-B.
  • [66] Stack Exchange Quantum Computing forum. Note: https://quantumcomputing.stackexchange.comLast accessed 05/04/2021 Cited by: §III-B.
  • [67] Stack Overflow forum. Note: https://stackoverflow.comLast accessed 05/04/2021 Cited by: §III-B.
  • [68] K. Svore, A. Geller, M. Troyer, J. Azariah, C. Granade, B. Heim, V. Kliuchnikov, M. Mykhailova, A. Paz, and M. Roetteler (2018) Q#: enabling scalable quantum computing and development with a high-level dsl. In Proceedings of the Real World Domain Specific Languages Workshop 2018, RWDSL2018, New York, NY, USA. External Links: ISBN 9781450363556, Link, Document Cited by: §II-B, §III-E.
  • [69] C. Treude and M. Wagner (2019) Predicting good configurations for github and stack overflow topic models. In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), pp. 84–95. Cited by: §II-D.
  • [70] G. Uddin, O. Baysal, L. Guerrouj, and F. Khomh (2021) Understanding how and why developers seek and analyze api-related opinions. External Links: 2102.08495 Cited by: §III-B, §V, §V.
  • [71] B. Vasilescu Academic papers using stack exchange data. Note: https://meta.stackexchange.com/questions/134495/academic-papers-using-stack-exchange-dataLast accessed 05/04/2021 Cited by: §II-D.
  • [72] P. K. Venkatesh, S. Wang, F. Zhang, Y. Zou, and A. E. Hassan (2016) What do client developers concern when using web apis? an empirical study on developer forums and stack overflow. In 2016 IEEE International Conference on Web Services (ICWS), pp. 131–138. Cited by: §II-D.
  • [73] S. Wang, D. Lo, and L. Jiang (2013) An empirical study on developer interactions in stackoverflow. In Proceedings of the 28th Annual ACM Symposium on Applied Computing, pp. 1019–1024. Cited by: §II-D.
  • [74] P. Willett (2006-07) The porter stemming algorithm: then and now. Program electronic library and information systems 40, pp. . External Links: Document Cited by: §III-D.
  • [75] X. Xia, D. Lo, Y. Ding, J. M. Al-Kofahi, T. N. Nguyen, and X. Wang (2016) Improving automated bug triaging with specialized topic model. IEEE Transactions on Software Engineering 43 (3), pp. 272–297. Cited by: §II-E.
  • [76] X. Xia, D. Lo, X. Wang, and B. Zhou (2013) Accurate developer recommendation for bug resolution. In 2013 20th Working Conference on Reverse Engineering (WCRE), pp. 72–81. Cited by: §II-E.
  • [77] X. Yang, D. Lo, X. Xia, Z. Wan, and J. Sun (2016) What security questions do developers ask? a large-scale study of stack overflow posts. Journal of Computer Science and Technology 31 (5), pp. 910–924. Cited by: §II-D.
  • [78] X. Yang, D. Lo, X. Xia, Z. Wan, and J. Sun (2016-09) What security questions do developers ask? a large-scale study of stack overflow posts. Journal of Computer Science and Technology 31, pp. 910–924. External Links: Document Cited by: §IV-2, §IV-2, §IV-2.
  • [79] T. Zhang, G. Yang, B. Lee, and E. K. Lua (2014) A novel developer ranking algorithm for automatic bug triage using topic model and developer relations. In 2014 21st Asia-Pacific Software Engineering Conference, Vol. 1, pp. 223–230. Cited by: §II-E.
  • [80] Y. Zhang, D. Lo, X. Xia, and J. Sun (2015) Multi-factor duplicate question detection in stack overflow. Journal of Computer Science and Technology 30 (5), pp. 981–997. Cited by: §II-D.
  • [81] J. Zhao (2020) Quantum software engineering: landscapes and horizons. arXiv preprint arXiv:2007.07047. Cited by: §I, §I, Fig. 1, §II-A, §II-B, §II-C.
  • [82] J. Zou, L. Xu, W. Guo, M. Yan, D. Yang, and X. Zhang (2015) Which non-functional requirements do developers focus on? an empirical study on stack overflow using topic analysis. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories, pp. 446–449. Cited by: §II-D.
  • [83] J. Zou, L. Xu, M. Yang, M. Yan, D. Yang, and X. Zhang (2016) Duplication detection for software bug reports based on topic model. In 2016 9th International Conference on Service Science (ICSS), pp. 60–65. Cited by: §II-E.
  • [84] J. Zou, L. Xu, M. Yang, X. Zhang, and D. Yang (2017) Towards comprehending the non-functional requirements through developers’ eyes: an exploration of stack overflow using topic analysis. Information and Software Technology 84, pp. 19–32. Cited by: §II-D.