Towards Just-Enough Documentation for Agile Effort Estimation: What Information Should Be Documented?

07/06/2021
by   Jirat Pasuksmit, et al.
The University of Melbourne
0

Effort estimation is an integral part of activities planning in Agile iterative development. An Agile team estimates the effort of a task based on the available information which is usually conveyed through documentation. However, as documentation has a lower priority in Agile, little is known about how documentation effort can be optimized while achieving accurate estimation. Hence, to help practitioners achieve just-enough documentation for effort estimation, we investigated the different types of documented information that practitioners considered useful for effort estimation. We conducted a survey study with 121 Agile practitioners across 25 countries. Our survey results showed that (1) despite the lower priority of documentation in Agile practices, 98 extremely important when estimating effort, (2) 73 would re-estimate a task when the documented information was changed, and (3) functional requirements, user stories, definition of done, UI wireframes, acceptance criteria, and task dependencies were ranked as the most useful types of documented information for effort estimation. Nevertheless, many respondents reported that these useful types of documented information were occasionally changing or missing. Based on our study results, we provide recommendations for agile practitioners on how effort estimation can be improved by focusing on just-enough documentation.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 5

page 6

page 7

page 8

11/24/2017

Non-functional Requirements Documentation in Agile Software Development: Challenges and Solution Proposal

Non-functional requirements (NFRs) are determinant for the success of so...
04/04/2019

Group Maturity and Agility, Are They Connected? - A Survey Study

The focus on psychology has increased within software engineering due to...
05/20/2021

Understanding the Perceived Relevance of Capability Measures: A Survey of Agile Software Development Practitioners

Context: In the light of the swift and iterative nature of Agile Softwar...
01/06/2019

A conversation around the analysis of the SiP effort estimation dataset

The analysis of over ten years of commercial development using Agile (10...
04/04/2019

Is it Possible to Disregard Obsolete Requirements? - An Initial Experiment on a Potentially New Bias in Software Effort Estimation

Effort estimation is a complex area in decision-making, and is influence...
04/02/2018

Why Software Effort Estimation Needs SBSE

Industrial practitioners now face a bewildering array of possible config...
03/24/2021

Is it Possible to Disregard Obsolete Requirements? A Family of Experiments in Software Effort Estimation

Context: Expert judgement is a common method for software effort estimat...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

I Introduction

Effort estimation is an important practice for planning activities in Agile iterative development. It helps a software development team to approximates the size of a task, enabling an optimal workload allocation for a short iteration of Agile [7, 16]. The estimated effort of the tasks in previous iterations also can be used to calculate the delivery capability, which helps the team to better plan the next iteration [6, 40]. Hence, accurate effort estimation benefits the team in creating a reliable iteration plan to deliver the product increment [7, 40].

The effort of a task is typically estimated based on the available information [40, 48]. One of the common approaches to manage and convey information is to use documentation [2, 18, 33]. For example, backlog items are used to document the information related to a feature (e.g., user stories, acceptance criteria) [29, 42]. A lower-level backlog item (like development tasks) may also include the detail information related to the implementation of the feature (e.g., a description of a function to be developed) [34].

On the other hand, in Agile practices, documentation has a lower priority than working software [15]. Based on this concept, it is possible that the information that is recorded in documentation (documented information, henceforth) may not be well maintained, resulting in inadequate information for the team to understand development tasks [29]. Several studies also discovered that documented information in Agile projects can be incorrect, incomplete, outdated, and changing [1, 29, 52]. Yet, little is known about how documentation effort can be minimized while achieving accurate estimation.

In this paper, we aim to understand how just-enough documentation can be provided (i.e., what information should be documented). Hence, practitioners can save documentation effort while achieving accurate effort estimation. Therefore, we conducted a survey study with Agile practitioners to investigate the importance and the types of documented information that are useful for effort estimation in the current practices. Our findings will be beneficial to Agile practitioners, tools developers, and researchers. Agile practitioners can pay attention to the quality of useful documented information to achieve more accurate estimation. Tool developers and researchers can focus their attention on developing tools and techniques to help practitioners maintain documentation for effort estimation. Based on the responses of 121 Agile practitioners, we answered the following research questions:

(RQ1) To what degree is documented information important for effort estimation?
Motivation
: Accurate effort estimation requires an adequate understanding of a task [28, 48, 44, 47]. While documentation can convey the information to understand the task [23, 2, 18], it has a lower priority than delivering working software in the Agile concept [15]. Hence, we investigated how Agile practitioners perceive the importance of the documented information for effort estimation. Results: 97% of the respondents used documentation to assist effort estimation, and 98% of them reported that documented information was moderately to extremely important when estimating effort. The majority of them attempted to keep documented information and documented effort up to date most of the time. Although documentation has a low priority in Agile, our result shows that the documented information is considered important for effort estimation.

(RQ2) Is a change of documented information more likely to influence re-estimation than the information from other sources?
Motivation
: Prior studies reported that effort may be re-estimated when the information is changed [22, 4]. However, in Agile, the information can be conveyed through various channels, e.g., documentation, in-person, discussion threads [2, 18, 46]. Hence, we set out to investigate how often a task was re-estimated when the documented information was changed in comparison to the information from other sources. Results: The reported frequency of re-estimation when documented information was changed is statistically higher than the reported frequency of re-estimation when information from other sources was changed. In general, the respondents noted that re-estimation will be performed if the information change is significant or is from a reliable source like documentation. This result shows that when documented information is changed, a task is often re-estimated.

(RQ3) What types of documented information are useful for effort estimation?
Motivation
: Comprehensive documentation can be considered as an overhead in Agile [15, 23]. On the other hand, inadequate documentation may lead to a misunderstanding of a task [44, 29]. Hence, just-enough and just-in-time documentation would be suitable for Agile practices [20]. Yet, little is known what types of information are considered useful for effort estimation and should be documented. Results: Functional requirements (i.e., the task information about the functionality of the system), user stories, definition of done, UI wireframes, acceptance criteria, and task dependencies were ranked as the six most useful documented information for effort estimation. Yet, many respondents reported that these useful types of documented information were occasionally (or more) changing or missing during (or after) effort estimation. Our results suggest that the information that is useful for effort estimation (especially for the six useful types of documented information) should be documented and reviewed prior to finalizing effort estimates.

Our findings highlight that documented information is important for effort estimation and a change of documented information can lead to re-estimation. Aligning with the Agile practices, our recommendations can be done in a just-in-time manner when the information is needed for effort estimation. In other words, attention should be paid to ensure that the useful documented information for effort estimation has sufficient detail prior to finalizing effort estimates to avoid making unnecessary assumptions. In addition, to help practitioners achieve accurate effort estimation, our work highlights the need for tools and techniques to timely assess the quality of documented information.

Paper organization. Section II defines the terms used in this paper. Section III presents our research methodology. Section IV presents the study results. Section V provides a broader implication of the results. Section VI discusses related work. Section VII discloses the threats to validity. Finally, Section VIII draws the conclusion.

Ii Terms & Definitions

This section defines the terms that are used in this paper.

Information

refers to a description that provides an understanding of a software product under development, e.g., software features [10, 25].

Documented information

refers to the information related to a software product that is captured and recorded in documentation (i.e., a collection of documents) [25, 14]. The documented information can be in various forms. For example, the information related to a feature (e.g., user stories, acceptance criteria) can be documented in a backlog item [42, 29]. A description of a function to be developed can be documented in a lower-level backlog item like a development task [34]. The documentation we consider can be recorded on a physical paper or in a task management system (e.g., JIRA) [23, 22, 31, 41, 18].

We do not refer the documentation to textual descriptions related to source code (e.g., code comments) [11, 52], software development process [10], user manuals, and informal communication (e.g., discussion thread, chat) [46, 18].

Iii Research Method

In this section, we describe our survey design, survey questions, recruitment process, and data analysis.

Item Question
S2: Preliminary questions
PQ1* Does your team perform effort estimation in any stage of the software development lifecycle? (yes/no)
PQ2 Have you ever participated in effort estimation of your team? (yes/no)
S4: Importance of documented information (RQ1)
Q1 Does your team use any documentation when performing effort estimation? (yes/no)
Q1.1 What is the source of information (other than the information in the documentation) that your team uses in effort estimation? (Open-ended)
Q2 In general, how important is the task-related documented information when you estimate the task?
Q3 How often do you update the documentation when new information is available?
Q4 How often do you update the estimated effort in the documentation when the task is re-estimated?
S5: Re-estimation (RQ2)
Q5* Before you work on a task, do you re-estimate the task if the documented information that is relevant to the task is changed or updated?
Q6* Before you work on a task, do you re-estimate the task if the information from other sources than documentation is changed or updated?
Q7* While you are working on a task, do you re-estimate the task if the documented information that is relevant to the task is changed or updated?
Q8* While you are working on a task, do you re-estimate the task if the information from other sources than documentation is changed or updated?
S6: Useful information and its quality (RQ3)
Q9 On average, which of the following documented information do you find most useful in effort estimation? (Ranking)
Q10 For each of the selected types of documented information, please select the following quality issues that you occasionally (or more) encounter
during (or after) effort estimation for an average task that you estimate.
Multiple selections of quality issues:
missing,
incorrect,
outdated,
changing, or ◯ generally no issue
* Questions with asterisk included an optional open-ended answer for further elaboration.
A Likert scale of importance: ◯ Extremely important, ◯ Very important, ◯ Moderately important, ◯ Slightly important, ◯ Not at all important
A Likert scale of frequency: ◯ Always (~100%), ◯ Most of the time (~75%), ◯ About half the time (~50%), ◯ Sometimes (~25%), ◯ Never (~0%)
TABLE I: Survey questions (excluding demographics questionsfootnote 2).

Iii-a Survey Design

The survey was designed based on the recommended survey design practice in software engineering to avoid known problems [17, 32]. Our survey consisted of seven sections which included 11 questions to address our research questions (see Table I) and 11 questions to collect demographics about the respondents. Participation was voluntary and the estimated time to complete a survey was 10-15 minutes. The survey responses were recorded anonymously. The respondent could drop out anytime, and the incomplete response was automatically deleted. The survey was created using an online survey platform named Qualtrics. To ensure the overall quality of responses, we used the features of Qualtric to prevent bots and multiple/duplicate responses from a single respondent.111http://qualtrics.com/support/survey-platform/survey-module/survey-checker/response-quality

Pilot Tests: Comprehensibility of the survey questions was our main construct concern when designing the survey [30]. As suggested by Ghazi et al. [17], we conducted two pilot tests with five software engineers from our connections. These five participants of the pilot tests were software engineers who used an Agile method and used effort estimation (i.e., the target population of our survey). They had 5 to 12 years (7.6 years on average) of experience. After the pilot tests, the comprehensibility of the survey was improved by adding details, rewording, adding and merging options, and providing definitions. Finally, our survey and related documents were reviewed and approved by the Human Ethics Advisory Group of the authors’ institution.

Iii-B Survey Questions

Table I provides an overview of our survey questions that address our three RQs.222The full survey and the replication package is available at http://figshare.com/s/74c2509bdb23de953da3 The questions with an asterisk (*) included an optional open-ended question to collect an elaboration for an answer of the corresponding question. We now briefly describe each section of our survey.

S1: Informed consent. This section informed the respondent about the researchers’ contact, the purpose of this study, how a response will be recorded, and how long will it take to complete the survey. Before proceeding to the next section, the respondent was asked to consent to participate in the study.

S2: Preliminary questions. We asked two preliminary questions to check whether the respondent had been involved in effort estimation. To be specific, we asked whether the team of the respondent performed effort estimation at any stage of the software development process (PQ1) and whether the respondent had ever participated in effort estimation of the team (PQ2). Given that a respondent who answered ‘no’ to one of these two questions is unlikely to have background knowledge about effort estimation of the team, the rest of the survey questions were skipped and the respondent was directed to the Demographics section.

S3: Terms and Definitions. This section provided the terms and definitions that we used in this survey (see section II). This is to ensure that the respondent will have the correct understanding of the survey questions. We also informed the respondent that the survey questions focused on effort estimation at the lowest granularity of a work item that s/he estimated. Since the respondent may be involved in multiple projects, we asked the respondent to answer the survey questions based on the project in which s/he was most involved in. Finally, we asked the respondent to confirm that s/he understood the terms and definitions.

S4: Importance of documented information. The survey questions in this section aimed to address our RQ1. We asked whether the respondent used any documentation when performing effort estimation (Q1). If the respondent answered ‘no’, we asked what is the other sources of information that the respondent used (Q1.1). Then, the respondent was directed to the Demographics section (S7). If the respondent answered ‘yes’ to Q1, we asked the respondent to rate the importance of documented information for effort estimation (Q2) using a 5-points Likert scale of importance (see Table I). Since the respondent who answered ‘yes’ to Q1 used documentation for effort estimation, we further asked the respondent about documentation maintenance. Specifically, we asked how often the respondent updated the document when new information is available (Q3) and how often s/he updated the documented effort when a task is re-estimated (Q4). To answer Q3 and Q4, we provided a 5-points Likert scale of frequency (see Table I).

S5: Re-estimation. The survey questions in this section aimed to address our RQ2. We asked how often the respondent re-estimates a task when the documented information was changed in comparison to the information from other sources [2, 18, 46]. We also considered whether the information was changed before or after the work had started as this may be a factor that influences a decision of re-estimation [22, 50, 37].

Fig. 1: Illustrative Scenario for Re-estimation section (S5).

Hence, we formulated four survey questions (Q5-Q8), where Q5 and Q7 focused on the changes of documented information, while Q6 and Q8 focused on the changes of information from other sources. Furthermore, Q5-6 focused on the re-estimation when the information was changed before the work had started, while Q7-8 focused on the re-estimation when the information was changed after the work had started. To ease the respondent’s understanding, we provided an illustrative scenario (Figure 1) in our survey. Additionally, we included open-ended questions for the respondents to elaborate their answers to Q5-8.

S6: Useful information and its quality. The survey questions in this section aimed to address our RQ3. We asked the respondent to select five types of documented information and rank them based on the usefulness for effort estimation at the lowest granularity that s/he estimated (Q9). We provided nine common types of documented information as the options. We also provided open-ended options where the respondent can fill out other types of documented information that were not on the list. We derived the nine common types of documented information from the literature which are as follows:

  • User Stories [40, 26, 42, 3],

  • Definition of Done [40, 26, 3],

  • Acceptance criteria (including acceptance tests and test cases) [40, 26, 8, 42, 18],

  • Functional requirements [34],

  • Non-functional requirements [34, 3],

  • Task dependencies [42, 18, 45],

  • Code examples [8, 53, 1],

  • UI wireframes (or pre-designed UI) [42, 20], and

  • API reference guides [1].

In our survey, we provided a definition for each of these types to ensure that the respondent had the same understanding.footnote 2 Note that, since we focused on Agile practices, functional requirements are referred to the task information about the functionality of the system [34] as opposed to software requirement specification (SRS) that is prepared upfront in non-Agile software development practices. Both user stories and functional requirements capture the information about the system behaviors. The key difference between the two types is that a user stories is a lightweight expression that focuses on the desired business value. In contrast, functional requirements focus on the technical aspect of a functionality to be developed.

For each selected type of documented information, we further asked the respondent to select the quality issues that s/he occasionally (or more) encountered during (or after) effort estimation (Q10). In this work, we study four common quality issues of documented information: (1) changing, (2) missing, (3) outdated, and (4) wrong [1, 29, 52]. In our survey, we also provided an option of ‘generally no issue’ if the respondent did not encounter any of these four issues.

S7: Demographics. The survey questions in this section aimed to collect the general background and effort estimation practice of the respondent.footnote 2 We asked the respondent about the software development methodology, experience in the IT industry (years), experience in effort estimation (years), the duration of the most involved project (years), office location (country), working team size, and team location (co-located, distributed-inshore, or distributed-offshore).

We asked what the effort estimation technique(s) and unit(s) that the team of the respondent used, and when the team performed effort estimation (e.g., iteration planning, backlog grooming, project planning). As the respondent’s team might use different effort estimation techniques, the questions in this section were multiple selections. We also asked an optional open-ended question about the benefit of effort estimation.

Iii-C Recruitment Process

To recruit software engineers to participate in our survey, we used purposive sampling and snowballing methods.

Purposive Sampling. We used purposive sampling as our main recruitment approach. This method allowed us to identify software engineers who are likely to use an Agile method and effort estimation [13]. In particular, we targeted software engineers who (1) work (or have worked) in an international software company or a top local software company; and (2) have at least one year of experience in software development. We mainly considered the international and top local companies as they are established companies which are more likely to use effort estimates in feature prioritization, while start-up companies tend to use business values as the prioritization target [33]. To find the top local companies, we searched for a list of top companies for each country.

We recruited software engineers via LinkedIn (i.e., a professional networking platform) as it allows us to access the target group of interest [38, 17]. We identified the target group of software engineers using publicly available LinkedIn profiles. Then, we sent a survey invitation to the targeted software engineers via the LinkedIn direct message.

To ensure the diversity of respondents, we controlled the number of invitations per software company. More specifically, for a small company (<200 employees), we randomly selected 10-20 targeted software engineers. For a large company (>200 employees), we randomly selected 20-50 targeted software engineers. Since LinkedIn limits the number of invitations,333https://www.linkedin.com/help/linkedin/topics/6096/6097/4800 we sent the survey invitation to about 20-30 software engineers per day for 60 consecutive days. Based on this process, we sent a survey invitation to 1,500 software engineers via LinkedIn. Broadly speaking, 36%, 20%, 33%, and 10% of direct invitations via LinkedIn were sent to software engineers in the Americas, Europe, Asia, and Oceania, respectively.

Snowballing. To further reach out to the target population, we performed a snowballing approach [9]. Specifically, if the respondent informed us that s/he completed the survey, we asked them to spread the survey link to other software engineers who use Agile methods and effort estimation. In total, we asked 16 respondents to spread the link.

Iii-D Data Preprocessing

In this study, we focused on effort estimation in Agile projects. However, when we sent the survey invitation, we could not determine whether a software engineer has worked in an Agile project or his/her team used effort estimation. Therefore, before analyzing the survey results, we selected the responses of software engineers of interest based on their answers to the preliminary and demographic questions. To be specific, we selected the responses using the following criteria.

Criterion 1 - Use an Agile method: Since we focused on Agile practices, we excluded the respondents who used non-Agile methods, i.e., Waterfall model, Incremental model, and not sure or N/A.

Criterion 2 - Have Effort Estimation Experience: To ensure that the respondents had experience in effort estimation, we excluded the respondents who answered ‘no’ to the preliminary questions about the involvement in effort estimation (i.e., PQ1 and/or PQ2).

Iii-E Card Sorting

Similar to prior work [11, 49], we applied card sorting to analyze the open-ended answers of Q5-8. As suggested by Spencer [43], we first conducted an open card sorting procedure to extract themes of the answers (cards). For each open-ended question, the first two authors of this paper independently grouped the answers based on the thematic similarities and defined a theme for each group. Then, the first two authors discussed to develop the agreed-upon themes of the answers. To validate the agreed-upon themes, the third author of this paper performed closed-card sorting, i.e., sorting the answers into the themes that were developed by the first two authors. Finally, we measured the inter-rater reliability using Cohen’s Kappa coefficient [35]. The overall Kappa values ranged from 0.65-0.80, indicating substantial agreements between the sorters.

Iv Results

In this section, we present the results of our survey study. In particular, we first provide an overview of the survey results. We then analyze the survey results to answer each of our RQs.

Iv-a Survey Results Overview

Fig. 2: The distributions of respondents across different groups of demographics and effort estimation practices. A respondent can select multiple estimation methodologies, estimation units, and time points when estimating effort.

We received 148 responses in total from the 1,500 direct LinkedIn invitations and snowballing (a response rate of 10%).444Since we used a snowballing approach and the responses are anonymized, we do not know the number of snowballed responses. Hence, the reported response rate may be varied. The median time taken by the respondents to complete the survey was 14 minutes. After the data pre-processing (see Section III-D), we retained 121 respondents of interest, i.e., software engineers who used an Agile method and had been involved in effort estimation.

Respondent Demographics. Figure 2 presents the distributions of respondents across different demographic groups. Most of the respondents used Scrum (78%) or Kanban (18%) as their development methodology. A majority of them (72%) had more than five years of experience in the IT industry. The software project on which a respondent reported typically had a duration of three years or under (83%). The typical team size was 6-10 people (50%) and 11-30 people (27%). About 52% of the respondents worked in a co-located team, while others worked in a distributed team. Our respondents were from 25 countries, e.g., Singapore (18), the United States of America (14), Thailand (13), Australia (12), India (11), others (53).

Effort Estimation Practices. Figure 2 also shows that our respondents had experience in diverse effort estimation practices. Figure 2 shows the effort estimation practices of our respondents. The majority of the respondents had 1-5 years (53%) or 5-10 years (27%) of experience in estimating effort. They performed effort estimation during iteration planning (83%) and/or backlog grooming (37%). The common estimation methods were Expert Judgement (50%) and/or Planning Poker (48%). The common effort unit(s) were story points (61%) and/or person-days (43%). Based on the open-ended questions, the respondents reported that they used effort estimation to 1) know the size and complexity of development tasks, 2) achieve better resource allocation, 3) accurately estimate the delivery capability and the time to deliver, and 4) build trust with stakeholders based on an accurate and transparent workflow.

Iv-B (RQ1) To what degree is documented information important for effort estimation?

A vast majority of the respondents (; 97%) reported that they used documentation when performing effort estimation (Q1). Figure 3 shows that most of the respondents (; 98%) who used documentation when estimating effort rated the importance of documented information as moderately to extremely important for effort estimation (Q2). On the other hand, only two respondents rated it as slightly importance and none rated it as not at all important. This result indicates that practitioners considered documented information to be moderately to extremely important for effort estimation.

Based on Q1.1, the respondents who did not use documentation when estimating effort (4 out of 121 respondents) reported that, instead of using documentation, they relied on 1) team velocity in the past iterations, 2) verbal communication, or 3) code-based statistics from code indexer (e.g., Google Kythe).

Fig. 3: Importance of documented information when performing effort estimation.
Fig. 4: The frequencies that the respondents updated the documentation when new information was available and updated the documented effort after re-estimation.

The survey results of Q3 and Q4 also show that the respondents who used documentation when estimating effort tend to keep both documented information and documented efforts up to date. Figure 4 shows that 34% () of the respondents always updated documentation when new information was available (Q3); and 42% () of them updated the document most of the time. Similarly, 53% () of the respondents always updated the documented effort after re-estimation (Q4). These results suggest that the respondents attempted to keep the documentation up to date.

Prior studies pointed out that the importance of documented information may be varied based on the team settings [21, 24, 29, 18]. Hence, we further checked whether the documented information has a different level of importance given a different team setting. We considered three aspects of team settings, i.e., the team location, size, and project duration. For each aspect, we used the Kruskal-Wallis rank sum test to compare the distributions of the importance level of documented information across the groups of respondents. For example, we examined whether the distributions of the importance level of documented information are statistically different among the three groups of team location (i.e., co-located, distributed-inshore, and distributed-offshore). The Kruskal-Wallis rank sum test is a non-parametric test that is robust to non-homogenous and a small sample. Similar to prior work [19, 27, 51], we assigned the numerical values to the Likert scale, i.e., 5 (for Extremely important) to 1 (for Not at all important) when performing a statistical test. For the three aspects of team settings, all the tests yielded not statistically significant (), indicating that the distributions of importance levels are not statistically different for the respondents from different team settings.

[boxsep=1pt,left=1pt,right=1pt,top=1pt,bottom=1pt] Findings: 97% of the respondents used documentation to assist effort estimation, and 98% of these respondents reported that documented information is moderately to extremely important when estimating effort. Furthermore, a majority of the respondents attempted to keep documented information and documented effort up to date most of the time.

Iv-C (RQ2) Is a change of documented information more likely to influence re-estimation than the information from other sources?

Fig. 5: The frequencies that the respondents would re-estimate if the documented information (or the information from other sources) was changed before (or after) the work had started.

We found that the respondents tend to re-estimate a task when the documented information was changed more often than when the information from other sources was changed. Figure 5 shows that 73% () of the respondents reported that they would re-estimate at least 50% of the time when the documented information was changed before the work had started (Q5).555Note that we excluded the four respondents who did not use any documents in effort estimation since we focus on the information in documents. On the other hand, 50% of the respondents reported that they would re-estimate at least 50% of the time when the information from other sources was changed before the work had started (Q6). Similarly, for the after-start-working case, 62% of the respondents reported that they would re-estimate at least 50% of the time when the documented information was changed (Q7); while 46% of the respondents reported that they would re-estimate at least 50% of the time when the information from other sources was changed (Q8).

To statistically confirm our finding, we used the one-sided Wilcoxon signed-rank test to examine whether a respondent reported the frequency in Q5 (i.e., when the documented information is changed) higher than the frequency in Q6 (i.e., when the information from other sources was changed) for the before-start-working case. Similarly, we performed the test to compare the difference of frequencies in Q7 and Q8 for the after-start-working case. We opted to use the one-sided Wilcoxon signed-rank test as it is a non-parametric test which is robust to non-homogenous and a small sample. Before performing the tests, we converted the Likert scale of frequency into numerical values (e.g., ‘Always (~100%)’ to 100). The tests showed that the reported frequency in Q5 is statistically higher than the reported frequency in Q6; and the reported frequency in Q7 is statistically higher than the reported frequency in Q8 (). These results confirm that the change of the documented information is more likely to influence re-estimation than the change of the information from other sources.

Theme #
Reasons to re-estimate
T1) Only re-estimate if the change is significant or from a reliable source like documentation. 52
T2) Always re-estimate to correct the accuracy, velocity, or development plan. 26
Reasons to not re-estimate
T3) The team has a policy (or norm) to not re-estimate after the work had started. 12
T4) Absorb the change (attempt to complete within the original time frame) and/or roll the work over to the next iteration. 9
T5) Over-estimate to cover future change. 2
Total responses 101
TABLE II: Themes derived from the answers of the open-ended questions of Q5-8.

For the optional open-ended questions of Q5-8, we received 101 responses. We sorted the responses into five themes of reasons based on the card sorting procedure (see Section III-E). Table II lists the themes which are grouped into (1) reasons to re-estimate and (2) reasons to not re-estimate.

In general, 52 responses indicated that the decision of re-estimation was based on the significance of the information change (T1): "If the change is not significant, and does not change the scope, the same points may remain. However, if due to the change, the original estimate is not accurate, new point will be estimated. […]". In addition to the significance of the change, some responses also emphasized that the re-estimation decision also depended on the source of information: "If it’s not documented, it likely won’t be important. This is the motivation for everything to be documented." Another 26 responses pointed out that re-estimation was required in order to keep the delivery capability and development plan realistic (T2): "Any re-estimation is always needed to have a correct velocity sprint productivity […]."

The common reason for not re-estimating was that they had a policy (or norm) to not re-estimate after the work had started (T3): "At my team, estimation is frozen since task is scheduled for current sprint and sprint is ongoing." Another nine responses indicated that they tried to absorb the change and/or rolled the remaining work over to the next iteration (T4): "Mostly, our team just try to done the task even it’s exceed the estimation. However, if the sprint ended and this task still exist and need to carry over to next sprint, we will re-estimate the remaining task […]." Other two responses noted that they intentionally over-estimated to cover the future change (T5): "Because I usually expect delays based on missing or incorrect documentation, I overestimate my tasks to take into account these issues. This allows me not to have to re-estimate."

[boxsep=1pt,left=1pt,right=1pt,top=1pt,bottom=2pt] Findings: The reported frequency of re-estimation when the documented information was changed is statistically higher than the reported frequency of re-estimation when the information from other sources was changed. In general, the respondents noted that re-estimation will be performed if the information change is significant or from a reliable source like documentation.

Iv-D (RQ3) What types of documented information are useful for effort estimation?

Type of documented information Ranked as Have Quality
Useful Issue(s)
Functional requirements 91 67 (74%)
User stories 74 41 (55%)
Definition of Done 76 41 (55%)
UI wireframes 61 46 (75%)
Acceptance criteria 79 65 (82%)
Task dependencies 61 51 (84%)
Non-functional requirements 40 36 (90%)
API reference guides 33 24 (73%)
Code examples 6 3 (50%)
TABLE III: The number of respondents who ranked the type of documented information as one of the five most useful information; and the number of respondents who indicated that the type of documented information occasionally (or more frequent) had a quality issue during (or after) effort estimation.
Fig. 6: The distributions of importance rank of documented information, where the rank of 1 indicates that that type of documented information is the most useful.

Table III shows that functional requirements, user stories, definition of done, UI wireframes, acceptance criteria, and task dependencies were frequently rated as useful documented information for effort estimation (Q9).footnote 5 Moreover, Figure 6 shows that functional requirements (i.e., the description of a function to be developed [34]) were ranked as the most useful type of documented information for effort estimation where 52% () of the respondents ranked this type as the top one. The user stories was also ranked as the second most useful type where 43% () of the respondents ranked this type as the top one. These results suggest that the documented information that provides an understanding of the software system’s behavior is important when estimating effort.

Nine respondents also provided additional types of information that are useful for effort estimation in the open-ended options. Most of the reported information is related to the team context, e.g., the knowledge, expertise, or workload of developers. In addition, one respondent pointed out that steps-to-reproduce and observed behaviors are useful documented information when estimating the effort of a bug-fixing task.

Table III shows that many respondents reported that these useful types of documented information occasionally (or more) had at least one kind of quality issue during (or after) effort estimation. For example, 74% () of the respondents who ranked functional requirements as useful indicated that this type of documented information had quality issues. Figure 7 shows that 81% () of the respondents indicated that the functional requirements in the documentation were occasionally changing. Furthermore, Table III shows that other useful types documented information, i.e., user stories, definition of done, UI wireframes, acceptance criteria, and task dependencies were reported as having quality issues by many respondents (55%-84%; see Table III). Figure 7 also shows that user stories and UI wireframes were occasionally changing, while definition of done, acceptance criteria, and task dependencies were occasionally missing. These results suggest that despite the usefulness of these types of documented information for effort estimation, they were occasionally changing or missing.

Fig. 7: The number of respondents who reported that the type of documented information occasionally (or more) had quality issues during (or after) effort estimation. Note that multiple quality issues could be selected.

[boxsep=1pt,left=1pt,right=1pt,top=1pt,bottom=1pt] Findings: Functional requirements, user stories, definition of done, UI wireframes, acceptance criteria, and task dependencies were ranked as the six most useful documented information for effort estimation. Yet, many respondents reported that these useful types of documented information were occasionally (or more) changing or missing during (or after) effort estimation.

V Discussion

We now discuss a broader implication and provide a recommendation based on our results.

V-a Implications

Despite the lower priority of documentation in Agile, the documented information is used and considered important for effort estimation. More specifically, our RQ1 shows that 117 out of the 121 respondents used documentation for effort estimation. Moreover, regardless of the team settings (i.e., the team location, size, or project duration), 98% of the respondents () considered documented information as moderately to extremely important for effort estimation (see Figure 3). From the open-ended question of Q6, some respondents emphasized the importance of documented information, e.g., "I prioritize the documentation over information from other sources. The information outside documentation is needed to be filtered if it is important or it is just good to have/know. […]." Furthermore, the responses of Q3-4 show that a majority of the respondents tend to maintain documentation as they reported that they updated documentation of both information and estimated effort most of the time (see Figure 4). These results highlight the importance of documented information for effort estimation for Agile practitioners.

The changing documented information can have an impact on estimation accuracy. Figure 5 shows that 73% () of the respondents re-estimated the task at least 50% of the time when the documented information was changed, which is more often than the change of the information from other sources. The respondents also noted in the open-ended questions of Q5-8 that they would re-estimate if the change was significant or was from a reliable source like documentation. Our statistical tests also confirm that the reported frequency of re-estimation when documented information was changed is statistically higher than that of the information from other sources. These results suggest that the practitioners are more likely to re-estimate the task if documented information was changed in comparison to a change of the information from other sources.

V-B Recommendations

Our survey results highlight the importance of documented information for effort estimation. The results also show that the quality of documented information can have an impact on estimation. Hence, we provide two recommendations on how documentation should be maintained while achieving accurate estimation. Aligning with the Agile practices, our recommendations can be done in the just-enough (i.e., focusing on the useful types of documented information) and just-in-time (e.g., before iteration planning) manner.

Prior to finalizing effort estimates (e.g., at iteration planning), attention should be paid to ensure that the information that is useful for effort estimation is documented to avoid making unnecessary assumptions. Our RQ3 shows the six types of documented information that were ranked as the most useful for effort estimation (see Figure 6). However, several respondents reported that these useful types of documented information had quality issues, especially for the changing issue (Table III). The responses of the open-ended question of Q5 also highlight that a change of such useful information often caused re-estimation: "Depends on changes. […] the change of user stories/functional requirements changes the original estimation often." Since the effort estimates are used in iteration planning, such uncertainty may cause the plan to become unreliable [44, 7]. Hence, to facilitate effort estimation and improve its accuracy, these useful types of information should be documented and reviewed. One possible actionable approach is to confirm the information with stakeholders. This is also aligned with one respondent who noted that "[…] It’s actually good to communicate with all stakeholders earlier rather than later about the assumption […]." While it is not required to document all the information at the very early stages of development, our recommendation can be done in the timely manner. In other words, before iteration planning, practitioners can check whether these useful types of information are documented, correct, and up to date to avoid making unnecessary assumptions.

Future research should focus on developing tools and techniques to assess the quality of documented information for effort estimation. While it is recommended to maintain the documentation for effort estimation, it may still be tedious for practitioners to assess the quality of information of all tasks. Techniques to detect missing or changing information should help practitioners to timely manage the useful documented information, which in turn will lead to more accurate effort estimation. Yet, less focus has been given to developing techniques that detect missing or changing information that is useful for effort estimation. To the best of our knowledge, existing work focused on developing techniques to detect missing information for software development, e.g., the information for bug-fixing tasks [5], feature requests in issue tracking systems [36], and user stories from developer-client conversation transcripts [39]. Therefore, future research should develop techniques that help practitioners be aware of the quality issues of documented information that could impact the accuracy of effort estimation.

Vi Related Work

In this section, we discuss related work with respect to effort estimation and documentation.

Information and Estimation. Several studies investigated the impact of inadequate information on effort estimation. Jørgensen et al. [28] found that unclear task information was one of the reasons for inaccurate estimation. Svensson et al. [44] pointed out that inadequate information could lead the team to misunderstood the task, which may lead to inaccurate estimation. Usman et al. [47, 48] reported that Agile practitioners perceived that a lack of details in requirements could lead to inaccurate estimation. Hoda and Murugesan [22] conducted a grounded theory study with Agile practitioners and reported that some information could be missing or delayed, which could lead to a wrong assumption and inaccurate estimation. Bick et al. [4] conducted a case study with Agile teams and reported that a task is re-estimated when the information change caused the estimated effort to became unrealistic. Complimenting the prior studies, this work shows that a change of documented information is more likely to influence re-estimation than a change of the information from other sources.

Documentation in Agile. Documentation is one of the approaches to record and convey information related to a software product, which a team can use as a reference [2]. Sedano et al. [42] reported that a product backlog could be used to remind the work to be done and assist the face-to-face communication. Kasauli et al. [29] reported that practitioners relied on information recorded in documentation (i.e., requirements specifications, backlog items) to build and maintain understanding for several purposes, e.g., task implementation, system maintenance, defining test cases. Aligning with the prior work, our results highlight that documentation is commonly used for effort estimation and documented information is considered moderately to extremely important when estimating effort.

Minimal Documentation. Comprehensive documentation is considered overhead based on the Agile manifesto [15]. On the other hand, Kasauli et al. [29] discussed that information might be inadequate for developers if the team aimed to minimize documentation effort by only documenting user stories and test cases. Ernst and Murphy [12] reported that information could be documented in a just-in-time fashion, i.e., requirements were first sketched out with simple natural language statements and then fully elaborated just before or during development. Heck and Zaidman [20] found that some types of information (e.g., use cases, UI mock-ups) might not be mandatory at the early stage. Motivated by the prior studies, we identified the useful types of information that should be documented for effort estimation, i.e., functional requirements, user stories, definition of done, UI wireframes, acceptance criteria, and task dependencies.

Quality Issues. Several studies analyzed the quality of documentation. Aghajani et al. [1] manually analyzed 878 documentation-related artifacts and found that documented information was often found incorrect, incomplete, and outdated. Kasauli et al. [29] reported that the information documented in backlog items was often changing while the information documented in requirement specifications was often outdated. Prior studies also reported that documented information could be missing or did not match the developer needs for bug-fixing tasks [5, 53]. Similarly, in the context of Agile effort estimation, we found that the useful types of documented information for effort estimation were occasionally (or more) changing or missing during or after effort estimation.

Vii Threats to Validity

Construct validity is related to the concerns on the survey design. The survey respondents might misinterpret the survey questions or definitions of the terms. To mitigate this threat, we conducted two pilot tests with five experienced software engineers to check the comprehensibility of our survey questions. We also provided the definitions of terms and the nine types of documented information we used in the survey. In addition, we informed the invitees that they could directly contact the authors for clarification or requesting support via video calls.

It might also be possible that the term "functional requirements" might be misinterpreted as the software requirement specification (SRS) of the non-agile processes. However, our study focused only on the software engineers who used Agile methods and we asked them to answer the survey questions at the lowest granularity that they estimate. We also confirmed the interpretation with the pilot test participants and they explained that the functional requirements were understood as the task information about the functionality of the system. Hence, it is less likely that the functional requirements were interpreted as the SRS.

Internal validity concerns the confounding factors that might impact our findings. Our survey results might be varied based on other factors, e.g., company practices or product domain. However, we did not ask the participants about their company or product because such sensitive information might cause them to be reluctant to participate, increased the risk of evaluation apprehension [17] and self-selection bias [30]. Some invitees also responded that they would not participate in the survey if it collects such information. Other study methods (e.g., interviews, Delphi sessions) might provide additional insights about the variation based on other factors.

In RQ2, the responses of Q5-8 might be biased toward re-estimation if documented information was changed since most of our respondents consider documentation important (RQ1). To check this, we used the Spearman rank correlation test to measure whether the responses of Q2 and Q5-8 are highly correlated (i.e., the higher the perceived importance of documented information, the more likely the re-estimation is performed when documented information was changed). We found that the correlations are small to moderate (i.e., is 0.2-0.39). Hence, the responses in RQ2 are less likely biased.

The first two authors conducted the analysis of open-ended answers (i.e., card sorting). Personal experiences of the authors might influence the analysis. Therefore, we validated the analysis by the closed-card sorting performed by the third author. The Cohen’s Kappa coefficient [35] ranged from 0.65-0.80, indicating substantial agreements between the sorters.

When we performed statistical tests, we converted the Likert scale into numerical values. The statistical test results could be impacted because the distance between the scale values were not ordinal. However, as suggested by Harpe [27] and Jamieson [19], it is a common practice to treat rating items with at least five numerical categories as continuous data.

External validity is related to the concern on the generalizability of our findings. Despite the diverse background of our respondents (c.f. Section IV-A), the survey results might not generalize to all kinds of Agile projects or effort estimation practices. Nevertheless, to facilitate a replicate study of this work, we provided a full set of survey questions and relevant documents in the replication package.2

Viii Conclusions

Effort estimation is an important practice of Agile planning activities. To estimate the effort, practitioners use the information to understand the task. One of the common approaches to convey the information is to use documentation. However, documentation has a lower priority than delivering working software in the Agile practices. It is possible that documentation may not be well maintained and may impact the accuracy of effort estimation. Yet, little is known about how documented effort can be minimized while achieving accurate estimation. Hence, in this paper, we investigated the importance of documentation and the types of documented information that are useful for effort estimation in the current Agile practices. Through a survey study with 121 Agile practitioners, we found that:

  • Despite the low priority of documentation in Agile, a majority of our respondents used documentation to assist effort estimation and considered the documented information as moderately to extremely important when estimating effort.

  • The reported frequency of re-estimation when the documented information was changed is statistically higher than the reported frequency of re-estimation when the information from other sources was changed. Furthermore, their re-estimation decision was based on the significance or the source of the information change.

  • Functional requirements, user stories, definition of done, UI wireframes, acceptance criteria, and task dependencies were ranked as the six most useful documented information for effort estimation, however many respondents reported that these useful types of information were occasionally (or more) changing or missing during (or after) effort estimation.

The key contribution of our work is to shed light on the importance of documented information and its impact on the accuracy of effort estimation in Agile practices. Our findings suggest that the quality of documented information can have an impact on effort estimation. Nevertheless, we do not expect Agile practitioners to prepare comprehensive documentation. Indeed, to achieve a practice of minimal documentation of Agile, our work highlights the six types of documented information that practitioners should pay attention before performing effort estimation. Our recommendations can be done in a timely manner, which helps practitioners achieve more accurate effort estimation with an optimal maintenance effort of documentation.
Acknowledgement. P. Thongtanunam was partially supported by the Australian Research Council’s Discovery Early Career Researcher Award (DECRA) funding scheme (DE210101091).

References

  • [1] E. Aghajani, C. Nagy, O. L. Vega-Márquez, M. Linares-Vásquez, L. Moreno, G. Bavota, and M. Lanza (2019) Software Documentation Issues Unveiled. In Proc. of the ICSE, pp. 1199–1210. Cited by: §I, 7th item, 9th item, §III-B, §VI.
  • [2] Y. Andriyani, R. Hoda, and R. Amor (2017) Understanding Knowledge Management in Agile Software Development Practice. In Proc. of the KSEM, pp. 195–207. Cited by: §I, §I, §I, §III-B, §VI.
  • [3] W. Behutiye, P. Karhapää, D. Costal, M. Oivo, and X. Franch (2017) Non-functional Requirements Documentation in Agile Software Development: Challenges and Solution Proposal. In Proc. of the PROFES, pp. 515–522. Cited by: 1st item, 2nd item, 5th item.
  • [4] S. Bick, K. Spohrer, R. Hoda, A. Scheerer, and A. Heinzl (2018) Coordination Challenges in Large-Scale Software Development: A Case Study of Planning Misalignment in Hybrid Settings. TSE 44 (10), pp. 932–950. Cited by: §I, §VI.
  • [5] O. Chaparro, J. Lu, F. Zampetti, L. Moreno, M. Di Penta, A. Marcus, G. Bavota, and V. Ng (2017) Detecting Missing Information in Bug Descriptions. In Proc. of the ESEC/FSE, pp. 396–407. Cited by: §V-B, §VI.
  • [6] E. Coelho and A. Basu (2012) Effort Estimation in Agile Software Development using Story Points. IJAIS 3 (7), pp. 7–10. Cited by: §I.
  • [7] M. Cohn (2006) Agile estimating and planning. Pearson Education. Cited by: §I, §V-B.
  • [8] S. Davies and M. Roper (2014) What’s in a Bug Report?. In Proc. of the ESEM, pp. 1–10. Cited by: 3rd item, 7th item.
  • [9] M. Denscombe (2014) The good research guide: for small-scale social research projects. McGraw-Hill Education (UK). Cited by: §III-C.
  • [10] C. Ebert and J. De Man (2008) Effectively utilizing project, product and process knowledge. IST 50 (6), pp. 579–594. Cited by: item Information, §II.
  • [11] F. Ebert, F. Castor, N. Novielli, and A. Serebrenik (2019) Confusion in code reviews: Reasons, impacts, and coping strategies. In Proc. of the SANER, pp. 49–60. Cited by: §II, §III-E.
  • [12] N. A. Ernst and G. C. Murphy (2012) Case Studies in Just-In-Time Requirements Analysis. In Proc. of the EmpiRE, pp. 25–32. Cited by: §VI.
  • [13] I. Etikan, S. A. Musa, and R. S. Alkassim (2016) Comparison of Convenience Sampling and Purposive Sampling. AJTAS 5 (1), pp. 1–4. Cited by: §III-C.
  • [14] A. Forward and T. C. Lethbridge (2002) The Relevance of Software Documentation, Tools and Technologies: A Survey. In Proc. of the ACM Symposium on Document Engineering, pp. 26–33. Cited by: item Documented information.
  • [15] M. Fowler and J. Highsmith (2001) The agile manifesto. Software Development 9 (8), pp. 28–32. Cited by: §I, §I, §I, §VI.
  • [16] D. Fucci, C. Palomares, X. Franch, D. Costal, M. Raatikainen, M. Stettinger, Z. Kurtanovic, T. Kojo, L. Koenig, A. Falkner, et al. (2018) Needs and Challenges for a Platform to Support Large-scale Requirements Engineering: a Multiple-case Study. In Proc. of the ESEM, pp. 1–10. Cited by: §I.
  • [17] A. N. Ghazi, K. Petersen, S. S. V. R. Reddy, and H. Nekkanti (2018) Survey Research in Software Engineering: Problems and Mitigation Strategies. IEEE Access 7, pp. 1–24. Cited by: §III-A, §III-A, §III-C, §VII.
  • [18] C. Gralha, D. Damian, A. Wasserman, M. Goulão, and J. Araújo (2018) The Evolution of Requirements Practices in Software Startups. In Proc. of the ICSE, pp. 823–833. Cited by: §I, §I, §I, item Documented information, §II, 3rd item, 6th item, §III-B, §IV-B.
  • [19] S. E. Harpe (2015) How to analyze Likert and other rating scale data. CPTL 7 (6), pp. 836–850. Cited by: §IV-B, §VII.
  • [20] P. Heck and A. Zaidman (2017)

    A framework for quality assessment of just-in-time requirements: the case of open source feature requests

    .
    RE 22 (4), pp. 453–473. Cited by: §I, 8th item, §VI.
  • [21] P. Heck and A. Zaidman (2018) A Systematic Literature Review on Quality Criteria for Agile Requirements Specifications. SQJ 26 (1), pp. 127–160. Cited by: §IV-B.
  • [22] R. Hoda and L. K. Murugesan (2016) Multi-Level Agile Project Management Challenges: A Self-Organizing Team Perspective. JSS 117, pp. 245–257. Cited by: §I, item Documented information, §III-B, §VI.
  • [23] R. Hoda, J. Noble, and S. Marshall (2012) Documentation Strategies on Agile Software Development Projects. IJAESD 1 (1), pp. 23–37. Cited by: §I, §I, item Documented information.
  • [24] I. Inayat, S. S. Salim, S. Marczak, M. Daneva, and S. Shamshirband (2015) A systematic literature review on agile requirements engineering practices and challenges. Computers in human behavior 51, pp. 915–929. Cited by: §IV-B.
  • [25] (2010) ISO/IEC/IEEE International Standard - Systems and software engineering – Vocabulary. ISO/IEC/IEEE 24765:2010(E), pp. 1–418. Cited by: item Information, item Documented information.
  • [26] M. James and L. Walter (2010) Scrum Reference Card. CollabNet. Cited by: 1st item, 2nd item, 3rd item.
  • [27] S. Jamieson (2004) Likert scales: How to (ab) use them?. Medical education 38 (12), pp. 1217–1218. Cited by: §IV-B, §VII.
  • [28] M. Jørgensen and T. M. Gruschke (2009) The Impact of Lessons-Learned Sessions on Effort Estimation and Uncertainty Assessments. TSE 35 (3), pp. 368–383. Cited by: §I, §VI.
  • [29] R. Kasauli, G. Liebel, E. Knauss, S. Gopakumar, and B. Kanagwa (2017) Requirements Engineering Challenges in Large-Scale Agile System Development. In RE, pp. 352–361. Cited by: §I, §I, §I, item Documented information, §III-B, §IV-B, §VI, §VI, §VI.
  • [30] M. Kasunic (2005) Designing an effective survey. Technical report Carnegie-Mellon Univ Pittsburgh PA Software Engineering Inst. Cited by: §III-A, §VII.
  • [31] C. Khalil and S. Khalil (2020) Exploring knowledge management in agile software development organizations. IEMJ 16 (2), pp. 555–569. Cited by: item Documented information.
  • [32] B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard, P. W. Jones, D. C. Hoaglin, K. El Emam, and J. Rosenberg (2002) Preliminary Guidelines for Empirical Research in Software Engineering. TSE 28 (8), pp. 721–734. Cited by: §III-A.
  • [33] E. Klotins, M. Unterkalmsteiner, P. Chatzipetrou, T. Gorschek, R. Prikladnicki, N. Tripathi, and L. Pompermaier (2019) A progression model of software engineering goals, challenges, and practices in start-ups. TSE, pp. To appear. Cited by: §I, §III-C.
  • [34] S. Lauesen (2002) Software Requirements: Styles and Techniques. Pearson Education. Cited by: §I, item Documented information, 4th item, 5th item, §III-B, §IV-D.
  • [35] M. L. McHugh (2012) Interrater reliability: the kappa statistic. Biochemia medica: Biochemia medica 22 (3), pp. 276–282. Cited by: §III-E, §VII.
  • [36] T. Merten, M. Falis, P. Hübner, T. Quirchmayr, S. Bürsner, and B. Paech (2016) Software Feature Request Detection in Issue Tracking Systems. In Proc. of the RE, pp. 166–175. Cited by: §V-B.
  • [37] F. Paetsch, A. Eberlein, and F. Maurer (2003) Requirements Engineering and Agile Software Development. In Proc. of the WETICE, pp. 308–313. Cited by: §III-B.
  • [38] C. Palomares, C. Quer, and X. Franch (2017) Requirements Reuse and Requirement Patterns: A State of the Practice Survey. ESE 22 (6), pp. 2719–2762. Cited by: §III-C.
  • [39] P. Rodeghero, S. Jiang, A. Armaly, and C. McMillan (2017) Detecting User Story Information in Developer-Client Conversations to Generate Extractive Summaries. In Proc. of the ICSE, pp. 49–59. Cited by: §V-B.
  • [40] K. S. Rubin (2012) Essential Scrum: A Practical Guide to the Most Popular Agile Process. Addison-Wesley. Cited by: §I, §I, 1st item, 2nd item, 3rd item.
  • [41] E. Schön, J. Thomaschewski, and M. J. Escalona (2017) Agile Requirements Engineering: A systematic literature review. Computer Standards & Interfaces 49, pp. 79–91. Cited by: item Documented information.
  • [42] T. Sedano, P. Ralph, and C. Péraire (2019) The Product Backlog. In Proc. of the ICSE, pp. 200–211. Cited by: §I, item Documented information, 1st item, 3rd item, 6th item, 8th item, §VI.
  • [43] D. Spencer (2009) Card sorting: Designing usable categories. Rosenfeld Media. Cited by: §III-E.
  • [44] R. B. Svensson, T. Gorschek, P. Bengtsson, and J. Widerberg (2019) BAM-Backlog Assessment Method. In Proc. of the XP, pp. 53–68. Cited by: §I, §I, §V-B, §VI.
  • [45] B. Tanveer, L. Guzmán, and U. M. Engel (2016) Understanding and improving effort estimation in agile software development: an industrial case study. In Proc. of the ICSSP, pp. 41–50. Cited by: 6th item.
  • [46] J. Tizard, H. Wang, L. Yohannes, and K. Blincoe (2019) Can a Conversation Paint a Picture? Mining Requirements in Software Forums. In Proc. of the RE, pp. 17–27. Cited by: §I, §II, §III-B.
  • [47] M. Usman and R. Britto (2016) Effort Estimation in Co-located and Globally Distributed Agile Software Development: A Comparative Study. In Proc. of the IWSM-MENSURA, pp. 219–224. Cited by: §I, §VI.
  • [48] M. Usman, R. Britto, L. Damm, and J. Börstler (2018) Effort estimation in large-scale software development: an industrial case study. IST 99, pp. 21–40. Cited by: §I, §I, §VI.
  • [49] C. Vassallo, S. Panichella, F. Palomba, S. Proksch, H. C. Gall, and A. Zaidman (2020) How developers engage with static analysis tools in different contexts. EMSE 25 (2), pp. 1419–1457. Cited by: §III-E.
  • [50] X. Wang, L. Zhao, Y. Wang, and J. Sun (2014) The Role of Requirements Engineering Practices in Agile Development: An Empirical Study. RE 432, pp. 195–209. Cited by: §III-B.
  • [51] X. Xia, Z. Wan, P. S. Kochhar, and D. Lo (2019) How Practitioners Perceive Coding Proficiency. In Proc. of the ICSE, pp. 924–935. Cited by: §IV-B.
  • [52] J. Zhi, V. Garousi-Yusifoğlu, B. Sun, G. Garousi, S. Shahnewaz, and G. Ruhe (2015) Cost, benefits and quality of software development documentation: a systematic mapping. JSS 99, pp. 175–198. Cited by: §I, §II, §III-B.
  • [53] T. Zimmermann, R. Premraj, N. Bettenburg, S. Just, A. Schroter, and C. Weiss (2010) What Makes a Good Bug Report?. TSE 36 (5), pp. 618–643. Cited by: 7th item, §VI.