Deep Learning for Agile Effort Estimation Have We Solved the Problem Yet?

01/14/2022
by   Vali Tawosi, et al.
0

In the last decade, several studies have proposed the use of automated techniques to estimate the effort of agile software development. In this paper we perform a close replication and extension of a seminal work proposing the use of Deep Learning for agile effort estimation (namely Deep-SE), which has set the state-of-the-art since. Specifically, we replicate three of the original research questions aiming at investigating the effectiveness of Deep-SE for both within-project and cross-project effort estimation. We benchmark Deep-SE against three baseline techniques (i.e., Random, Mean and Median effort prediction) and a previously proposed method to estimate agile software project development effort (dubbed TF/IDF-SE), as done in the original study. To this end, we use both the data from the original study and a new larger dataset of 31,960 issues, which we mined from 29 open-source projects. Using more data allows us to strengthen our confidence in the results and further mitigate the threat to the external validity of the study. We also extend the original study by investigating two additional research questions. One evaluates the accuracy of Deep-SE when the training set is augmented with issues from all other projects available in the repository at the time of estimation, and the other examines whether an expensive pre-training step used by the original Deep-SE, has any beneficial effect on its accuracy and convergence speed. The results of our replication show that Deep-SE outperforms the Median baseline estimator and TF/IDF-SE in only very few cases with statistical significance (8/42 and 9/32 cases, respectively), thus confounding previous findings on the efficacy of Deep-SE. The two additional RQs revealed that neither augmenting the training set nor pre-training Deep-SE play a role in improving its accuracy and convergence speed. ...

READ FULL TEXT
research
09/01/2022

Agile Effort Estimation: Have We Solved the Problem Yet? Insights From A Second Replication Study (GPT2SP Replication Report)

Fu and Tantithamthavorn have recently proposed GPT2SP, a Transformer-bas...
research
02/02/2022

A Versatile Dataset of Agile Open Source Software Projects

Agile software development is nowadays a widely adopted practise in both...
research
05/10/2018

Human Capital in Software Engineering: A Systematic Mapping of Reconceptualized Human Aspect Studies

The human capital invested into software development plays a vital role ...
research
06/22/2022

Heterogeneous Graph Neural Networks for Software Effort Estimation

Software effort can be measured by story point [35]. Current approaches ...
research
12/09/2019

Sequential Model Optimization for Software Process Control

Many methods have been proposed to estimate how much effort is required ...
research
09/02/2016

A deep learning model for estimating story points

Although there has been substantial research in software analytics for e...
research
01/06/2019

A conversation around the analysis of the SiP effort estimation dataset

The analysis of over ten years of commercial development using Agile (10...

Please sign up or login with your details

Forgot password? Click here to reset