On the Lack of Consensus Among Technical Debt Detection Tools

03/08/2021
by   Jason Lefever, et al.
0

A vigorous and growing set of technical debt analysis tools have been developed in recent years – both research tools and industrial products – such as Structure 101, SonarQube, and DV8. Each of these tools identifies problematic files using their own definitions and measures. But to what extent do these tools agree with each other in terms of the files that they identify as problematic? If the top-ranked files reported by these tools are largely consistent, then we can be confident in using any of these tools. Otherwise, a problem of accuracy arises. In this paper, we report the results of an empirical study analyzing 10 projects using multiple tools. Our results show that: 1) these tools report very different results even for the most common measures, such as size, complexity, file cycles, and package cycles. 2) These tools also differ dramatically in terms of the set of problematic files they identify, since each implements its own definitions of "problematic". After normalizing by size, the most problematic file sets that the tools identify barely overlap. 3) Our results show that code-based measures, other than size and complexity, do not even moderately correlate with a file's change-proneness or error-proneness. In contrast, co-change-related measures performed better. Our results suggest that, to identify files with true technical debt – those that experience excessive changes or bugs – co-change information must be considered. Code-based measures are largely ineffective at pinpointing true debt. Finally, this study reveals the need for the community to create benchmarks and data sets to assess the accuracy of software analysis tools in terms of commonly used measures.

READ FULL TEXT

page 1

page 7

research
06/12/2023

Assessing the Impact of File Ordering Strategies on Code Review Process

Popular modern code review tools (e.g. Gerrit and GitHub) sort files in ...
research
02/17/2023

Towards the Assisted Decomposition of Large-Active Files

Tightly coupled and interdependent systems inhibit productivity by requi...
research
03/19/2018

Automated Localization for Unreproducible Builds

Reproducibility is the ability of recreating identical binaries under pr...
research
05/28/2020

An Empirical Study of Bots in Software Development – Characteristics and Challenges from a Practitioner's Perspective

Software engineering bots - automated tools that handle tedious tasks - ...
research
05/17/2021

In Search of Socio-Technical Congruence: A Large-Scale Longitudinal Study

We report on a large-scale empirical study investigating the relevance o...
research
02/27/2022

Technical Debt and Maintainability: How do tools measure it?

The technical state of software, i.e., its technical debt (TD) and maint...
research
10/11/2020

Further Investigation of the Survivability of Code Technical Debt Items

Context: Technical Debt (TD) discusses the negative impact of sub-optima...

Please sign up or login with your details

Forgot password? Click here to reset