Anchoring Code Understandability Evaluations Through Task Descriptions

03/25/2022
by   Marvin Wyrich, et al.
0

In code comprehension experiments, participants are usually told at the beginning what kind of code comprehension task to expect. Describing experiment scenarios and experimental tasks will influence participants in ways that are sometimes hard to predict and control. In particular, describing or even mentioning the difficulty of a code comprehension task might anchor participants and their perception of the task itself. In this study, we investigated in a randomized, controlled experiment with 256 participants (50 software professionals and 206 computer science students) whether a hint about the difficulty of the code to be understood in a task description anchors participants in their own code comprehensibility ratings. Subjective code evaluations are a commonly used measure for how well a developer in a code comprehension study understood code. Accordingly, it is important to understand how robust these measures are to cognitive biases such as the anchoring effect. Our results show that participants are significantly influenced by the initial scenario description in their assessment of code comprehensibility. An initial hint of hard to understand code leads participants to assess the code as harder to understand than participants who received no hint or a hint of easy to understand code. This affects students and professionals alike. We discuss examples of design decisions and contextual factors in the conduct of code comprehension experiments that can induce an anchoring effect, and recommend the use of more robust comprehension measures in code comprehension studies to enhance the validity of results.

READ FULL TEXT
research
09/28/2021

Code Comprehension Confounders: A Study of Intelligence and Personal

Literature and intuition suggest that a developer's intelligence and per...
research
03/08/2019

A Replication Study on Code Comprehension and Expertise using Lightweight Biometric Sensors

Code comprehension has been recently investigated from physiological and...
research
05/12/2023

Do RESTful API Design Rules Have an Impact on the Understandability of Web APIs? A Web-Based Experiment with API Descriptions

Context: Web APIs are one of the most used ways to expose application fu...
research
03/15/2021

Considerations and Pitfalls in Controlled Experiments on Code Comprehension

Understanding program code is a complicated endeavor. As such, myriad di...
research
04/11/2023

Program Comprehension Does Not Primarily Rely On the Language Centers of the Human Brain

Our goal is to identify brain regions involved in comprehending computer...
research
08/20/2021

Readability and Understandability Scores for Snippet Assessment: an Exploratory Study

Code search engines usually use readability feature to rank code snippet...
research
05/11/2023

Taking Advice from ChatGPT

A growing literature studies how humans incorporate advice from algorith...

Please sign up or login with your details

Forgot password? Click here to reset