On the Evaluation of NLP-based Models for Software Engineering

03/31/2022
by   Maliheh Izadi, et al.
0

NLP-based models have been increasingly incorporated to address SE problems. These models are either employed in the SE domain with little to no change, or they are greatly tailored to source code and its unique characteristics. Many of these approaches are considered to be outperforming or complementing existing solutions. However, an important question arises here: "Are these models evaluated fairly and consistently in the SE community?". To answer this question, we reviewed how NLP-based models for SE problems are being evaluated by researchers. The findings indicate that currently there is no consistent and widely-accepted protocol for the evaluation of these models. While different aspects of the same task are being assessed in different studies, metrics are defined based on custom choices, rather than a system, and finally, answers are collected and interpreted case by case. Consequently, there is a dire need to provide a methodological way of evaluating NLP-based models to have a consistent assessment and preserve the possibility of fair and efficient comparison.

READ FULL TEXT

page 1

page 2

page 3

research
05/13/2018

Deep Learning in Software Engineering

Recent years, deep learning is increasingly prevalent in the field of So...
research
07/27/2020

Case Survey Studies in Software Engineering Research

Background: Given the social aspects of Software Engineering (SE), in th...
research
02/20/2020

How to Evaluate Solutions in Pareto-based Search-Based Software Engineering? A Critical Review and Methodological Guidance

With modern requirements, there is an increasing tendancy of considering...
research
08/21/2023

Large Language Models for Software Engineering: A Systematic Literature Review

Large Language Models (LLMs) have significantly impacted numerous domain...
research
12/01/2021

BERT_SE: A Pre-trained Language Representation Model for Software Engineering

The application of Natural Language Processing (NLP) has achieved a high...
research
05/28/2020

What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP

SemEval is the primary venue in the NLP community for the proposal of ne...
research
08/18/2021

Towards Mapping Control Theory and Software Engineering Properties using Specification Patterns

A traditional approach to realize self-adaptation in software engineerin...

Please sign up or login with your details

Forgot password? Click here to reset