DeepAI AI Chat
Log In Sign Up

Can NMT Understand Me? Towards Perturbation-based Evaluation of NMT Models for Code Generation

03/29/2022
by   Pietro Liguori, et al.
UNC Charlotte
University of Naples Federico II
0

Neural Machine Translation (NMT) has reached a level of maturity to be recognized as the premier method for the translation between different languages and aroused interest in different research areas, including software engineering. A key step to validate the robustness of the NMT models consists in evaluating the performance of the models on adversarial inputs, i.e., inputs obtained from the original ones by adding small amounts of perturbation. However, when dealing with the specific task of the code generation (i.e., the generation of code starting from a description in natural language), it has not yet been defined an approach to validate the robustness of the NMT models. In this work, we address the problem by identifying a set of perturbations and metrics tailored for the robustness assessment of such models. We present a preliminary experimental evaluation, showing what type of perturbations affect the model the most and deriving useful insights for future directions.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/06/2019

Robust Neural Machine Translation with Doubly Adversarial Inputs

Neural machine translation (NMT) often suffers from the vulnerability to...
05/01/2020

Evaluating Robustness to Input Perturbations for Neural Machine Translation

Neural Machine Translation (NMT) models are sensitive to small perturbat...
12/20/2022

ReCode: Robustness Evaluation of Code Generation Models

Code generation models have achieved impressive performance. However, th...
10/07/2022

NMTSloth: Understanding and Testing Efficiency Degradation of Neural Machine Translation Systems

Neural Machine Translation (NMT) systems have received much recent atten...
04/27/2021

Shellcode_IA32: A Dataset for Automatic Shellcode Generation

We take the first step to address the task of automatically generating s...
09/01/2021

Shared Certificates for Neural Network Verification

Existing neural network verifiers compute a proof that each input is han...
04/14/2021

The Curious Case of Hallucinations in Neural Machine Translation

In this work, we study hallucinations in Neural Machine Translation (NMT...