A Search-Based Testing Framework for Deep Neural Networks of Source Code Embedding

01/20/2021
by   Maryam Vahdat Pour, et al.
0

Over the past few years, deep neural networks (DNNs) have been continuously expanding their real-world applications for source code processing tasks across the software engineering domain, e.g., clone detection, code search, comment generation. Although quite a few recent works have been performed on testing of DNNs in the context of image and speech processing, limited progress has been achieved so far on DNN testing in the context of source code processing, that exhibits rather unique characteristics and challenges. In this paper, we propose a search-based testing framework for DNNs of source code embedding and its downstream processing tasks like Code Search. To generate new test inputs, we adopt popular source code refactoring tools to generate the semantically equivalent variants. For more effective testing, we leverage the DNN mutation testing to guide the testing direction. To demonstrate the usefulness of our technique, we perform a large-scale evaluation on popular DNNs of source code processing based on multiple state-of-the-art code embedding methods (i.e., Code2vec, Code2seq and CodeBERT). The testing results show that our generated adversarial samples can on average reduce the performance of these DNNs from 5.41 retraining the DNNs with our generated adversarial samples, the robustness of DNN can improve by 23.05 adversarial test generation strategy has the least negative impact (median of 3.56 other methods.

READ FULL TEXT
research
07/31/2021

Adversarial Robustness of Deep Code Comment Generation

Deep neural networks (DNNs) have shown remarkable performance in a varie...
research
06/17/2021

CoCoFuzzing: Testing Neural Code Models with Coverage-Guided Fuzzing

Deep learning-based code processing models have shown good performance f...
research
04/04/2023

CGDTest: A Constrained Gradient Descent Algorithm for Testing Neural Networks

In this paper, we propose a new Deep Neural Network (DNN) testing algori...
research
05/01/2020

Computing the Testing Error without a Testing Set

Deep Neural Networks (DNNs) have revolutionized computer vision. We now ...
research
02/25/2021

Statistical Testing for Efficient Out of Distribution Detection in Deep Neural Networks

Commonly, Deep Neural Networks (DNNs) generalize well on samples drawn f...
research
12/11/2020

Automatic Test Suite Generation for Key-points Detection DNNs Using Many-Objective Search

Automatically detecting the positions of key-points (e.g., facial key-po...
research
12/02/2022

SimpleMind adds thinking to deep neural networks

Deep neural networks (DNNs) detect patterns in data and have shown versa...

Please sign up or login with your details

Forgot password? Click here to reset