Log In Sign Up

Evaluation of Generalizability of Neural Program Analyzers under Semantic-Preserving Transformations

by   Md Rafiqul Islam Rabin, et al.

The abundance of publicly available source code repositories, in conjunction with the advances in neural networks, has enabled data-driven approaches to program analysis. These approaches, called neural program analyzers, use neural networks to extract patterns in the programs for tasks ranging from development productivity to program reasoning. Despite the growing popularity of neural program analyzers, the extent to which their results are generalizable is unknown. In this paper, we perform a large-scale evaluation of the generalizability of two popular neural program analyzers using seven semantically-equivalent transformations of programs. Our results caution that in many cases the neural program analyzers fail to generalize well, sometimes to programs with negligible textual differences. The results provide the initial stepping stones for quantifying robustness in neural program analyzers.


COSET: A Benchmark for Evaluating Neural Program Embeddings

Neural program embedding can be helpful in analyzing large software, a t...

Towards Neural Functional Program Evaluation

This paper explores the capabilities of current transformer-based langua...

Source Code Optimization using Equivalent Mutants

A mutant is a program obtained by syntactically modifying a program's so...

A Generative Neural Network Framework for Automated Software Testing

Search Based Software Testing (SBST) is a popular automated testing tech...

An iterative approach to precondition inference using constrained Horn clauses

We present a method for automatic inference of conditions on the initial...

Cobra: A Framework for Cost Based Rewriting of Database Applications

Database applications are typically written using a mixture of imperativ...