DeepVS: An Efficient and Generic Approach for Source Code Modeling Usage

10/15/2019 ∙ by Yasir Hussain, et al. ∙ 27

Recently deep learning-based approaches have shown great potential in the modeling of source code for various software engineering tasks. These techniques lack adequate generalization and resistance to acclimate the use of such models in a real-world software development environment. In this work, we propose a novel general framework that combines cloud computing and deep learning in an integrated development environment (IDE) to assist software developers in various source code modeling tasks. Additionally, we present DeepVS, an end-to-end deep learning-based source code suggestion tool that shows a real-world implementation of our proposed framework. The DeepVS tool is capable of providing source code suggestions instantly in an IDE by using a pre-trained source code model. Moreover, the DeepVS tool is also capable of suggesting zero-day (unseen) code tokens. The DeepVS tool illustrates the effectiveness of the proposed framework and shows how it can help to link the gap between developers and researchers.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 3

page 4

page 7

Code Repositories

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Recently, deep learning techniques have been widely applied for various source code modeling tasks such as source code suggestion Raychev et al. (2014); White et al. (2015), error fixing Gupta et al. (2018); Santos et al. (2018), method naming Alon et al. (2019), etc. Source code suggestion and completion are vital features of an integrated development environment (IDE). Software developers rely mostly on such features while developing software. Most of the modern IDE’s source code suggestion tools provide the next possible source code by leveraging the information already exists in the code editor of the IDE. Due to the limited historical data, the IDE may not be able to provide the correct suggestions or may possibly suggest irrelevant suggestions. The deep learning Raychev et al. (2014); White et al. (2015) based approaches have recently shown how they can significantly improve such tools by leveraging historical data (e.g. GitHub). However, there are various limitations and concerns:

  • Most methods require a huge amount of computation power as well as time to train such models.

  • Current techniques lack to illustrate, how they can assist software developers in a real-world software development environment.

  • The existing works present no enlightenment, how these models can be integrated into an IDE.

In this work, our goal is to enable the usage of deep learning-based models directly in an IDE for various source code modeling tasks. Here, we choose the source code suggestion task as a motivation example. In this work, first, we introduce a novel framework that helps in using machine learning or deep learning-based models directly in an IDE. Secondly, we present

DeepVS an end-to-end deep learning-based source code suggestion tool. The DeepVS tool leverages from a pre-trained deep neural language model train on over 13M code tokens for the task of source code suggestion directly in an IDE. The tool is capable of suggesting zero-day (unseen) code tokens.

This work makes the following unique contributions.

  • To the best of our knowledge, this work is the first one to propose a general framework that enables the usage of machine learning and deep learning-based source code models directly in an IDE.

  • We present DeepVS tool, end-to-end implementation of our proposed framework for source code suggestion task.

  • The quantitative evaluation with a real-world dataset and qualitative analysis of DeepVS tool verifies that the projected method is accurate and effective in predicting code suggestions.

2 Approach

The overall high-level architecture of the proposed framework is presented in Fig. 1

. The framework consists of three layers. The first layer consists of a machine learning or deep learning-based pre-trained model, which is used to estimate the likelihood scores for a given source code modeling task. The second layer is a cloud/local platform, which is used to serve pre-trained models locally or over a cloud platform for the purpose of likelihood estimation. The final layer is an IDE plugin interface to interacts with the cloud/local platform for a specific source code modeling task.

Figure 1: A general framework for usage of machine learning based source code models directly in an IDE.

the overall workflow of our DeepVs tool is presented in Fig. 2. On the first layer of DeepVS

, we are using Gated Recurrent Unit (GRU)

Cho et al. (2014)

based deep learning classifier in a bidirectional fashion which learns the source code context in both directions to help suggest the next source code token. On the second layer, we present a Cloud-based platform named

Deep Cloud, which uses the pre-trained bidirectional Gated Recurrent Unit (BiGRU) neural net for the source code suggestion task. Finally, for demonstration purpose, we implement a plugin interface in Visual Studio Code IDE, which interacts with Deep Cloud for source code suggestion task in real-time.

Figure 2: Architecture of our DeepVS tool which is a real world implementation of our proposed framework.

2.1 The Deep Cloud Layer

The Deep Cloud platform is used as a centralized deep learning model attendant to determine the likelihood scores for the next source code suggestion task. Here, we choose cloud platform Bahsoon et al. (2013) because of its centralized nature and scalability. Our Deep Cloud is a platform as a service-based cloud. Deep Cloud consists of a single core Intel CPU with 1GB Ram running ubuntu-18 x64-bit non-GUI edition. Deep cloud takes source code context as input and provides a JSON object which contains most likely next source code suggestions.

2.2 The plugin Layer

For the demonstration purpose, we implement the plugin interface in Visual studio code IDE. The DeepVS tool can be triggered by using shortcut key CTRL+SPACE. By triggering DeepVS, it takes the source code prior to the current cursor position as context and requests the Deep Cloud

for most probable next source code tokens by leveraging the pre-trained BiGRU model.

3 Evaluation

The evaluation of this work is two-fold, quantitative and qualitative. In quantitative evaluation, we assess the classification outcomes to test the performance of the proposed method and compare it with state-of-the-art methods. Metrics used to assess the quantitative efficiency of this work are top-k accuracy and mean reciprocal rank (MRR), which are mostly applied. For the qualitative evaluation of the work, we conduct an experimental study involving seven university students having software development experience. The experiment was performed with the help of a questionnaire. All the participants were demonstrated with the basic usage of the DeepVS plugin. The questionnaire given to the participants is shown in Table 2.

3.1 Dataset

For the training and testing of the proposed method, this work used the dataset anticipated in Hindle et al. (2012); Nguyen et al. (2018). The dataset comprises of ten java projects (ant, cassandra, db40, jgit, poi, batik, antlr, itext, jts, maven). The dataset consists of over 13M code tokens with large vocabulary of size 145,457 (singletons removed). Table 1 shows the statistics of our training, and testing sets in detail.

Line of Code (LoC) Total Code Tokens Vocab Size
Training 1,684,158 12,086,347
Testing 187,129 1,342,927
Total 1,871,287 13,429,274 145,457
Table 1: Code database statistics
Questions Choices
Q1: How easy it is to set up and use the DeepVS tool? (1=Very Easy, 5=Very Difficult)


Q2: Does DeepVS tool provides suggestions in real-world coding environment?
(1=Strongly Agree, 5=Strongly Disagree)


Q3: Does DeepVS tool performs better than IDE’s default source code suggestion engine?
(1=Strongly Agree, 5=Strongly Disagree)


Q4: I will recommend DeepVS to others.
(1=Strongly Agree, 5=Strongly Disagree)

Table 2: Questionnaire used for evaluation.

4 Results

4.1 Quantitative Results

In order to realistically evaluate the effectiveness of the proposed method, we compare the performance of our proposed method with other state-of-the-art methods. We choose Hindle el al. Hindle et al. (2012) and White et al. White et al. (2015) work as baselines because they are related to our target task. The comprehensive results are given in Table 3. It can be witnessed that the proposed method proves noticeably better performance as compared to other baselines. Fig. 3 and Fig. 4 represents the accuracy and loss of training and validation accordingly.

Figure 3: Training and validation accuracy comparison.
Figure 4: Training and validation loss comparison.
Model Accuracy MRR
K-1 K-3 k-5 K-10
N-gram Hindle et al. (2012) 48.47% 57.68% 59.87% 61.78% 0.535
RNN White et al. (2015) 51.30% 67.67% 72.18% 76.21% 0.596
BiGRU 73.77% 84.39% 86.97% 89.34% 0.792
Table 3: Accuracy and MRR scores comparison.

4.2 Qualitative Results

As reported in Fig. 5, participants found the DeepVS tool is easy to set up and use. In Q2, all participants have agreed that DeepVS tool is capable of providing suggestions in a real-world development environment. In Q3, out of seven participants three voted strongly agree, three of them voted agree and one voted neutral. In Q4, most of the participants have agreed that they would recommend DeepVS to their peers. The illustrative examples of source code suggestion tasks are provided in Section 4.3.

Figure 5: Questionnaire result.

4.3 DeepVS Tool Demo

To show the effectiveness of our proposed framework for real-world usage, we present DeepVS, a deep learning-based source code suggestion tool for Visual Studio Code IDE. Consider the example presented in Fig. 6 were a software developer is writing a complex program to find the transpose of a matrix. Hereby triggering (line 9) our DeepVS tool it suggest the most probable next source code tokens (Fig. (a)a) i,j and k where the Visual Studio Code’s default suggestion engine fails to provide any relevant suggestions (Fig. (b)b), instead it torment the developer with irrelevant suggestions. Later on, (Fig. 7) at line 21 the software developer is writing a print statement. Here the most probable next source code token is column based on the given context. Hereby triggering our DeepVS tool it suggests the correct next source code token column (Fig. (b)b) at its first index whereas the IDE’s default source code suggestion tool rank the correct suggestion (Fig. (a)a) on its third index.

(a)
(b)
Figure 6: Source code suggestion example with and without the DeepVs tool along with their probabilities.
(a)
(b)
Figure 7: Another Source code suggestion example along with their probabilities.

4.3.1 Proposed Work’s Major Benefits

The qualitative and quantitative evaluation verifies the feasibility and practicality of this work. We summarize the benefits of our proposed framework and the DeepVS tool as follows:

  • The proposed framework enables the usage of machine/deep learning-based source code models directly in an IDE.

  • An end-to-end deep learning-based source code suggestion tool named DeepVS shows a real-world implementation of the proposed framework, verifying its practicality.

  • The DeepVS tool is trained on over 13M code tokens, thus capable of suggesting zero-day (unseen) code tokens by leveraging large scale historical codebase.

  • The DeepVS tool ranks the suggestions efficiently by leveraging a deep learning-based source code model, without tormenting the developer with unnecessary suggestion.

5 Conclusion

In this work, we propose a novel general framework which helps in using machine or deep learning-based source code models for various software engineering tasks. Further, we presented DeepVS, an end-to-end deep learning-based source code suggestion tool, which leverages from a pre-trained deep learning model to help suggest the next source code token directly in an IDE in real-time. The DeepVS tool shows the effectiveness of the proposed framework and shows that it is of practical use.

Acknowledgments

This work was supported by the National Key R&D sponsored by Qing Lan Project of China [grant no. 2018YFB1003902].

References

  • [1] U. Alon, M. Zilberstein, O. Levy, and E. Yahav (2019)

    Code2vec: learning distributed representations of code

    .
    Proceedings of the ACM on Programming Languages 3 (POPL), pp. 40. Cited by: §1.
  • [2] R. Bahsoon, I. Mistrík, N. Ali, T. Mohan, and N. Medvidovic (2013) The future of software engineering in and for the cloud. Journal of Systems and Software 86 (9), pp. 2221–2224. Cited by: §2.1.
  • [3] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio (2014) Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. Cited by: §2.
  • [4] R. Gupta, A. Kanade, and S. Shevade (2018)

    Deep reinforcement learning for programming language correction

    .
    arXiv preprint arXiv:1801.10467. Cited by: §1.
  • [5] A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu (2012) On the naturalness of software. In 2012 34th International Conference on Software Engineering (ICSE), pp. 837–847. Cited by: §3.1, §4.1, Table 3.
  • [6] A. T. Nguyen, T. D. Nguyen, H. D. Phan, and T. N. Nguyen (2018) A deep neural network language model with contexts for source code. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 323–334. Cited by: §3.1.
  • [7] V. Raychev, M. Vechev, and E. Yahav (2014) Code completion with statistical language models. In Acm Sigplan Notices, Vol. 49, pp. 419–428. Cited by: §1.
  • [8] E. A. Santos, J. C. Campbell, D. Patel, A. Hindle, and J. N. Amaral (2018) Syntax and sensibility: using language models to detect and correct syntax errors. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 311–322. Cited by: §1.
  • [9] M. White, C. Vendome, M. Linares-Vásquez, and D. Poshyvanyk (2015) Toward deep learning software repositories. In Proceedings of the 12th Working Conference on Mining Software Repositories, pp. 334–345. Cited by: §1, §4.1, Table 3.