PUMiner: Mining Security Posts from Developer Question and Answer Websites with PU Learning

03/08/2020
by   Triet H. M. Le, et al.
0

Security is an increasing concern in software development. Developer Question and Answer (Q A) websites provide a large amount of security discussion. Existing studies have used human-defined rules to mine security discussions, but these works still miss many posts, which may lead to an incomplete analysis of the security practices reported on Q A websites. Traditional supervised Machine Learning methods can automate the mining process; however, the required negative (non-security) class is too expensive to obtain. We propose a novel learning framework, PUMiner, to automatically mine security posts from Q A websites. PUMiner builds a context-aware embedding model to extract features of the posts, and then develops a two-stage PU model to identify security content using the labelled Positive and Unlabelled posts. We evaluate PUMiner on more than 17.2 million posts on Stack Overflow and 52,611 posts on Security StackExchange. We show that PUMiner is effective with the validation performance of at least 0.85 across all model configurations. Moreover, Matthews Correlation Coefficient (MCC) of PUMiner is 0.906, 0.534 and 0.084 points higher than one-class SVM, positive-similarity filtering, and one-stage PU models on unseen testing posts, respectively. PUMiner also performs well with an MCC of 0.745 for scenarios where string matching totally fails. Even when the ratio of the labelled positive posts to the unlabelled ones is only 1:100, PUMiner still achieves a strong MCC of 0.65, which is 160 fully-supervised learning. Using PUMiner, we provide the largest and up-to-date security content on Q A websites for practitioners and researchers.

READ FULL TEXT
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

10/10/2020

Broken External Links on Stack Overflow

Stack Overflow hosts valuable programming-related knowledge with 11,926,...
08/10/2020

Demystifying the Mysteries of Security Vulnerability Discussions on Developer Q A Sites

Detection and mitigation of Security Vulnerabilities (SVs) are integral ...
05/30/2020

Improving Quality of a Post's Set of Answers in Stack Overflow

Community Question Answering platforms such as Stack Overflow help a wid...
11/02/2018

Performance Evaluation of Shared Hosting Security Methods

Shared hosting is a kind of web hosting in which multiple websites resid...
01/04/2019

How Reliable is the Crowdsourced Knowledge of Security Implementation?

Stack Overflow (SO) is the most popular online Q&A site for developers t...
03/20/2018

SOTorrent: Reconstructing and Analyzing the Evolution of Stack Overflow Posts

Stack Overflow (SO) is the most popular question-and-answer website for ...
10/03/2018

A Puff of Steem: Security Analysis of Decentralized Content Curation

Decentralized content curation is the process through which uploaded pos...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.