Mining Shape of Expertise: A Novel Approach Based on Convolutional Neural Network

04/05/2020 ∙ by Mahdi Dehghan, et al. ∙ University of Zanjan, Zanjan, Iran VNU Shahid Beheshti University 0

Expert finding addresses the task of retrieving and ranking talented people on the subject of user query. It is a practical issue in the Community Question Answering networks. Recruiters looking for knowledgeable people for their job positions are the most important clients of expert finding systems. In addition to employee expertise, the cost of hiring new staff is another significant concern for organizations. An efficient solution to cope with this concern is to hire T-shaped experts that are cost-effective. In this study, we have proposed a new deep model for T-shaped experts finding based on Convolutional Neural Networks. The proposed model tries to match queries and users by extracting local and position-invariant features from their corresponding documents. In other words, it detects users' shape of expertise by learning patterns from documents of users and queries simultaneously. The proposed model contains two parallel CNN's that extract latent vectors of users and queries based on their corresponding documents and join them together in the last layer to match queries with users. Experiments on a large subset of Stack Overflow documents indicate the effectiveness of the proposed method against baselines in terms of NDCG, MRR, and ERR evaluation metrics.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Community Question Answering (CQA) networks such as Stack Overflow111stackoverflow.com, Quora222quora.com, and Yahoo! Answers333answer.yahoo.com have gained much popularity and interests among people due to providing a valuable interface to share or exchange knowledge and information. This growing popularity makes people contribute more in such networks to discuss their questions or respond to other users’ questions. In addition, people can vote up or down questions or answers to specify their importance and quality in such networks. Also, there is an ability for the questioner to determine the best answer of its question in some CQAs. The huge source of users’ generated information in CQA networks, have attracted attention of many researchers to define challenging and practical problems such as expert finding Wang et al. (2018), question routing Mumtaz et al. (2019); Zhou et al. (2009), question classification Momtazi (2018); Mohasseb et al. (2018). Among all problems, expert finding with the aim of detecting and ranking talented people in the subject of user’s query is a well-studied problem in the field of Information Retrieval (IR). Expert finding as a well-studied problem has been investigated in other domains such as bibliographic networks Hashemi et al. (2013), microblogs and social networks Neshati et al. (2013). Finding knowledgeable people has a large number of real-world applications. Detecting experts and offering them to recruiters who seek talented people in a specific skill area is one of the most important applications of expert finding.

Stack Overflow as one of the most popular and active CQAs, provides its users a good platform to share knowledge and seek information. Figure 1 demonstrates a typical page of Stack Overflow in which important informative parts are highlighted. In Stack Overflow, the questioner has to determine one or more tags to its question. Tags indicate the skills needed to respond the question and are used as query terms in expert finding problem. As mentioned, linking talented people to companies intending to hire new staff is the most significant application of expert finding in CQA networks. Human resources are a valuable property to companies since they possess a range of skills which can benefit the companies. Recruiters prefer to hire new staff with the lowest cost. Employees with high range of expertise areas (i.e. C-shaped expert) demand high salary. Therefore, mining shape of expertise helps companies to find appropriate person with reasonable cost for their job positions. Gharebagh et al. Gharebagh et al. (2018) have categorized users of Stack Overflow into three: T-shaped, C-shaped and non-expert groups. T-shaped users have deep knowledge or skills in one skill area and a broad base of general supporting knowledge or skills. C-shaped users have deep knowledge or skills in several skill areas. Non-expert users don’t have deep knowledge or skills in any skill area.

Figure 1: A typical page of Stack Overflow.

Nowadays, agile methodology has attracted attention of many companies for software development. Mining shape of expertise helps companies to form an agile software development team with reasonable cost. Among all strategies to form an agile team, the best one forms a team with the lowest cost and the highest performance simultaneously. Consider a team in which all members are C-shaped experts or full-stack developers in different aspects of the project. Although team formation by this strategy brings about a team with high performance, it imposes much costto companies because C-shaped experts always ask for higher salaries. The best way of forming an agile team is to hire T-shaped experts based on aspects of the project. In this way, each team member covers a particular aspect of the project and have general knowledge of other aspects of the project. This will preserve the principles of modern software engineering up to date, and all team members will be involved in all aspects of the project if necessary Gharebagh et al. (2018). Therefore, T-shaped expert finding is a practical and industry-motivated problem in the field of IR Gharebagh et al. (2018); Dehghan et al. (2019).

In this paper, we have proposed a novel deep neural network based architecture to model both users expertises and skill areas jointly from their corresponding documents. In other words, we have tried to find a matching between skill areas and users semantically. The proposed method consists of two parallel Convolutional Neural Networks that are joined in the last layers. One CNN extracts user expertise vector form his documents, and the other one extracts vector for skill area from existing documents in a

specified skill area. These two parallel networks are joined in the last layer of the proposed architecture. Finally, outputs of the joint layer are connected to an output neuron for expertise shape detection.

The rest of this paper is organized as follows. Research objective and contribution of this work are highlighted in Section 2. Related works are discussed in Section 3. In Section 4, we present the proposed method for T-shaped expert finding. Details for the dataset, baseline models, evaluation measures, parameters setting, and implementation are given in Section 5. Finally, the paper concludes with conclusions and future directions in Section 6.

2 Contribution of this work

T-shaped expert finding aims to detect and rank T-shaped experts in the subject of user’s query. This practical problem has great application in forming agile software development teams. In this study, we have proposed a new deep architecture for T-shaped experts finding based on Convolutional Neural Networks. The proposed architecture tries to match queries and users by extracting local and position-invariant features from their corresponding documents. In other words, it detects users’ shape of expertise by learning patterns from documents related to users and queries simultaneously. As a consequence, it can relieve the vocabulary gap problem to some extent. Unlike the method proposed in Gharebagh et al. (2018), our proposed method can predict both C- and T-shaped ranking of experts based on user’s query. Also, we predict the shape of expertise for each user by extracting text features from his documents. Moreover, this study addresses the main issue of the method proposed in Dehghan et al. (2019) that is not feasible with large datasets. In fact, it is a common problem in many learning based expert profiling methods because all queries have to be determined in advance. To sum up, we have proposed a CNN-based model which detects different shapes of expertise by learning the latent vectors of users and queries in Stack Overflow.

3 Related Work

In this section, we first define the expert finding problem and outline some of its applications. Then, we review some prior studies on community question answering. Finally, we survey some important studies in mining shape of expertise.

3.1 Expert Finding

Expert finding that addressed the task of detecting and ranking right people in the subject of user’s query, has become an important issue in the field of IR. Over the past years, many research studies have conducted to solve this problem in different domains, each of which tried to improve the quality of ranking by dealing with a specific challenge Balog et al. (2012); Wang et al. (2018). In the past, expert finding methods were categorized into two graph-based Erten et al. (2004); Mutschke (2003); White and Smyth (2003); Zhang et al. (2007) and topic modeling- based Momtazi and Naumann (2013); Zhou et al. (2012); Li et al. (2015) groups. Over time, researchers have proposed probabilistic methods for expert finding Balog et al. (2009). Finding talented people has found many applications in different environments such as CQA networks Neshati et al. (2017); Zhou et al. (2016); Zhao et al. (2016), bibliographic networks Moreira et al. (2015); Moreira and Wichert (2013), academic Alarfaj et al. (2012); Deng et al. (2008), organizations Karimzadehgan et al. (2009). Balog et al. Balog et al. (2009) have proposed two outstanding language models for expert finding. Their language models were named candidate-based and document-based model. In candidate-based model, for each candidate expert a textual representation is built

upon his associated documents. In fact, candidate-based model represents a candiate expert as a language model, then the probability of generating query topic given this candidate (i.e. its language model) is considered as candidate’s level of expertise. In the document-based model, first, the probability of generating a user’s query (referred to as document value) is calculated for any document in the collection. Then, each candidate, according to his contribution in the document, obtains the score from that document. In the end, the candidates are sorted by their total scores.

3.2 Expert Finding in CQA Networks

Over the recent years, many research studies have been conducted to solve different challenging problems on CQA networks Wang et al. (2018); Mumtaz et al. (2019); Mohasseb et al. (2018). In CQA networks, documents are posted by registered users in the forms of questions and answers. In such networks, a questioner posts a question, one ore more answerers reply the question, and finally the questioner marks an answer as accepted answer. Also, the questioner has to tag his question with some keywords indicating the skills needed to respond to a question. CQA networks provide valuable source of information for expert finding researchers. Expert finding in CQA networks is challenging task due to query term mismatch or vocabulary gap problem Dehghan and Abin (2019a, b); Dargahi Nobari et al. (2017); Li et al. (2014).

Dargahi et al. Dargahi Nobari et al. (2017) have conducted a research study to investigate the efficiency of translation models in relieving the query term mismatch problem in CQA networks. Their translation models were named mutual information-based (referred to as MI-based) and word embedding-based (referred to as WE-based) approach. MI measure shows the relevancy of words according to their co-occurrence in a document. In the MI-based approach, they have used Normalized MI as translation probability. In their WE-based approach, they have proposed a neural network structure to model the relation between topics and queries to decrease the gap between query and words. Eventually, they have utilized query translations in a binary scoring based approach to find experts. Authors in Nobari et al. (2020) have utilized other features to improve the results. Accordingly, they have introduced the concept of vote share to determine the quality of the posts. Eventually, the posts with higher vote share would have more impact on the translation model.

Although proposed translation models in Dargahi Nobari et al. (2017); Nobari et al. (2020) can bring about an improvement in the quality of expert finding, they did not consider the diversification concept that can cause to cover more query topics and improve the quality of final expert ranking. Dehghan et al. Dehghan and Abin (2019b) have proposed a translation model taking the diversification concept into account to cope with the vocabulary gap problem in CQA networks. Their translation model is based on data clustering in two query and co-occurrence spaces. Their proposed translation model clusters all important words extracted by topic modeling in the query space at the first step and then clusters each query space cluster in the co-occurrence space again to remove semantically similar words. Then, they translate each query into a set of relevant and diverse words through a probabilistic framework. Finally, they utilize the query translations in a voting-based approach to identify experts.

Topic modeling is one of the semantic matching approaches utilized to deal with the query term mismatch problem. Momtazi et al. Momtazi and Naumann (2013) have proposed a topic modeling based approach to retrieve and rank experts on the Text Retrieval Conference (TREC) Enterprise track for 2005 and 2006. They have extracted main topics of queries by using Latent Dirichlet Allocation (LDA) Blei et al. (2003) in the first step. Then, they have utilized the topics that are extracted from their document collection as a bridge to connect candidate experts to user’s query. Van Gysel et al. Van Gysel et al. (2016) have introduced an unsupervised neural network based approach to solve an expert finding task. Their model learns latent vectors of candidate experts and queries based on the occurrence of query words in the documents posted by candidate experts.

Temporal property of expertise is another important feature that has been investigated by researchers in CQA networks. Neshati et al. Neshati et al. (2017) have introduced future expert finding problem in CQA networks, which is the task of predicting talented users in the future according to their expertise evidence at the current time. They have proposed two learning algorithms in which they have investigated the impact of four different feature groups, including topic similarity, emerging topics, user behaviour, and topic transition in the future expert finding task. Patil et al. Patil and Lee (2016) have proposed a method to detect experts in Quora. They have analysed the behaviour of both expert and non-expert users in Quora according to four feature groups, namely:

user activity, quality of answers, linguistic and temporal. Then, they have developed statistical classification models in order to detect expert people by choosing high-resolution features.

3.3 Mining Shape of Expertise

Up to now, we have discussed some previous works in expert finding that only concentrated on detecting expert people. Authors in Dehghan et al. (2019); Gharebagh et al. (2018) have proposed methods to detect the shape of expertise in CQAs networks. Gharebagh et al. Gharebagh et al. (2018) have built a dataset of Stack Overflow users in which all users are categorized into three T-shaped, C-shaped and non-expert groups according to their knowledge and skills in the extracted skill areas. Then, they have proposed a probabilistic model to find T-shaped experts. Dehghan et al. Dehghan et al. (2019) have proposed an outstanding temporal expert profiling model on Stack Overflow. They have conducted their study to prove the effect of time in expertise. In fact, they have introduced an LSTM neural network based approach to model the relation between temporal expertise trees and shape of expertise.

4 Proposed Method

In this section, the proposed method is discussed in more detail. Figure 2 shows the idea behind the proposed method. As this figure shows, the proposed method, jointly models both user expertise and query properties by using the answers in Stack Overflow. It learns latent vectors of users and queries from their corresponding answers such that the learned vectors can predict the user’s shape of expertise. This process is done by a CNN-based model consisting of two parallel convolution neural networks (CNN) joined with together in the last layer. The joint model is then trained in a supervised manner to predict the user’s level of expertise with minimum prediction error. In fact, the proposed method outputs a ranking of users for a given query based on their shape of expertise. In this work, we intend to create a ranking of T-shaped experts. There, we adjust the output neuron of the model to indicate the level of being a T-shaped expert. In other words, we intend to solve the problem of T-shaped expert finding. However, our proposed method can also propose a ranking of C-shaped experts by a slight modification in the training phase (Section 4.4). In the following, we first describe notations used in this paper. Then, the architecture of the proposed model is explained. Finally, we illustrate how to train the proposed model.

Figure 2: The proposed CNN-based approach for detecting shape of expertise.

4.1 Preliminaries

In the setting of this work, we have a collection of document (i.e. answers in Stack Overflow) , set of candidate experts , and set of queries . We let and denote set of documents posted by candidate expert , and set of documents related to query , respectively. Table 1 indicates the summary of notations used in this work. In the following, we explain the proposed method in more detail.

Notation Notation description
Query
Candidate expert
Document collection (i.e. Stack Overflow answers)
Set of all candidate experts
Set of all queries
Set of documents posted by candidate expert
Set of documents related to query
Size of document embedding vector
Convolutional neural network layer in the proposed model for candidate experts
Convolutional neural network layer in the proposed model for queries
Size of candidate latent vector (i.e. the number of output neurons of )
Size of query latent vector (i.e. the number of output neurons of )
Document embedding matrix
Maximum number of documents that have to select from and
Number of convolution filters in and
Number of rows in convolution filters

Size of feature map in max pooling kernel

Latent vector
Table 1: Summary of notations.

4.2 Problem Specification

Given query , T-shaped expert finding task is to return a ranked list of candidate experts , ,in the given query. To do so, we propose a deep architecture based on two parallel CNNs that learns latent vectors of users and queries in a supervised manner. For each query , we feed all pairs as the inputs to the proposed model. The proposed model assigns a value to each pair (i.e. ) indicating the level of being T-shaped expert. Therefor, is generated by sorting all users based on the predicted value by model.

4.3 Model Architecture

Figure 2 demonstrates the architecture of the proposed model for predicting level of being a T-shaped expert. As mentioned before, the model is composed of two parallel convolutional neural networks joined in the last layers, and for candidate experts and queries, receptively. In the model, the documents posted by candidate expert , , and the documents related to query , , are fed to and respectively as inputs, and level of being T-shaped expert is produced as the output. In the first layer, referred to as look-up embedding layer, and are represented as matrices of document embedding to preserve the semantic information in the documents. The next layers of the model are the common CNN layers consisting of convolution, pooling, and fully connected layers utilized to extract high-resolution feature vector for candidates and queries. At the top of the model, the output of and are joined together to connect latent vectors of candidates and queries. Eventually, the output layer consists of a single neuron that minimizes the level of being T-shaped expert based on latent vectors and . In the following, we explain how to extract . Extraction of is same as .

4.3.1 Look up Embedding Layers

A document embedding is a parametrized mapping of documents into -dimensional vectors. Recently, this approach has found many applications in a large number of text mining tasks Markov et al. (2016); Maslova and Potapov (2017). We utilize LDA Blei et al. (2003) to extract semantic vectors of documents. In the look up embedding layer, each candidate has a set of posted documents that is represented as a matrix of document embeddings . Since should be square matrix, we just select documents for each candidate expert. For candidates having less than

documents, we pad the end of matrix by zeroes.

The following equation illustrates how the look up embedding layer works:

(1)

To sum up, look up embedding layer receives set of documents posted by candidate experts and passes a matrix of document embedding to the next layer (i.e. CNN layer).

4.3.2 CNN Layers

Next layers consist of convolution, max pooling, and fully connected layers. A convolution layer consists of filters, , that generates new features by doing convolution operator on matrix embedding . Each vector generated by a filter is passed to the max pooling layer. The max pooling layer selects the maximum value as the most important feature of each feature map with a size of . In fact, it performs a down sampling operation along the spatial dimensions of its inputs. After max pooling layer, there is a fully connected layer that receives the results of max pooling layer. The fully connected layer produces a vector with size of indicating latent vector for a candidate expert (i.e. ). As mentioned earlier, and have the same structure. So, the latent vector of query with size of (i.e. ) are produced as the outputs of .

4.3.3 Integration Layers

Having latent vectors of candidate expert , , and query , , we intend to predict the level that the candidate expert is a T-shaped expert in . To this end, we have utilized these vectors in two different ways: 1) their dot product is connected to a single neuron and, 2) their concatenation is connected to a single neuron . Finally, the output neuron returns the level that candidate is T-shaped expert in query by using and .

4.4 Constructing Golden Set

In this section, we explain how to create training and testing datasets in order to train and evaluate the network. As mentioned, the inputs to the network are set of documents posted by a candidate expert and set of documents related to query , and the desired output is the scalar indicating the level that the candidate expert is a T-shaped expert in query . To train the network, we have to know the true value of for each input pair . Knowing the shape of each candidate expert in our dataset, we determine the true value of in the following way:

  1. If the candidate expert is a T-shaped expert in query , will be assigned to .

  2. If the candidate expert is a C-shaped expert in query , will be assigned to .

  3. If the candidate expert is a non-expert in query , will be assigned to .

By these assignments, the output neuron indicates the level that a candidate expert is a T-shaped expert in query . If we intend to predict the level of being C-shaped experts, we have to assign to the output neuron . After constructing network inputs and outputs by this way, of the available data for T-shaped, C-shaped and non-expert users are allocated for training. The remaining data are equally partitioned and referred to as validation and test data sets.

4.5 Ranking Candidate Experts

The ranking of candidate experts is the final step in the proposed method. Since indicates the level of being T-shaped expert, to find a ranking of T-shaped experts in query , all candidate experts will be ordered by their corresponding and return the ranked list of T-shaped experts in query .

5 Experimental Setup

In this section, we compare the proposed method with the baselines by using three well-known evaluation metrics in the IR community. In the following, we present details of evaluation metrics, parameters settings, and implementations.

5.1 Dataset

We use a real-world CQA dataset from Stack Overflow provided by Gharebagh et al. (2018). This dataset crawled from Stack Overflow contains all posts from August to March and consists information of about million users and posts Gharebagh et al. (2018). The dataset consisting of three separate data collections has been generated from all posts tagged with , , and . Each domain includes a number of skill areas or queries each of which consists of a number of tags. We categorized each user in one of T-shaped, C-shaped, and non-expert group based on his knowledge level. In the following, we explain the way of extracting queries for a domain and labeling users as T-shaped, C-shaped, and non-expert.

5.2 Extraction of Skill Areas

As mentioned, a query consists of a set of related tags. Gharebagh et al. Gharebagh et al. (2018) did three steps to extract queries for a particular domain. At first, they have extracted most frequent tags of the domain. In the second step, they used an agglomerative clustering algorithm to cluster the extracted tags. Finally, a number of computer industry artisans revised and refined the clusters based on the interest and need for recruitment to label them as a query. Table 2 shows some queries in Java, Android and C# domains.

 

Domain Query Tags

 

Java Spring spring, spring-mvc, spring-security
Android Push-notifications android-gcm, push-notifications, android-notifications
C# Unit-testing moq, unit, unit-testing

 

Table 2: Some skill areas or queries in Java, Android, and C# domains Dehghan et al. (2019).

5.3 Classification of users

In this section, we introduce how to classify users into one of T-shaped, C-shaped, or non-expert group by using two measures named precision and recall.

  1. : The ratio of the number of accepted answers posted by user in query to the total number of his answers in query .

  2. : The ratio of the number of accepted answers posted by user in query to the total number of accepted answers in query .

To classify users, firstly, we rank them based on theharmonic mean of precision and recall. The top of the ranked users are labelled as advanced knowledge level users. The next of them are labelled as intermediate knowledge level users. The rest of the list are considered to be beginner knowledge level users. Finally, we classify each user in one of T-shaped, C-shaped or non-expert groups based on his knowledge level by using the following rules:

  1. Non-expert user has no advanced knowledge level in any skill area.

  2. T-shaped expert user has advanced knowledge level in only one skill area and intermediate knowledge level in at least one skill area.

  3. C-shaped expert user has advanced knowledge level in more than one skill area.

Table 3 indicates the statistics of the dataset. Also, Figure 3 plots share of different expertise shapes in Java , Android and C# domains.

 

Domain #Questions #Answers #Queries
#All
users
#T-shaped
users
#C-shaped
users

 

 

Table 3: General statistics of dataset.
(a) Java (b) Android (c) C#
Figure 3: Share of different shapes of expertise in Java, Android and C# domains.

5.4 Baseline Models

In this section, we introduce five well-known methods in expert finding literature as our baselines. The first one is a document-based approach Balog et al. (2012) referred to as DBA. The second and third baselines are entropy-based approach and extended entropy-based approach referred to as EBA and XEBA, respectively Gharebagh et al. (2018). Another baseline, proposed in Dehghan et al. (2019), is a temporal expert profiling method utilizing LSTM neural network to find shape of expertise. Two different traversal approach, DFS and BFS, have been utilized to train LSTM neural network in Dehghan et al. (2019). Therefore, our forth and fifth baselines are named as LSTM-BFS and LSTM-DFS approaches in the rest of this paper.

5.5 Evaluation Metrics

In this section, we introduce three well-known evaluation metrics utilized to compare the proposed method against baselines. The first one is Normalized Discounted Cumulative Gain (NDCG). To evaluate the quality of ranking by using NDCG, we have to assign a relevancy score to each retrieved item (i.e. users). Here, we assume , and as relevancy scores for T-shaped, C-shaped, and non-expert users, respectively. The second measure is Mean Reciprocal Rank (MRR). MRR measure indicates the first occurrence of the true retrieved item in the ranking on average. The third one is Expected Reciprocal Rank (ERR) Chapelle et al. (2009) inspired by cascade models. ERR is the expected reciprocal amount of time that any user spends on finding a relevant result Dehghan et al. (2019).

5.6 Parameters Setting and Implementation Details

We have used Keras

444https://keras.io to implement the proposed neural network model. We have used Gensim Doc2vec package555https://radimrehurek.com/gensim/ that contains an implementation of doc2vec model proposed in Le and Mikolov (2014), to extract semantic feature vector with size of dimension (i.e. ) of our documents. Also, to investigate the effect of document embedding method on the efficiency of the proposed method, we have used LDA topic modelling approach Blei et al. (2003) to extract semantic feature vector of documents. In this work, the well-known MALLET666http://mallet.cs.umass.edu/ topic modelling package has been utilized to group document into topics. Number of documents that have been selected as corresponding documents to candidate expert , , and query , , has been set to (i.e. ) for Java and C# domain. For Android domain, the effect of parameter has been investigated in the quality of results. Also, we have considered two kernels, , in the both convolution neural networks (i.e. ) of the proposed model. The size of candidate and query latent vectors, and , has been set to .

5.7 Experimental Results and Analysis

In this section, we have compared the proposed method against five well-known methods in expert finding literature. To investigate the quality of ranking, we have utilized three prominent evaluation metrics, including NDCG, MRR, and ERR.

Analysis of Parameter : At first, we have conducted a set of experiments to investigate the effect of number of documents related to queries (i.e. ) on the effectiveness of the proposed method. Figures 4-(a) to (c) shows the sensitivity analysis of the proposed method in Android domain when the parameter varies. As these figures show, the proposed method doesn’t perform well for small values of because of the lack of expertise evidence in small amount of documents. Although the proposed method performs better for bigger values of , the model performance for small values of is better than some baselines. Another noteworthy point is that by increasing the value of the quality of results increases unless for large values of . It can be explained by that the high increase in the value of makes the inputs of CNN layers to be sparse and decreases the quality of results consequently.

(a) NDCG
(b) ERR
(c) MRR
Figure 4: Sensitivity analysis of the proposed method in Android domain when the parameter varies.

Analyzing the Effect of Document Embedding: In this section, we have investigated the effect of two different document embedding approaches proposed in Le and Mikolov (2014); Blei et al. (2003) on the quality of results. Tables 4, 5 and 6 compares the quality of ranking for two different document embedding approaches against baselines on three Java, Android, and C# domains, respectively. As these tables show, our proposed method has greatly improved the quality of ranking in all three domains. The most important reason behind this improvement is extracting text features from documents by CNN to match users and queries semantically. Also, our proposed method detects more T-shaped experts in top of the ranked list than baselines for two reasons. First, it performs better than baselines in terms of MRR evaluation metric. Second, it improves baselines in terms of our evaluation metrics for the small values of . To better understand the effect of parameter in the quality of ranking, we have compared the proposed method against baselines in Figures 5, 6, and 7.

 

Baselines Proposed
Metric

DBABalog et al. (2012)

EBAGharebagh et al. (2018)

XEBAGharebagh et al. (2018)

LSTM-BFSDehghan et al. (2019)

LSTM-DFSDehghan et al. (2019)

Doc2vec

LDA

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

Table 4: Comparison of the proposed methods against baselines in Java domain.

 

BaseLines Proposed
Metric

DBABalog et al. (2012)

EBAGharebagh et al. (2018)

XEBAGharebagh et al. (2018)

LSTM-BFSDehghan et al. (2019)

LSTM-DFSDehghan et al. (2019)

Doc2vec

LDA

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

Table 5: Comparison of the proposed methods against baselines in Android domain.

 

BaseLines Proposed
Metric

DBABalog et al. (2012)

EBAGharebagh et al. (2018)

XEBAGharebagh et al. (2018)

LSTM-BFSDehghan et al. (2019)

LSTM-DFSDehghan et al. (2019)

Doc2vec

LDA

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

NDCG@R
ERR@R
MRR@R

 

Table 6: Comparison of the proposed methods against baselines in C# domain.
(a) NDCG
(b) ERR
(c) MRR
Figure 5: Sensitivity analysis of the proposed method in Java domain when the parameter varies.
(a) NDCG
(b) ERR
(c) MRR
Figure 6: Sensitivity analysis of the proposed method in Android domain when the parameter varies.
(a) NDCG
(b) ERR
(c) MRR
Figure 7: Sensitivity analysis of the proposed method in C# domain when the parameter varies.

Statistical comparison: Finally, we have conducted a paired test with significance level . We have provided statistical comparison of the proposed method against baselines by using a paired test in Table 7.

 

Domain Metric Paired test

 

NDCG@100 EBADBAXEBALSTM-BFSLSTM-DFSP-Doc2vecP-LDA
Java ERR@100 DBAEBAXEBALSTM-BFSP-Doc2vecLSTM-DFSP-LDA
MRR@100 DBAEBAXEBAP-Doc2vecLSTM-DFSLSTM-BFSP-LDA
NDCG@100 DBAEBAXEBALSTM-BFSLSTM-DFSP-Doc2vecP-LDA
Android ERR@100 DBAEBAXEBALSTM-BFSP-Doc2vecLSTM-DFSP-LDA
MRR@100 DBAEBAXEBAP-Doc2vecLSTM-BFSLSTM-DFSP-LDA
NDCG@100 DBAEBAXEBALSTM-BFSLSTM-DFSP-Doc2vecP-LDA
C# ERR@100 DBAEBAXEBALSTM-DFSLSTM-BFSP-Doc2vecP-LDA
MRR@100 DBAEBAXEBALSTM-DFSLSTM-BFSP-Doc2vecP-LDA
P in P-Doc2vec and P-LDA stands for the proposed method.

 

Table 7: Statistical comparison of the proposed method against baselines using paired test. stands for that two methods and does not differ significantly.

6 Conclusion

Community Question Answering networks provide valuable platforms for user to exchange knowledge or information. Stack Overflow network is an outstanding CQA providing a huge source of knowledge that can be utilized by recruiters to hire new staff. Hiring new staff with reasonable cost is a vital concern of many companies. Forming a team consisting of T-shaped experts for software development is an excellent solution that optimizes company costs. This work proposed a new model for T-shaped expert finding that is based on convolutional neural networks. We proposed a model for T-shaped expert finding that learns latent vectors of users and queries from their corresponding documents simultaneously. Then, these latent vectors are utilized to match users and queries semantically. T-shaped experts grouping is a future direction to the proposed method.

Acknowledgement

This research is funded by Vietnam National University, Hanoi (VNU) under project number QG.18.40.

References

References

  • F. Alarfaj, U. Kruschwitz, D. Hunter, and C. Fox (2012) Finding the right supervisor: expert-finding in a university domain. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: human language technologies: student research workshop, pp. 1–6. Cited by: §3.1.
  • K. Balog, L. Azzopardi, and M. de Rijke (2009) A language modeling framework for expert finding. Information Processing & Management 45 (1), pp. 1–19. Cited by: §3.1.
  • K. Balog, Y. Fang, M. de Rijke, P. Serdyukov, L. Si, et al. (2012) Expertise retrieval. Foundations and Trends® in Information Retrieval 6 (2–3), pp. 127–256. Cited by: §3.1, §5.4, Table 4, Table 5, Table 6.
  • D. M. Blei, A. Y. Ng, and M. I. Jordan (2003) Latent dirichlet allocation.

    Journal of machine Learning research

    3 (Jan), pp. 993–1022.
    Cited by: §3.2, §4.3.1, §5.6, §5.7.
  • O. Chapelle, D. Metlzer, Y. Zhang, and P. Grinspan (2009) Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM conference on Information and knowledge management, pp. 621–630. Cited by: §5.5.
  • A. Dargahi Nobari, S. Sotudeh Gharebagh, and M. Neshati (2017) Skill translation models in expert finding. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1057–1060. Cited by: §3.2, §3.2, §3.2.
  • M. Dehghan and A. A. Abin (2019a) Retrieve and rank the experts using a cluster-based translation model. Tabriz Journal of Electrical Engineering 49 (3), pp. 1095–1106. Cited by: §3.2.
  • M. Dehghan and A. A. Abin (2019b) Translations diversification for expert finding: a novel clustering-based approach. ACM Transactions on Knowledge Discovery from Data (TKDD) 13 (3), pp. 32. Cited by: §3.2, §3.2.
  • M. Dehghan, M. Biabani, and A. A. Abin (2019) Temporal expert profiling: with an application to t-shaped expert finding. Information Processing & Management 56 (3), pp. 1067–1079. Cited by: §1, §2, §3.3, §5.4, §5.5, Table 2, Table 4, Table 5, Table 6.
  • H. Deng, I. King, and M. R. Lyu (2008) Formal models for expert finding on dblp bibliography data. In 2008 Eighth IEEE International Conference on Data Mining, pp. 163–172. Cited by: §3.1.
  • C. Erten, P. J. Harding, S. G. Kobourov, K. Wampler, and G. Yee (2004) Exploring the computing literature using temporal graph visualization. In Visualization and Data Analysis 2004, Vol. 5295, pp. 45–57. Cited by: §3.1.
  • S. S. Gharebagh, P. Rostami, and M. Neshati (2018) T-shaped mining: a novel approach to talent finding for agile software teams. In European Conference on Information Retrieval, pp. 411–423. Cited by: §1, §1, §2, §3.3, §5.1, §5.2, §5.4, Table 4, Table 5, Table 6.
  • S. H. Hashemi, M. Neshati, and H. Beigy (2013) Expertise retrieval in bibliographic network: a topic dominance learning approach. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management, pp. 1117–1126. Cited by: §1.
  • M. Karimzadehgan, R. W. White, and M. Richardson (2009) Enhancing expert finding using organizational hierarchies. In European conference on information retrieval, pp. 177–188. Cited by: §3.1.
  • Q. Le and T. Mikolov (2014) Distributed representations of sentences and documents. In International conference on machine learning, pp. 1188–1196. Cited by: §5.6, §5.7.
  • H. Li, S. Jin, and L. Shudong (2015) A hybrid model for experts finding in community question answering. In Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), 2015 International Conference on, pp. 176–185. Cited by: §3.1.
  • H. Li, J. Xu, et al. (2014) Semantic matching in search. Foundations and Trends® in Information Retrieval 7 (5), pp. 343–469. Cited by: §3.2.
  • I. Markov, H. Gómez-Adorno, J. Posadas-Durán, G. Sidorov, and A. Gelbukh (2016) Author profiling with doc2vec neural network-based document embeddings. In

    Mexican International Conference on Artificial Intelligence

    ,
    pp. 117–131. Cited by: §4.3.1.
  • N. Maslova and V. Potapov (2017)

    Neural network doc2vec in automated sentiment analysis for short informal texts

    .
    In International Conference on Speech and Computer, pp. 546–554. Cited by: §4.3.1.
  • A. Mohasseb, M. Bader-El-Den, and M. Cocea (2018) Question categorization and classification using grammar based approach. Information Processing & Management 54 (6), pp. 1228–1243. Cited by: §1, §3.2.
  • S. Momtazi and F. Naumann (2013) Topic modeling for expert finding using latent dirichlet allocation. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 3 (5), pp. 346–353. Cited by: §3.1, §3.2.
  • S. Momtazi (2018) Unsupervised latent dirichlet allocation for supervised question classification. Information Processing & Management 54 (3), pp. 380–393. Cited by: §1.
  • C. Moreira, P. Calado, and B. Martins (2015) Learning to rank academic experts in the dblp dataset. Expert Systems 32 (4), pp. 477–493. Cited by: §3.1.
  • C. Moreira and A. Wichert (2013) Finding academic experts on a multisensor approach using shannon’s entropy. Expert Systems with Applications 40 (14), pp. 5740–5754. Cited by: §3.1.
  • S. Mumtaz, C. Rodriguez, and B. Benatallah (2019) Expert2Vec: experts representation in community question answering for question routing. In International Conference on Advanced Information Systems Engineering, pp. 213–229. Cited by: §1, §3.2.
  • P. Mutschke (2003) Mining networks and central entities in digital libraries. a graph theoretic approach applied to co-author networks. In International Symposium on Intelligent Data Analysis, pp. 155–166. Cited by: §3.1.
  • M. Neshati, E. Asgari, D. Hiemstra, and H. Beigy (2013) A joint classification method to integrate scientific and social networks. In European Conference on Information Retrieval, pp. 122–133. Cited by: §1.
  • M. Neshati, Z. Fallahnejad, and H. Beigy (2017) On dynamicity of expert finding in community question answering. Information Processing & Management 53 (5), pp. 1026–1042. Cited by: §3.1, §3.2.
  • A. D. Nobari, M. Neshati, and S. S. Gharebagh (2020) Quality-aware skill translation models for expert finding on stackoverflow. Information Systems 87, pp. 101413. Cited by: §3.2, §3.2.
  • S. Patil and K. Lee (2016) Detecting experts on quora: by their activity, quality of answers, linguistic characteristics and temporal behaviors. Social network analysis and mining 6 (1), pp. 5. Cited by: §3.2.
  • C. Van Gysel, M. de Rijke, and M. Worring (2016) Unsupervised, efficient and semantic expertise retrieval. In Proceedings of the 25th International Conference on World Wide Web, pp. 1069–1079. Cited by: §3.2.
  • X. Wang, C. Huang, L. Yao, B. Benatallah, and M. Dong (2018) A survey on expert recommendation in community question answering. Journal of Computer Science and Technology 33 (4), pp. 625–653. Cited by: §1, §3.1, §3.2.
  • S. White and P. Smyth (2003)

    Algorithms for estimating relative importance in networks

    .
    In Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 266–275. Cited by: §3.1.
  • J. Zhang, M. S. Ackerman, and L. Adamic (2007) Expertise networks in online communities: structure and algorithms. In Proceedings of the 16th international conference on World Wide Web, pp. 221–230. Cited by: §3.1.
  • Z. Zhao, Q. Yang, D. Cai, X. He, and Y. Zhuang (2016) Expert finding for community-based question answering via ranking metric network learning.. In IJCAI, pp. 3000–3006. Cited by: §3.1.
  • G. Zhou, S. Lai, K. Liu, and J. Zhao (2012) Topic-sensitive probabilistic model for expert finding in question answer communities. In Proceedings of the 21st ACM international conference on Information and knowledge management, pp. 1662–1666. Cited by: §3.1.
  • G. Zhou, Y. Zhou, T. He, and W. Wu (2016) Learning semantic representation with neural networks for community question answering retrieval. Knowledge-Based Systems 93, pp. 75–83. Cited by: §3.1.
  • Y. Zhou, G. Cong, B. Cui, C. S. Jensen, and J. Yao (2009) Routing questions to the right users in online communities. In IEEE international conference on data engineering, pp. 700–711. Cited by: §1.