. ANNs consist of a number of interconnected processing units, wherein each unit performs a weighted sum followed by the evaluation of a given activation functionKnoblauch2013Neural . ANNs has the ability of self-learning, associative storage capabilities and high-speed search for optimal solution D2002Behavioral
The concept of deep learning originates from the study of artificial neural networks. Multi-layer perceptron with multiple hidden layers is a deep learning model. The deep learning can achieve the feature selection and organization of high dimensional data, and update the model parameters dynamically according to the feedback. It can adaptively detect deceptive opinions.
Deceptive opinions detection is an important application of deep learning model. The existence of deceptive opinions makes customers who are lack of relevant experience difficult to obtain accurate judgments of the reviewed products and buy the appropriate products. To achieve effective deceptive opinions detection, representative training data sets are highly desired. There are two types of training data sets, the constructed the data sets based on the semantic or polarity analysis of on-line opinions, the true data sets of user opinions[25,26,28], which include opinion texts[13,14,15,16], behaviors of users or between users[10,11,16]. In addition, the inputed options are classified by support vector machine and other machine learning methods. On-line opinions are short texts, varied in types and content, these existing approaches cannot adapt to various short texts and detect deceptive opinions with high accuracy. In order to achieve effective deceptive opinions identification, we need to adapt all relevant features of deceptive opinions to design a comprehensive deep learning model of deceptive opinions identification.
Considering the sparse and various expression of text opinions, we first introduce the text word order into the process of the deceptive opinion analysis. In this way, we expand the characteristic dimension of our deceptive opinions model and proposes a novel word order-preserving pooling layer, which is additionally embedded in the existing CNN (Convolutional Neural Network) model to improve the deceptive opinions detection effectively.
Contributions. The main contributions are as follows:
Since on-line opinions are short and various in forms, this paper introduces a novel feature of opinion texts, the word order of opinions, and proposes a word order-preserving CNN model, which preserves the word order characteristics in the process of opinion text feature analysis.
We implement our word order-preserving CNN (OPCNN) model on an open source deep learning platform, TensorFlow, and demonstrate that compared with basic CNN model, OPCNN can achieve more accurate detection for deceptive opinions.
2 Background and Related Work
2.1 CNN Model and Its Applications
The neural network model is connected with a large number of neurons to form a complex network system with adaptive and self-learning ability, and suitable for dealing with the unclear inherent characteristics of the data. As a new class of neural network model, the deep learning model can be used to learn the characteristics of various real things from large-scale data sets, and these features can be directly applied to various computing models by the computer.
CNN (Convolutional Neural Network), is a type of deep learning models and a research hotspot in recent years. CNN has a good fault tolerance, parallel processing and self-learning ability Wei2013Study and is widely used in image processing, speech recognition, natural language processing and other fields, and has been widely used in the text classification. Compared with other popular neural networks RNN 2015rnn
(Recurrent Neural Networks), the results of the analysis in the field of text classification are similar. Moreover, due to the opinion is generally a short sentence text, convolution function of the overall structure of the sentence has a general ability, which makes CNN in dealing with short text when the accuracy rate slightly better. Compared with the RNN, CNN’s training time is shorter, more efficient, to save time costs. Due to the short length of the deceptive opinions, compact structure, and independently expressing the meaning of the characteristics of short text analysis task, it is possible for CNN to deal with deceptive opinions detection.
2.2 Related Work
Several researches on deceptive opinions detection have been proposed. Jindal and Liu Jindal2008Opinion first studied deceptive opinions problem and trained models using features based on the opinion content, user, and the product itself. Myle Ott et al. ott2011finding created a benchmark dataset by employing Turkers to write fake opinions. Fei et al. Fei2013Exploiting proposed that a large number of opinions made use of a sudden burst either caused by the sudden popularity of the product or by a sudden invasion of a large number of fake opinions, including some of the features of real users. Markov Random Field (MRF) was used to construct users and their co-occurrence in emergencies by establishing a network for critics in different periods of emergency. Finally, Belief Propagation(BP) was used to infer whether a user is a fake user or not.Wang et al. Wang2015Semantic proposed an innovative heterogeneous opinion graph model to capture the relationship between the users and users’ opinions on the shop, and used the interaction and the role of the nodes in the figure to reveal the causes of deceptive opinion, and designed an iterative algorithm to identify deceptive opinions. Mukherjee Mukherjee2013What et al. found that more than 70% of deceptive opinion publishers issued opinions between the similarity is greater than 0.3, and real opinion publishers published opinions similarity between less than 0.18 in the Yelp data set. The content similarity calculation for the opinions made by the same commentator can reflect the characteristics of the opinion’s behavior.
There have been some studies using deep learning models to identify deceptive opinions. Raymond Lau2012Text team builds a semantic language model to identify semantic repetitive opinions and makes deceptive opinions detection. However, due to the opinion itself has a certain degree of semantic similarity and content on the repeatability, there may be a miscarriage of justice. Li et al. Li2015Learning took the word vector as input, with CNN, the emotional polarity feature can also be applied to unsupervised methods for deceptive opinions text detection. However, only considering the emotional polarity of the deceptive opinion on the identification is not sufficient. At the same time, the local sampling of the CNN model can not take into account the existence of the word order in the text. Jindal Jindal2010Finding thought that the same user that gives his all positive opinions or negative opinions to the same brand of products is a kind of abnormal behavior and the corresponding opinion maybe deceptive opinion. The researchers proposed a ”one-condition rules” and a ”two-condition rules” model, to predict the falseness of the text by probabilistic prediction. Yapeng Jing Jingyapeng2014 sets the data set on the AMT of hotel opinions, uses the information gain to select the feature of the word bag and then detects deceptive opinions through the ordinary neural network, DBN-DNN network and LBP network. However, the artificial data set can not accurately reflect the true opinions.
Deceptive opinions detection is a type of complex text classification. The deceptive opinion is very short, varied type and content. In order to effectively identify deceptive opinions, besides the textual semantics and emotional polarity that have been widely used in text analysis, we need to further extract the deep features of deceptive opinions to characterize deceptive opinion effectively. Therefore, we introduce the word order into the CNN model, design the preservation of the k-max pooling technology and expand the deceptive opinion feature mining range to solve the difficulties in the identification of deceptive opinions and to enhance the accuracy of deceptive opinions detection.
3 Deceptive Opinion Detection Model
To achieve an accurate deceptive opinion detection, we study the word order characteristic in on-line opinion texts. In addition, we design a word order-preserving CNN network to model various short opinion texts. In this way, we embed a foundational textual characteristic into deceptive opinion detection process and obtain more accurate detection results.
3.1 Chinese Word Order
Almost all languages have its word ordering of the subject(S), object(O), and the verb(V), and among the languages of the world, all six possible basic word orders exist Dryer2005Order especially SVO((Subject-Verb-Object) and SOV(Subject-Object-Verb). The study has shown that the earliest human language had rigid word order. Nowadays, SOV basic word order is common among the languages of the world and that many other word orders can be reconstructed back to an SOV stage. It can be concluded that SOV must have been the word order of the ’ancestral language’ among the six possible word orders FJ2000On ; Gellmann2011The . In addition, there are researches to demonstrate that besides SOV, SVO is such a prominent word order in the languages of the world. For example, a sentence like ’fireman kicks boy’, both nouns could in principle be the agent. SVO is used to avoid expressing two plausible agents (’fireman’ and ’boy’) at the same side of the verb instead of SOV Gibson2013A . As a traditional language, Chinese text also possesses word order(SVO). Word order in Chinese text is an inherent feature of text classification. In this paper, in order to describe the short opinion text, we need the word order feature to the process of detective opinion feature mining, and optimize CNN model to identify deceptive opinions. Moreover, we will use the sentence with word order as the input of our model to prove the idea of word order-preserving in this paper.
3.2 OPCNN model
CNN model includes input layer, convolution layer, pooling layer and output layer. We proposed an improved CNN model considering the Chinese word order characteristic. The input layer takes the opinion sentences with a certain word order as input values. In convolution layer and the pooling layer, we preserve the word order of inputed sentences and apply the word order persevering pooling method instead of the original pooling layer,as shown in Fig.1. Ultimately, optimize the characteristic selection process of CNN model is optimized (the detailed model is illustrated in Fig.2).
3.2.1 Input Layer
We use word vectors to represent the word frequency of each word Johnson2014Effective and take them as the training inputs of our model. We use the word2vec model to predict words that appear in the context by training a neural network language model to generate word vectors. The input layer consists of an two-dimensional matrix, where n is the length of the sentence and m is the dimension of word vectors. The text representation process can be formulated as Eq.1, where a represents the matrix, w represents word vector of every word and v represents the value of every word vector. Ultimately, each opinion is represented by a two-dimensional word vector matrix.
3.2.2 Convolution Layer
The input layer transfers the word vector matrix A to the convolution layer for convolution operations. The padding of convolution has two types:same and valid. As is shown in Eq.2, we perform the i th convolution in the l-layer
on matrix A, taking the ReLU function as activation function, the biasas the the valid padding of convolution, and the matrix as feature map. The size of the convolution window is
, where h is the width of the convolution kernel and m is the dimension of the word vector. The width of the convolution kernel(h) needs to be set and adjusted dynamically, as is shown in Fig.1. As the convolution kernel continues to move down, the corresponding eigenvalues of the convolution kernel are generated. According to this convolution window, we will get a few of all “1” columns on the feature map.
The input value of the window is converted to an eigenvalue by the nonlinear transformation of the neural network. As the window moves down, the corresponding eigenvalues of the convolution kernel are generated and the eigenvectors corresponding to the convolution kernel are formed. We use the nonlinear transformation activation function called ReLU.
3.2.3 Word Order Persevering Pooling Layer
The word order persevering pooling layer reduces the number of feature parameters. The output of the order pooling layer is the maximum value of each feature map. The max pooling method can keep the location of the feature and the invariance value of the pooling operation. This feature affects the accuracy of text analysis, since the Chinese texts exists the word order characteristics. The position of each word in a sentence is a very important feature in the text analysis, so it is particularly important to preserve the word order of the sentences. Thus, the word order persevering k-max pooling method is proposed here to replace the original max pooling method in the paper.
As is shown in Eq.3, we use the word order persevering k-max pooling method generally to deal with the result of the convolution (l-1)layer. The method idea is to select the k maximum values from the one-dimensional feature map obtained from the previous convolution layer operation, and discard the other eigenvalues.
As shown in Fig.3, the word order-preserving k-max pooling method selects the k highest values in the sequence s, where length of s is longer than k. The order of the selected values corresponds to their original order in s. The word order-preserving k-max pooling method can discern more finely the number of times that the feature is highly activated in s Kalchbrenner2014A than that of max-pooling methods. What is more, the method can also distinguish the progression by which the high activations of the feature change across s. In this method, we can get the k highest feature values in the sequence s.
3.2.4 Output Layer
We concat the obtained features form the pooling layer. It is a two classification problem which distinguishes the deceptive opinion from real opinion. The result of the concat function is then entered into the softmax function to assess the probability that the opinion is deceptive. Finally, we use cross entropy as a model of the loss function to measure the difference between the predicted value and the true value in the OPCNN model.
3.3 Deceptive Opinions Detection Algorithm
To detect deceptive opinions with OPCNN model, we manually annotate the opinion data obtained from the websites. We construct the word vector model and preprocess the experimental data. Additionally, we take the OPCNN to obtain the final text classification results, to distinguish deceptive opinions from other opinions. The complete process of deceptive opinions detection is depicted in Algorithm 1 and Algorithm 2. Specifically, we train the OPCNN model according to Algorithm 1, and detect deceptive opinions with Algorithm 2.
Complexity analysis. Assuming that the number of iterations is k times, the number of samples per input sentence of OPCNN model is m, the number of words of each sentence is v, the word vector dimension is d, the convolution window size is w, and the number of output channels is n. The model tackles an inputed sentence with a time complexity of O(v*n*(2d*+w-1)). Therefore, the time complexity of the OPCNN model can be expressed as O(*k*m*n*d*v), when the model performs k iterations.
4.1 Experimental Data Set
To evaluate the performance of our deceptive opinions detection scheme, we use the Ott ott2011finding data set. Additionally, we labeled on-line opinions including the opinions about 23,166 hotels. On some on-line opinions websites, each user can write the relevant opinions and give the evaluation level regardless of buying a product or service or not. Therefore, false reviews and false scoring phenomenons become common. To training the proposed model, we hand-annotated 10000 hotel review data by the data annotation method presented by Li Li2011Learning .
In detail, we check whether the opinion is related to the products or not, and if there is no relevance, the opinion is a deceptive opinion. In addition, we observe whether the opinion is with too much emotion, such as the opinion including a large number of commendatory with strong emotion. Similarly, opinion that contains a large number of derogatory words may also be false opinion Crawford2015Survey ; renyafeng2015 .
Eventually, all opinions about the 23,166 hotels are marked. Among them, 2132 opinions are fake, as depicted in Table 1. In this experiment, 80% of the data set is used as the experimental training set, and the others are used as the experimental test set. In order to illustrate the generalization ability of the proposed method, the data set proposed by ott2011finding is applied in this paper.
|Opinion type||Opinion number||Opinion length|
In order to evaluate the performance of the proposed detection scheme, we implement the proposed detection scheme and three baseline schemes on TensorFlow. TensorFlow is an open source platform to implement deep learning model in piratical.
(1) The first experimental baseline uses the classical statistical method called tf-idf for feature extraction, supports vector machine (SVM) as a classifierasadullah2017classification and supervises the above-mentioned tagged data.
(2) The second baseline uses Bigram to extract the feature data Seneviratne2017Spam . Bigram is assumed to be in a statement that the probability condition of the second word depends on one word in front of it, that is the context of a word is defined as a word that appears in front of the word Lupker2012An . Some of the two consecutive characters usually have the ability to represent the features of the text. Then the support vector machine (SVM) is used as the classifier to obtain the classification result.
(3) The third baseline uses the Convolution Neural Network (CNN) in the deep learning framework huang2017detection
, combined with the short text feature extraction to apply the CNN to the deceptive comment identification. The experiment uses 3x cross validation to adjust the hyperparameters in the classifier model. The specific parameters are shown in Table 2. We use the ReLU function as a non-linear function, the super-parameter of the weight attenuation L2 is set to 0.5. Other parameters include dropout set to 0.5 and mini-batch to 50. In the CNN, we use the word2vec to get word vector as the embedding of the input layer. In the convolution layer, we use valid convolution and conv2d function to get feature map with TensorFlow. In the pooling layer, we use max pooling function to get the maximum feature value with TensorFlow. Lastly, we use the softmax function to implement test classification.
(4) We implement the OPCNN and set the parameters of OPCNN as the CNN used in the third baseline.
|d||word vector dimension||100|
|H||number of convolution||64,64,64|
4.3 Evaluation Metrics
In order to illustrate the experimental scheme, we evaluate the experiment from five aspects: accuracy, precision, recall, f1-measure and accuracy gain.
Accuracy (A): The ratio of the samples correctly sorted by the classifier to the total number of samples for a given test data set. That is, the loss function is 0-1 loss on the test data set on the accuracy rate. true positives(TP), false positives(FP), false negatives(FN) and true negatives(TN) are the related concepts of experiment effect.
Precision (P): It calculates the ratio of all ”correctly retrieved items (TP)” to all ”actually retrieved (TP + FP)”.
Recall (R): The item (TP) that is correctly retrieved is the item (TP + FN) that should be retrieved.
Accuracy gain(): The ratio of the experimental group method accuracy and the control group method accuracy. When the value of is lager, the accuracy of the experimental group is higher than that of the control group. When the value of is smaller, the accuracy of the experimental group is lower than that of the control group.
4.4 Analysis of Results
4.4.1 Word Vector Dimension Selection
In the paper, we use the word2vec model to get word vector. Firstly we should determine the dimension of the word vector because there is a certain relationship between the word vector matrix dimension input layer and convolution kernel width in OPCNN and CNN. In this experiment, we discuss the dimension of the word vector of word2vec. The specific evaluation index is the accuracy rate, as shown in Fig.4.
It can be concluded from Fig.4 that the accuracy of the dimension 100 get better result respectively. However compared with other results, the gain of the accuracy is not obvious, and the curve shape of the diagram is not V or M. The dimension of the word vector in word2vec has the influence on the accuracy of the OPCNN and CNN moedel in our paper.
4.4.2 K Value Selection
In the previous chapter we mention that the k-max pooling method is used in the pooling layer instead of the original max pooling method. The essence of the k-max pooling method is the use of the top-k function, where the choice of k is particularly important. In this experiment, we discuss the effect of k value on OPCNN model. The specific evaluation index is the accuracy rate, as shown in Fig.5. In this experiment, when k is equal to 3, the accuracy reaches the maximum. This is because when the value of k is too small, it may lose the eigenvalue if model encounters the same eigenvalue. When the value of k is too large, it may get interference items to affect the accuracy.
4.4.3 Accuracy Analysis
In this experiment, the classification results of the three groups of experiments are evaluated from the three evaluation indexes of accuracy, recall rate and f1-measure. The specific experimental results are shown in Table 3. It can be concluded from Table 3 that the accuracy, recall and f1-score of CNN is 67.33%, 64.79% and 67.20% respectively. Compared with tf-idf and Bigram, the accuracy, recall and f1-measure of CNN have been improved.
Compared with tf-idf+svm, Bigram on the division of the word takes into account the problem of word order to a certain extent. On the other hand, CNN can explore the characteristics of higher latitudes and can reduce the impact of sparseness of data, making the text classification better. If we incorporate CNN model with word order characteristics, we will obtain a more accurate detection result.
Due to the Chinese word order be taken into account the important role of deceptive opinions detection, the k-max pooling method is used to improve the traditional CNN in the pooling layer, which is more suitable for the research of text classification. Through the above experiment, we have set the k value. The experimental group uses the OPCNN model, and the parameters are consistent with CNN. The results of the classification of CNN and OPCNN models are evaluated from the accuracy, the recall and the f1-score. The specific experimental results are shown in Table 3.
From Table 3, compared with 67.33%, 64.69% and 67.20% of CNN, the method has achieved 70.02%, 66.83% and 69.76% of accuracy, recall and f1-measure respectively. In the field of Chinese text categorization, compared with the max pooling method used by the pooling layer, the word order preserving k-max pooling method solves the order problem of Chinese text to some extent. As mentioned earlier in this paper, The classification of the text effect is more obvious.
4.4.4 Scalability Analysis
In order to validate the generalization capabilities of the proposed method at the beginning of this chapter, this experiment uses the CNN and OPCNN models on Ott proposing data set. The OTT data set includes 1600 opinions which are divided into four types equally, positive truthful(P and T) opinions, positive deceptive(P and D) opinions, negative truthful(N and T) opinions and negative deceptive(N and D) opinions, as shown in Table 4. We set the same parameters and use the precision, recall rate and f1-score as evaluation indexes. The specific experimental results are shown in Table 5.
|Opinion type||P and T||P and D||N and T||N and D|
It can be seen from the experimental results that, on the data set proposed by Ott et al., OPCNN and CNN have the same improvement in the evaluation index. So it can be verified that the method has a good generalization ability. Compared with CNN, the improvement of the evaluation index in this paper is not due to the fact that the method has a certain dependency relationship with the data set used in this paper.
4.5 Effect of Sample Size
In order to fully verify the performance advantage of OPCNN compared with other classification methods in deceptive opinions detection, we can compare the classification results by changing the size of the training set. The evaluation index is the accuracy gain(). At the same time, in order to prevent the imbalance of the data in the experiment probably having the impact on the experimental results, this experiment uses the same number of deceptive opinion and real opinion. The effect of the number of specific training set samples on the experimental results is shown in Fig.6.
Compared with other methods, the classification method used in this paper obtains the value of more than 1. At the same time, as the number of samples increases, the accuracy rate of OPCNN model is increasing compared with the other three groups of control experiments, as shown in Fig.6. Since OPCNN and CNN are data driven, with the training sample increasing, the deeper the ability to characterize the depth model has, the higher the accuracy rate is. Compared with CNN, OPCNN has solved the influence of word order on Chinese text classification to a certain extent, so its accuracy is higher. When the number of samples reaches 3000, the accuracy rate is gradually stable, indicating that the accuracy of OPCNN model classification tends to be stable.
In our paper, the CNN in the deep learning model is used to identify the detective opinions. Against the short opinion text and the various forms of characteristics, we introduce the text order into the deceptive opinion analysis process and extend the scope of the opinion feature. In order to effectively excavate and merge the feature of the opinion text, this paper proposes a guaranteed k-max pooling operation on the basis of CNN. The text order feature is preserved in the process of text feature mining using CNN and the depth of opinion feature is optimized. Experiments show that the improvement of CNN model proposed in this paper can improve the recognition effect of deceptive opinions detection. However, there are still some shortcomings in this paper, such as: hand-annotated method costs much of manpower. Due to the subjective, the artificial marked data may be awareness of each person to some deviation. In the future experiments, we will continue to improve the above deficiencies to make a better accuracy of deceptive opinions detection.
This work was supported by Natural Science Foundation of China (61540004, 61502255 and 61650205), and was supported by Natural Science Foundation of Inner Mongolia Autonomous Region (2017MS(LH)0601).
- (1) C. L. Black, Method and system for training an artificial neural network (2001).
- (2) F. Pulvermller, A. Knoblauch, Discrete combinatorial circuits emerging in neural networks: A mechanism for rules of grammar in the human brain?, Neural Networks the Official Journal of the International Neural Network Society 22 (2) (2009) 161.
- (3) A. Knoblauch, Neural associative memories based on optimal bayesian learning (2013).
- (4) D. D’Amore, V. Piuri, Behavioral simulation of artificial neural networks: A general approach for digital and analog implementation, in: IEEE International Sympoisum on Circuits and Systems, 2002, pp. 1327–1330 vol.3.
X. Wei, J. D. Zhang, X. M. Qi, Study for workpiece identity based on the bp neural network and the zernike moment, Advanced Materials Research 694-697 (694-697) (2013) 1958–1963.
- (6) B. Q. [Aut, D. F. [Aut, rnn: Recurrent neural network, in: Advances in Neural Information Processing Systems, 2015, p. S88S98
- (7) N. Jindal, B. Liu, Opinion spam and analysis (2008) 219–230
- (8) M. Ott, Y. Choi, C. Cardie, J. T. Hancock, Finding deceptive opinion spam by any stretch of the imagination, in: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, Association for Computational Linguistics, 2011, pp. 309–319.
- (9) G. Fei, A. Mukherjee, B. Liu, M. Hsu, M. Castellanos, R. Ghosh, Exploiting burstiness in reviews for review spammer detection, 2013.
- (10) P. Wang, J. Xu, B. Xu, C. Liu, H. Zhang, F. Wang, H. Hao, Semantic clustering and convolutional neural network for short text categorization, Digital Content Technology and Service Research Center (2015) 352– 357.
- (11) A. Mukherjee, V. Venkataraman, B. Liu, N. Glance, What yelp fake review filter might be doing?, 2013.
- (12) R. Y. K. Lau, S. Y. Liao, C. W. Kwok, K. Xu, Y. Xia, Y. Li, Text mining and probabilistic language modeling for online review spam detection, Acm Transactions on Management Information Systems 2 (4) (2012) 1– 30.
- (13) L. Li, W. Ren, B. Qin, T. Liu, Learning Document Representation for Deceptive Opinion Spam Detection, Springer International Publishing, 2015.
- (14) N. Jindal, B. Liu, E. P. Lim, Finding unusual review patterns using unexpected rules, in: ACM International Conference on Information and Knowledge Management, 2010, pp. 1549–1552.
- (15) Y. Jing, Research of deceptive opinion spam recognition based on deep learning, Ph.D. thesis, East China Normal University,shagnhai (2014).
- (16) M. Dryer, Order of subject, object, and verb.
- (17) F. Newmeyer, On the reconstruction of proto-world word order, Evolutionary Emergence of Language.
- (18) M. Gellmann, M. Ruhlen, The origin and evolution of word order., Proceedings of the National Academy of Sciences of the United States of America 108 (42) (2011) 17290–5.
- (19) E. Gibson, S. T. Piantadosi, K. Brink, L. Bergen, E. Lim, R. Saxe, A noisy-channel account of crosslinguistic word-order variation., Psychological Science 24 (7) (2013) 1079.
- (20) R. Johnson, T. Zhang, Effective use of word order for text categorization with convolutional neural networks, Eprint Arxiv.
- (21) N. Kalchbrenner, E. Grefenstette, P. Blunsom, A convolutional neural network for modelling sentences, Eprint Arxiv 1.
F. Li, M. Huang, Y. Yang, X. Zhu, Learning to identify review spam, in: International Joint Conference on Artificial Intelligence, 2011, pp. 2488– 2493.
- (23) M. Crawford, T. M. Khoshgoftaar, J. D. Prusa, A. N. Richter, H. A. Najada, Survey of review spam detection using machine learning techniques, Journal of Big Data 2 (1) (2015) 23.
- (24) Y. Ren, Identifying deceptive reviews based on labeled and unlabeled data, Ph.D. thesis, Wuhan University (2015).
- (25) S. M. Asadullah, S. Viraktamath, Classification of twitter spam based on profile and message model using svm.
- (26) S. Seneviratne, A. Seneviratne, M. A. Kaafar, A. Mahanti, P. Mohapatra, Spam mobile apps: Characteristics, detection, and in the wild analysis, Acm Transactions on the Web 11 (1) (2017) 1–29.
- (27) S. J. Lupker, J. Acha, C. J. Davis, M. Perea, An investigation of the role of grapheme units in word recognition., Journal of Experimental Psychology Human Perception & Performance 38 (6) (2012) 1491–1516.
- (28) H.-H. Huang, Y.-W. Wen, H.-H. Chen, Detection of false online advertisements with dcnn, in: Proceedings of the 26th International Conference on World Wide Web Companion, International World Wide Web Conferences Steering Committee, 2017, pp. 795–796.