Disclaimer: This paper is concerned with violent online harassment. To describe the subject at an adequate level of realism, examples of our collected tweets involve violent, threatening, vulgar and hateful speech language in the context of racial, sexual, political, appearance and intellectual harassment. While these examples are shared to portray reality, readers are alerted in advance and may wish to avoid reading this material if it could cause discomfort and disagreeable response.
The presence of a significant amount of harassment in user-generated content and its negative impact calls for robust automatic detection approaches. This requires the identification of different types of harassment. Earlier work has classified harassing language in terms of hurtfulness, abusiveness, sentiment, and profanity. However, to identify and understand harassment more accurately, it is essential to determine the contextual type that captures the interrelated conditions in which harassing language occurs. In this paper
we introduce the notion of contextual type in harassment by distinguishing between five contextual types: (i) sexual, (ii) racial, (iii) appearance-related, (iv) intellectual and (v) political. We utilize an annotated corpus from Twitter distinguishing these types of harassment. We study the context of each kind to shed light on the linguistic meaning, interpretation, and distribution, with results from two lines of investigation: an extensive linguistic analysis, and the statistical distribution of uni-grams. We then build type- aware classifiers to automate the identification of type-specific harassment. Our experiments demonstrate that these classifiers provide competitive accuracy for identifying and analyzing harassment on social media. We present extensive discussion and significant observations about the effectiveness of type-aware classifiers using a detailed comparison setup, providing insight into the role of type-dependent features.
Disclaimer: This paper is concerned with violent online harassment. To describe the subject at an adequate level of realism, examples of our collected tweets involve violent, threatening, vulgar and hateful speech language in the context of racial, sexual, political, appearance and intellectual harassment. While these examples are shared to portray reality, readers are alerted in advance and may wish to avoid reading this material if it could cause discomfort and disagreeable response.
Although social media has enabled people to connect and interact with each other, it has also made people vulnerable to insults, humiliation, hate, bullying–facing threats from individuals who are either known (e.g., colleagues, friends) or unknown (e.g., fans, clients, anonymous entities). A Pew research center report  (e.g., offensive name-calling, and shaming, In this work, cyberbullying and harassment are used interchangeably.). One-in-five (18%) victims characterized their exposure as severe. The resulting negative impact from emotional distress, privacy concerns and threats to physical safety and mental health, affect individuals online and offline. This calls for tool-based, automatic detection, monitoring, and analysis of hurtful language to protect online users. The prior state-of-the-art is limited to detecting specific hurtful language such as hate speech , abusive language , and profanity , collectively termed Negative Affective Language (NAL). In the following, we present the definitions and terms for variants of harassing language:
Hate speech is “speech that denigrates a person because of their innate and protected characteristics” . Furthermore, it is divided into two categories: directed and generalized, depending upon whether there is an explicit target or not.
Abusive Language is “the collection and misuse of private user information, cyberbullying and the distribution of offensive, misleading, false or malicious information”.
These definitions are highly subjective and overlap, making them hard to differentiate. For example, the definition of harassing language is similar to aggressive language. We posit that all of these NALs are hurtful and thus harassing. But they might vary in their level of severity, presence or absence of target (victim), contextual interpretation and purpose. In this paper, we frame harassing language as offensive language where a given post/message contains “profanity, strongly impolite, rude, vulgar or threatening language”.
State-of-the-art harassment detection fails to exploit the contextual type of harassing language. Webster’s dictionary  provides the following definition for context: “the parts of a discourse that surround a word or passage and can throw light on its meaning”. Here, we describe the notion of contextual type as the linguistic or statistical conditions that help in differentiating the type of harassment. For example, the circumstance of a student who has been subjected to sexual harassment by her ex-partner differs from a student racially harassed because of her/his color. We suggest that contextual type influences the linguistic characteristics of harassment. We propose five contextual types of harassment in online communication on social media: (i) sexual harassment, (ii) racial harassment, (iii) appearance-related harassment, (iv) intellectual harassment, and (v) political harassment. This categorization is represented in Fig. 1. Below, we define each type of harassment using illustrative examples from the Twitter corpus we have created.
Sexual harassment is offensive sexual speech that usually targets females. E.g., the harasser might comment on the victim’s body in a vulgar manner or mention sexual relationships in an aggressive way. Note that using sexually profane words is not sufficient to indicate offensive sexual harassment [33, 6].
Racial harassment targets race and ethnicity characteristics of a victim such as skin color, country of origin, culture, or religion, in an offensive manner.
Appearance-related harassment uses embarrassing language referring to body appearance. Fat shaming  and body shaming are key subtypes of this type of harassment.
Intellectual harassment offends the intellectual power or opinions of individuals. Even smart people may be ridiculed and become victims.
Determining the real intent behind a tweet regarding the type of harassment can have serious implications for public perception. Consider the controversial tweet from Roseanne Barr targeting Valerie Jarett: characterizing Jarrett, an African-American woman born in Iran, as a child of the Muslim Brotherhood and an ape. Twitter Users regarded this tweet as racist, while Barr defended herself as making a bad joke about Jarrett’s politics and looks. Thus, whether the tweet is considered to be racist or regarded as appearance-related or political makes a significant difference. Reliable assessment of the type of harassment can have significant repercussions. We are unaware of any prior work on studying harassment concerning these five types.
We summarize our contributions as follows: (i) We introduce five contextual types of harassment. Then, we provide a systematic, and comparative analysis to assess offensive language from linguistic and statistical perspectives for each contextual type. This allows us to exploit relevant features for developing classifiers to identify these critical types of harassment on social media. (ii) We develop type-aware classifiers and capture their effectiveness using a detailed comparative study. This paper is organized as follows. The next section reviews the related literature. We then present the type-aware corpus that we have developed. Subsequently, we analyze our compiled corpus linguistically as well as statistically, which shows us the significant type-specific features for various types of harassment. We then discuss supervised learning approaches and classifiers for detecting the harassing language in comparative settings. We also provide an error analysis study regarding the pitfalls and challenges of our strategy. We close with the conclusions and our future plans.
State-of-the-art in Harassment Research
The previous research studies targeted various social media sources such as Twitter, Instagram, and Facebook. In Table 1, we summarize the prior literature with their corresponding goals, conclusions, and underlying data sets. Here, we specifically note particularly prominent related work. In , the authors seek to predict cyberbullying incidents on Instagram. They built a predictive model for the incidence of cyberbullying using features from initially posted data, a social graph, and temporal properties. The work in  proposed an approach for detecting harassment features based on the content, sentiment, and context. Using Slashdot and MySpace data, they showed significant improvement using TFIDF supplemented with sentiment and contextual features. The authors of  proposed an approach to spotting harassers as well as victims on social media. They considered the social structure and infer which user is a likely instigator and which user is expected to be a victim. This model is based on social interactions and the language of users in social media. Similarly,  proposes a method that simultaneously discovers instigators and victims of bullying incidents. It extends an initial bullying vocabulary using twitter and ask.fm. In , the authors proposed a supervised learning method for detecting cyberbullying in Japan. In , the authors propose a supervised learning method based on fuzzy logic and genetic algorithm
to identify the presence of cyberbullying terms and classify activities, such as flaming, harassment, racism, and terrorism on social media. Fuzzy rules were used to classify data, and a genetic algorithm was used for optimizing the parameters.
 explores the correlation of behaviors and actions of people and their emotions. The authors developed a large emotion-labeled dataset of harassing tweets. They applied 131 emotion hashtag keywords categorized into seven groups and collected 5 million tweets. To find useful features for emotion identification, they applied LIBLINEAR 
and Multinomial Naive Bayes
algorithms. They extracted n-gram features to analyze the emotion, and they applied Linguistic Inquiry and Word Count (LIWC) to expand the feature set with related emotional words. Interestingly, the authors of  target cyber-aggression and cyberbullying in a multi-modal context with text comments and media objects on Instagram. They concluded that non-text features are not able to substantially improve the performance of cyberbullying detection compared to text-based features.
Different from the previous work, some literature examines the psychological implications of harassment incidents . The authors in  sought the reasons behind the updates of posts on Facebook. They noticed that: (i) the majority of posts are about social activities and everyday life, (ii) people with low self-esteem updated their status on relationship whereas those with high self-esteem update their status with respect to their children. Moreover, people with narcissistic personality disorder updated their status through their achievements. Furthermore, they observed a correlation between the number of likes and comments with esteem level of people (e.g., the people with the low self-esteem receive fewer likes and comments because their status expresses greater negative affect). Similarly, the authors of  discuss narcissism personality disorder in Facebook users and its implications in harassing incidents. Our own past work [11, 31] focused on (i) using a conversation between a sender and a receiver to better capture its normal linguistic nature (e.g., base rates for curse word usage) and the nature of the relationship between participants (e.g., friends vs. strangers), and (ii) analyze comments/review threads to better identify offensive content in non-text media such as YouTube videos , to reliably detect harassment between participants.
|||Detecting offensive and hateful speech language||
|Accuracy improved wrt. network features.|
|scoring level of cyber bully and victim.|
|||Detecting cyber bullying in the Japanese community.||Data from Japanese secondary schools||
|||Detecting bullying incident on social networks||2M tweets collected in 4 weeks||
|256 Facebook users from locations around the world.||
|English and Dutch corpora from ASKfmsocial media sites.||
|corpus contain 4500 YouTube comments.||
|K entries from Instagram and Vine Dataset||
|980 entries from fakeNewsAMT and celebrity Dataset||
Type-aware Harassment Corpus
We published a type-aware annotated corpus and lexicon in . Our corpus consists of 25,000 annotated tweets for the five types of harassment content and is available on the Git repository. In the following, we discuss our strategies for corpus compilation and annotation. The identification of cyberbullying typically begins with a lexicon of potentially profane or offensive words. We created a lexicon (compiled from online resources   ) containing offensive words covering five different types of harassment context. The resulting compiled lexicon includes six categories: (i) sexual, (ii) racial, (iii) appearance-related, (iv) intellectual, (v) political, and (vi) a generic category that contains profane words not exclusively attributed to the five specific types of harassment. A native English speaker conducted this categorization.
Corpus Development and Annotation.
We employ Twitter as the social media data source because of its extensive public footprint. Twitter reports 313 million monthly active users that generate over 500 million tweets per day . Although the size of a tweet is restricted (140 characters at the time of corpus collection), once we consider a more extensive aggregation of tweets on a specific topic, mining approaches reveal valuable insights. We utilized the first five categories of our lexicon as seed terms for collecting tweets from Twitter between December 18th, 2016 to January 10th 2017 (This date was close to the US presidential election. Then our political sub-corpus has many tweets with the subject of Trump). Requiring the presence of at least one lexicon item, we collected 10,000 tweets for each contextual type for a total of 50,000 tweets. As shown in Table 2, nearly half of these tweets were annotated. However, the mere presence of a lexicon item in a tweet does not assure that the tweet is harassing because the individuals might utilize these words with a different intention, e.g., in a friendly manner or as a quote. Therefore, human judges annotated the corpus to discriminate harassing tweets from non-harassing tweets. Three native English speaking annotators determined whether or not a given tweet was harassing with respect to the type of harassment content and assigned one of three labels yes, no, and other. The last label indicates that the given tweet either does not belong to the current context or cannot be decided. Ultimately, we acquired 24,000 annotated tweets represented in Table 2. Note that the annotation task was done on a per tweet basis although it can be improved using the entire conversation history.
|Contextual Type||Annotated Tweets||Harassing ✓||Non-Harassing ✗|
Although the annotators employed three labels, i.e., yes, no, and other, the eventual corpus excluded all tweets without a consensus label of “yes” or “no”. That is, the corpus contains only those tweets that received at least two “yes” or two “no” labels. Cohen’s kappa coefficient  measures the quality of annotation by category in Table 3. The appearance-related context shows the highest agreement rate whereas political and sexual contexts have the lowest, indicating that they are more challenging to judge due to higher ambiguity.
|Content Type||Agreement Rate|
Annotating Golbeck Corpus.
The public state-of-the-art harassment-related corpus is the Golbeck corpus  that only provides generic annotation, i.e., (i) harassing and (ii) non-harassing. This corpus contains 20,428 non-redundant annotated tweets of which only 5,277 are labeled as harassing. Since we require context-aware annotations, we re-annotated the harassing tweets of Golbeck. The agreement rate (Cohen’s kappa) between the two annotators is 86%. As shown in Table 4, more than 75% of the harassing tweets are racial. This statistic confirms Golbeck’s observation. While this may be an accurate reflection of the base rate, our view is that different harassment contexts may have different consequence. An imbalanced corpus at the foundation of our research effort could result in misses of particular practical import to teenage mental health, concerning sexuality, appearance and intellect.
|Contextual Type||#of Tweets|
LIWC Analysis for Different Types of Harassment
Linguistic analysis of our corpus sheds light on the differences between the harassing corpus versus non-harassing corpus for each type. Furthermore, it provides a comparison between various types of harassment. We divided our corpus into 12 sub-corpora: (i) one generic corpus containing all harassing tweets regardless of their type, called the combined harassing corpus, (ii) one generic corpus containing all non-harassing tweets irrespective of the type called the combined non-harassing corpus, (iii) five contextual type-aware corpora including only harassing tweets per type, (iv) five contextual type-aware corpora including only non-harassing tweets per type. For linguistic analysis, we utilized LIWC  
. This tool tallies 96 linguistic features using a multiword lexicon for each feature. We individually analyzed each of the 12 sub-corpora using LIWC. An effect size , statistic estimates the magnitude of an effect (e.g., mean difference, regression coefficient, Cohen’sd, and correlation coefficient)  metric was used to determine significant discriminators . Conventionally, a proportion (feature) is considered moderately discriminating when its effect size is more than (i.e., ), and is considered unhelpful if . The effect size for each feature is calculated as follows:
where, is the mean of the experimental group on the given feature , is the mean of the control group wrt. the given feature and
is the standard deviation. For each content corpus as well as for the combined corpus, we consider the harassing corpus as the experimental group and the non-harassing corpus as the control group. We compared the prevalence of the 96 LIWC features in the harassing corpus to their prevalence in the corresponding non-harassing corpus. Out of the 96 original features, we removed features that were not significant in any of the contextual types and retained 38 of the most discriminating features as shown in Fig.2. The extreme red (green) color represents significance (regarding effect size) of the corresponding feature in the harassing (non-harassing) corpus. In the following, we highlight specific significant features to make three points. First, a feature is often diagnostic of the non-harassing corpus. Second, feature significance is type dependent. The third is related to both points: a given feature, such as “you”, can be a positive indication of harassment for one type and a negative indication of harassment for another. In the following, we indicate highly significant linguistic features derived from Fig. 2 for each individual type. Note that our corpus is already biased towards curse words because curse words are present as seeds for crawling. Thus, our observations on discriminatory features are conditional on a “high recall curse word-laden corpus”.
The pronoun “I” is prevalent in the sexually non-harassing corpus with , which is highly significant, e.g., i’m le***an kiss. Furthermore, the feature “MONEY” is prevalent in the harassing corpus with . E.g., send free moneyb**ch hoe wont give dance hoe a*s industry bitch d*cks*** p*r* star people.
The pronoun “YOU” is prevalent in the harassing corpus with , e.g., Vishalp sikanda, Quideazam hahahaha u p**i can block u cant debat u p**i I***an. The “COMPARATIVE” feature is prevalent in racial non-harassing corpus with , e.g., save block p**i like po yung comment ni richard fronda (the word ‘like’ is an indicator of comparison in LIWC). Thus, these features can be used to discriminate between harassing and non-harassing tweets.
The pronoun “SHE” and “HE” with and the pronoun “WE” with are prevalent in the non-harassing corpus, e.g., realdonaldtrump putin a**hat just like word can express displeasure leader god help us (us indicates the pronoun ‘WE’). The “RISK” feature is significant in non-harassing with , e.g., f*** wrong democratic senators. The word ‘wrong’ represents a risk feature in LIWC dictionary. Other sample risk related words are ‘danger’, ‘doubt’, etc. Furthermore, the “ANXIETY” feature with is significant in the non-harassing corpus. E.g., well i’m true dumb f*** democrat wouldn’t doubt.
is prevalent in the harassing corpus (probably because of the negative language used for referring to the body and appearance-related subjects). E.g.,Taylor swift cant shake c**el toe. The other significant feature in the harassing corpus is the ‘‘PAST TENSE”. E.g., Ugli a** didn’t go run yesterday get work f*t**s. Furthermore, the “COMPARATIVE” feature is prevalent in appearance-related harassing corpus with . E.g., hey lardass notice your look pizza perhaps like f*** salad a**hole. The word ‘like’ indicates a comparative feature.
The “FEMALE REFERENCE” feature with is highly significant in intellectual harassing corpus (perhaps because girls are harassed more wrt. intellectual issues.) E.g., She is dumb f***.
“DISCREPANCY” with is prevalent in the non-harassing corpus e.g., boss brought drunken sugar cook explain there alcohol just sh**face.
Statistical Analysis of Different Types
We investigate the relationship between the offensive words employed in collecting our corpora and the specific lexical items in the crawled corpora. We determine Q1: whether or not offensive words are observed as frequent words, Q2: whether or not the frequent words in harassing corpora differ from those in non-harassing corpora, and Q3: whether or not frequent words are type-sensitive, in other words, whether the frequent words vary with type of context. Fig. 3 shows the 2D visualization of the word embeddings of the top-25 most frequent words for the harassing corpora, whereas Fig. 4 represents a similar display for the top-25 most frequent words for the non-harassing corpora (the following section presents the details of word embedding). The prevalence of curse words in the non-harassing corpora is comparable to the harassing corpora. This confirms that the presence of curse words is not a sufficient indicator of harassment. In the following, we mention our key observations.
Regarding Q1, as expected, we observed that offensive words are commonplace in both harassing and non-harassing corpora across types (cf. Fig. 3 and 4). In addition, we observed some emerging, frequent offensive words, such as “grab” and “camel”that can now be added to our initial offensive lexicon . Furthermore, there are frequent words that are not necessarily offensive. E.g., consider “look” or “eat” in the appearance-related type where they are implicitly related to the associated type, applicable to the appearance of a subject. Regarding Q2, we observe that the frequent words in the harassing corpora are different from those in the non-harassing corpora. The particular words in the harassing corpora also can be added to the initial lexicon of seed words. The result of this analysis can be utilized for weighting the severity of offensiveness for every single word included in our lexicon.
To reply quantitatively to Q3, we ran an annotation task on the top-15 most frequent words for each type of harassing corpus as well as the corresponding non-harassing corpus. The description of this task is as follows: we asked the human annotators (i.e., graduate students) to determine whether or not a given frequent word is related to the associated type either explicitly or implicitly. E.g., the words “eat” or “food” are implicitly related to appearance while they seem far from the type racial. The results of this exercise appear in Table 5. In the harassing corpora, the percentage of relatedness of words to the associated type is higher than 67% and in sexual and racial types, it even reaches 80%. This percentage fluctuates for non-harassing corpora. E.g., in appearance-related type, it is higher than 93% while in racial it reaches 53%. In sum, we conclude that the frequent words are mostly type-sensitive. Moreover, the prevalence of apparently offensive language in the non-harassing corpus reinforces our claim that offensive language per se is not necessarily harassing.
One caveat is that the most frequent words appearing in the sub-corpus associated with each type are predominantly stop-words or curse words, as our initial seed terms are biased to an offensive lexicon. Ignoring these words, whose presence cuts across different types of harassment, revealed that the following prominent word groups are associated with various harassment types, shedding light on the possible features that may elicit harassment: (i) In the appearance-related harassment corpus, target words such as “eat”, “ugly”, “fat”, “gym”, and “weight”, are present. (ii) In the intellectual harassment corpus, target words such as “dumb”, “stupid”, “work”, and “head”, are present. (iii) In the political harassment corpus, the target words such as “realdonaldtrump”, “libtard”, “dumb”, “touch bag”, “stupid”, and “cnn”, are present. (iv) In the racial harassment corpus, target words such as “m*ki”, “n**ger”, “b**ner”, “ch**k”, “muslim”, “i**ian”, “moron”, and “jew”, are present. (iv) In the sexual harassment corpus, target words such as “hump”, “hussy”, “l**k”, and “grab”, are present.
Predicting Different Types of Harassing Posts
We aim to develop effective supervised learning methods to detect harassing language automatically and distinguish it from non-harassing language for each contextual type. The state-of-the-art contains various approaches for detecting harassing content from non-harassing content but not for discriminating the type of harassment. We approach this gap in two ways. The first is to build individual binary classifiers that identify a particular type of harassment, e.g., a binary classifier that identifies only racial content or a binary classifier that classifies just offensive political content. The second approach uses the state-of-the-art methods to detect harassing language; after such recognition, we can employ a type-aware classifier to predict the associated type for that harassment incident. We implemented both approaches. Initially, we trained the individual classifiers for each type. In another approach, we built up a binary classifier that differentiates harassing content from non-harassing content regardless of their type. Note that any classifier from the state-of-the-art can substitute for this part. Then, we built up a multi-class classifier that predicts the type of harassment incident. The results of our experiments for both approaches reveal high accuracy. Furthermore, to verify the effectiveness of our classifier, we apply transfer learning by running our classifier on the Golbeck corpus and assess its performance for how successfully it predicts the type of harassment. In the following, we present the details of our experiments.
Transforming Tweets to Vectors
We utilized four approaches for transforming tweets to numerical representations (i.e., vectors): (i) the conventional vectorization approach TFIDF, (ii) word2vec, (iii) fastText and (iv) a LIWC vector. We feed our classifiers with each of these individual vectors or a combination of them.
The Term Frequency and Inverse Document Frequency (TFIDF).
We use this approach  to transform each given tweet into a weighted vector T.
Distributional semantics (i.e., word2vec and fastText).
Distributional semantics (so-called embedding models) 
play a vital role in many Natural Language Processing (NLP) applications. They capture the semantics of text units (e.g., words, characters, tweets, paragraphs or documents) from the underlying corpus and represent them in a low dimensional vector space. We use two major embedding models for representing each tweet. The first one is word2vec and the second one is fastText  . The first one learns a dense representation at the unigram level and the second one learns at the character level. Both of these approaches have two models, i.e., skip-gram model and CBOW model [38, 40] that are roughly similar. The skip-gram model (CBOW model) computes the probability of the target word (i.e. context word) appearing in the neighborhood of the context word (i.e. target word), . In this work, the vector representation of a tweet is computed as the concatenation of the vector of all tokens within the tweet. In the rest of this paper we rely on the following notations to specify a vector. W(S) and W(C) denote the low dimensional vector obtained respectively by the skip-gram model and CBOW model of the word2vec approach. F(S) and F(C) denote the low dimensional vector obtained respectively by the skip-gram model and the CBOW model of the fastText approach. We compiled a corpus containing 15,999,557 sentences from the Twitter and Leipzig Collection Corpora  leveraging our offensive lexicon presented in  as the underlying seed words. Then, we trained the embedding models on this accumulated corpus using the learning parameters reported in [38, 40]. Our dimension size equals 300, the window size is 3, and the minimum count equals 10.
The vector obtained by running the LIWC tool is denoted by L.
Evaluation of the Harassment Classifiers
Preparing training datasets.
As the number of harassing tweets is not equal to the number of non-harassing ones in our corpus – in fact, it varies for each type – we prepared balanced datasets for training the classifier. We prepared five type-aware training data sets using an under-sampling approach taking all of the harassing tweets with an equal number of non-harassing (randomly sampled). Also, we prepared a combined training data set considering all of the harassing tweets regardless of their type and an equal number of non-harassing tweets. Table 6 shows the size of the training data sets for each type. Each data set contains an equal number of harassing tweets versus non-harassing tweets. Later, we employ the remaining tweets to test the robustness of the classifiers against unseen data.
|Category||Number of tweets|
Training Binary Classifiers.
(KNN), (iii) Gradient Boosting Machine (GBM) , and (iv) Naive Bayes (NB) 
. We rely on the following settings for the GBM classifier: the learning rate is 0.1, loss function is logistic regression, the number of trees is 100, sub-sample is 1.0, the criteria function is Friedman MSE, the minimum sample is 2, the minimum number of samples required to be at a leaf node is 1, and the maximum depth of the individual regression estimators is 3. We ran 10-fold cross-validation with re-sampling and iteration strategies (repeated five times). Figure5
shows the performance of the classifiers based on an F-score measure using a TFIDF vector. Generally, the results of the NB classifier in all of the cases were inferior whereas the GBM classifier outperforms others in the majority of settings except for a few instances comparable to the SVM classifier. Thus, in the following experiments, we rely on the GBM classifier.
To gain insight over the effectiveness of various features, we feed the GBM classifier with various feature settings. The fine-grained results of our experiment are listed in Figure 6. We employed a various combination of vectors, for example, F(S)+W(S) means the input features were the skip-gram models of fastText and word2vec. In addition to the typical precision, recall and F-score measures, we provide specificity (True negative rates) and accuracy rates. We offer the following observations: (i) The tweet representation using F(S)+W(S) vector is the most effective input representation as it provides high and balanced rates for all measures including precision, recall, F-score and specificity. Note that in multiple settings such as F(S)+L+T, precision, recall, and f-score are high whereas specificity is low meaning that the classifier is biased towards one of the classes and does not perform reasonably on both classes. (ii) In the settings for which the LIWC vector L is included, typically the specificity rate is low. This probably means L vector does not provide a discriminative representation for the classifier. (iii) Generally learning the representation of tweets using the fastText approach either with skip-gram or CBOW shows high performance. This might come from the fact that encoding tweets at the character level is more effective for detecting harassment. (iv) The sexual type resulted in the classifier with the highest accuracy (with F-score 96% and specificity 94%), racial and intellectual are in the next positions (respectively with F-score 88%, 86% and specificity 83%, 79%).
Binary Classifier for Harassment Detection.
We also trained a binary classifier on our combined corpus where it can differentiate the harassing language from non-harassing regardless of the contextual type. In situations that the type of harassment does not play a role, or type detection must occur after the harassment detection, using such a generic classifier is necessary. Table 7 shows the detailed results of this classifier in various settings of input features. Generally, the vector of FastText F shows an effective role, especially when it is coupled with the W vector; the specificity score reaches its optimum.
Type Prediction using a Multi-Class Classifier.
Apart from building binary classifiers for predicting types, we trained a multi-class classifier to predict the type of harassment incidents. We trained several multi-class classifiers, among them the GBM classifier outperformed others. Herein we report the result for GBM classifier only. We used W(S)+F(S) vectorization approach as the input feature. Then, we trained this classifier on a corpus containing all of the sub-corpora from the previous step. This corpus has samples with six various labels where five labels indicate a particular type of harassment and the last label indicates “non-harassing” implying there is no harassing language. Table 8
shows the details of the evaluation on the performance of this classifier where the micro F-score is 0.92 and the macro F-score is 0.82. Note that in the macro-level, we calculate the performances of each class and then average whereas, in the micro-level, we calculate the performance for all classes, as computing contingency table and then evaluate precision/recall and F-score
. Digging into fine-grained efficiency shows that the accuracy across various classes holds similar behaviors except for a decrease in the precision and recall of the sexual type. As we will discuss in error analysis below, this type is prone to mis-classification with the other types particularly the racial type. However, comparing the performance of multi-class classifier and binary classifiers shows that the multi-class classifier mostly outperforms the binary classifiers by as much as. Note that the accuracy of our classifier will improve on a generic tweet corpus because our current corpus has been crawled using curse words with a significantly higher proportion of harassing tweets compared to that in a generic tweets corpus, which is predominantly non-harassing and devoid of curse words. On the downside, it will miss harassment conveyed through “clean” words. However, to demonstrate the effectiveness of the current version of this classifier, in the next step we apply it on an unseen corpus to predict the type of harassment incident.
|Micro Precision||0.92||Macro Precision||0.82|
|Micro Recall||0.92||Macro Recall||0.83|
|Micro F-score||0.92||Macro F-score||0.82|
Comparison to the state-of-the-art.
Since this work was the first to introduce contextual type for harassment, comparison to the state-of-the-art that relies only on two or three variants of harassment, is unfair. However, to verify the effectiveness of our type-oriented multi-class classifier, we tested it on the harassing tweets from the Golbeck corpus (an external corpus unseen to our classifier) that is a publicly available state-of-the-art harassment-related corpus . This corpus contains 20,428 annotated tweets of which only 5,277 are labeled as harassing. It does not distinguish the nature of the harassment. In Table 4, we represented our annotations for the harassing tweets of the Golbeck corpus with respect to our types using human judges which yielded in an agreement rate of 86%. The proportion of harassing tweets per type is represented in the last column of Table 9. We ran our type-aware multi-class classifier (GMB classifier) to predict the associated type of harassing tweets on Golbeck corpus. Table 9 shows the precision, recall and F-score for each type. We observe an F-score of more than 94% for all types except for the type appearance. In the case of the racial type, the F-score reaches 98%. This high performance exceeds the state-of-the-art where they are mostly concerned about detecting the general harassing language (the reported accuracy ranges between 70% and 85%) [51, 37, 66, 21]. In addition, it shows robustness with unseen data. Note that the racial type is dominant in the Golbeck corpus. We also ran our classifier on a portion of 5,000 non-harassing tweets from Golbeck corpus, which resulted in the F-score > 98% (cf. Table 9). The last three rows of Table 9 show micro and macro precision, recall, and F-score. The closeness of the micro and macro measures shows that the classifier is not biased towards a dominant class.
|Micro Precision||0.97||Macro Precision||0.91|
|Micro Recall||0.97||Macro Recall||0.90|
|Micro F-score||0.97||Macro F-score||0.91|
To make sense of classifier errors, we examined a couple of tweets classified as sexual. E.g., for @usr you deserved to be raped by a thousand Muslims in your c**t a**hole , our classifier classified that as sexual harassment and not racial because of the word ‘rape’. Similarly, the tweet @usr @usr lol it’s not against women. It’s against f***ing feminist c***s like you. #feminazi #womenagainstfeminism was classified as sexual. Such cases are ambiguous because even manual annotation is highly subjective. In other words, categorizing harassment is highly subjective and the boundary between types is not rigid. In majority of the overlapping cases (racial and sexual), the tweets were classified as sexual rather than racial. We also analyzed errors in political tweets and concluded that harassment signal can be: (i) implicit, e.g., John Boehner blames Democrats for #shutdown. He better stop drinking cuz a few more drinks and he starts blaming the J*ws f, (ii) ambiguous ??? You’re a wh*** to the telecom industry, i hope your constituents vote you out., (iii) unreliable, e.g., It’s going to be a republican government in the US next term. Democrats can kiss their presidency bid goodbye. Let the J*ws rule!, (iv) poorly captured through annotation, e.g., the tweet @TrueNugget @FeministPeriod @OregonState Man college is becoming more and more a mistake. in the Golbeck corpus. Our classifier misses them as they are weak cases of harassment.
We anonymized data. The study was approved by the Wright State University Institutional Review Board entitling "Student Use of Social Media" as protocol number IRB #: 06251.
Conclusion and Future Plans
In this paper, we introduced five contextual types for harassment, namely, (i) sexual, (ii) racial, (iii) intellectual, (iv) appearance-related and (v) political. We presented experiments with a type-aware tweets corpus to analyze, learn, and understand harassing language for each type. Our contribution lies in providing a systematic and comparative approach to assessing harassing language from linguistic and statistical perspectives. Furthermore, we built type-specific classifiers, and the results of our experiments show the importance of considering the contextual type for identifying and analyzing harassment on social media.
In general, a single tweet identified as “harassing” may not provoke the same intense negative feeling that we associate with that word in the real-world scenario. However, in practice, “conversational” exchanges containing a sequence of such tweets can rise to the level of harassment causing mental and psychological anguish, and fear of physical harm. Nevertheless, our current Twitter dataset is limited to annotating single tweets in isolation for harassment. Furthermore, the reliable assessment of the type of harassment is a difficult problem because it requires significant knowledge of current events and common-sense. We plan to extend this work by learning the language of harassers as well as victims, and further study the contribution of non-verbal cues (i.e., conversational features, network features, and community features) for identifying online harassment activities, particularly on social media.
We acknowledge support from the National Science Foundation (NSF) award CNS 1513721: Context-Aware Harassment Detection on Social Media. Any opinions, findings, and conclusions, recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.
- 1. Sofia Berne, Ann Frisén, and Johanna Kling. Appearance-related cyberbullying: A qualitative investigation of characteristics, content, reasons, and effects. Body image, 11(4):527–533, 2014.
- 2. Despoina Chatzakou, Nicolas Kourtellis, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, and Athena Vakali. Mean birds: Detecting aggression and bullying on twitter. In Proceedings of the 2017 ACM on web science conference, pages 13–22. ACM, 2017.
- 3. Ying Chen, Yilu Zhou, Sencun Zhu, and Heng Xu. Detecting offensive language in social media to protect adolescent online safety. In 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, pages 71–80. IEEE, 2012.
- 4. Lu Cheng, Jundong Li, Yasin N Silva, Deborah L Hall, and Huan Liu. Xbully: Cyberbullying detection within a multi-modal context. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 339–347. ACM, 2019.
- 5. Robert Coe. It’s the effect size, stupid: What effect size is and why it is important. 2002.
- 6. US Equal Employment Opportunity Commission. Sexual harassment, 2018.
- 7. Correlations. Ranking bully types, 2018.
- 8. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In Eleventh International AAAI Conference on Web and Social Media, 2017.
- 9. Karthik Dinakar, Roi Reichart, and Henry Lieberman. Modeling the detection of textual cyberbullying. In fifth international AAAI conference on weblogs and social media, 2011.
- 10. MAEVE DUGGAN. Online harassment 2017, 2017.
- 11. Venkatesh Edupuganti. Harassment detection on twitter using conversations. Master’s thesis, Master Thesis, Department of Computer Science and Engineering, Wright State University, USA, 2017.
- 12. Mai ElSherief, Vivek Kulkarni, Dana Nguyen, William Yang Wang, and Elizabeth M. Belding. Hate lingo: A target-based linguistic analysis of hate speech in social media. In Proceedings of the Twelfth International Conference on Web and Social Media, ICWSM 2018, Stanford, California, USA, pages 42–51, 2018.
- 13. Facebook. Fasttext, 2013.
Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin.
Liblinear: A library for large linear classification.
Journal of machine learning research, 9(Aug):1871–1874, 2008.
- 15. Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. Large scale crowdsourcing and characterization of twitter abusive behavior. In Proceedings of the Twelfth International Conference on Web and Social Media, ICWSM 2018, Stanford, California, USA, pages 491–500, 2018.
- 16. Jerome H Friedman. Stochastic gradient boosting. Computational statistics & data analysis, 38(4):367–378, 2002.
- 17. Jennifer Golbeck, Zahra Ashktorab, Rashad O Banjo, Alexandra Berlinger, Siddharth Bhagwan, Cody Buntain, Paul Cheakalos, Alicia A Geller, Quint Gergory, Rajesh Kumar Gnanasekaran, et al. A large labeled corpus for online harassment research. In Proceedings of the 2017 ACM on Web Science Conference, pages 229–233. ACM, 2017.
- 18. Eckart T. Quasthoff Goldhahn, D. Large monolingual dictionaries at the leipzig corpora collection: From 100 to 200 languages. In Proceedings of LREC 2012 (pp. 759-765), 2012.
- 19. Google. Word2vec, 2013.
- 20. Michael Herz and Péter Molnár. The content and context of hate speech: rethinking regulation and responses. Cambridge University Press, 2012.
- 21. Homa Hosseinmardi, Sabrina Arredondo Mattson, Rahat Ibn Rafiq, Richard Han, Qin Lv, and Shivakant Mishra. Detection of cyberbullying incidents on the instagram social network. arXiv preprint arXiv:1503.03909, 2015.
- 22. Homa Hosseinmardi, Rahat Ibn Rafiq, Richard Han, Qin Lv, and Shivakant Mishra. Prediction of cyberbullying incidents in a media-based social network. In 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 186–192. IEEE, 2016.
- 23. Homa Hosseinmardi, Rahat Ibn Rafiq, Richard Han, Qin Lv, and Shivakant Mishra. Prediction of cyberbullying incidents in a media-based social network. In 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 186–192. IEEE, 2016.
- 24. Bright Hub. Guide to dealing with political harassment in the workplace, 2010.
- 25. Internet. Swear word list.
- 26. Internet. Banned word list, 2009.
- 27. Internet. Liwc2015, 2015.
- 28. Internet. Online dictionary, 2017.
- 29. Internet. Turns out, there is political discrimination and harassment too, 2018.
- 30. Dan Jurafsky and James H Martin. Speech and language processing, volume 3. Pearson London, 2014.
- 31. Rajeshwari Kandakatla. Identifying offensive videos on youtube. Master’s thesis, Master Thesis, Department of Computer Science and Engineering, Wright State University, USA, 2016.
- 32. New York Employment Discrimination Lawyers. Racial slurs and racial harassment, 2018.
- 33. New York Employment Discrimination Lawyers. Sexual harassment, 2018.
- 34. Mohammad Saeid Mahdavinejad, Mohammadreza Rezvan, Mohammadamin Barekatain, Peyman Adibi, Payam Barnaghi, and Amit P Sheth. Machine learning for internet of things data analysis: A survey. Digital Communications and Networks, 4(3):161–175, 2018.
- 35. Amrita Mangaonkar, Allenoush Hayrapetian, and Rajeev Raje. Collaborative detection of cyberbullying behavior in twitter data. In 2015 IEEE international conference on electro/information technology (EIT), pages 611–616. IEEE, 2015.
- 36. Tara C Marshall, Katharina Lefringhausen, and Nelli Ferenczi. The big five, self-esteem, and narcissism as predictors of the topics people write about in facebook status updates. Personality and Individual Differences, 85:35–40, 2015.
- 37. Tolba Marwa, Ouadfel Salima, and Meshoul Souham. Deep learning for online harassment detection in tweets. In 2018 3rd International Conference on Pattern Analysis and Intelligent Systems (PAIS), pages 1–5. IEEE, 2018.
- 38. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781, 2013.
- 39. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. Advances in pre-training distributed word representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, 2018., 2018.
- 40. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held 2013, Lake Tahoe, Nevada, United States., pages 3111–3119, 2013.
- 41. Gilad Mishne et al. Experiments with mood classification in blog posts. In Proceedings of ACM SIGIR 2005 workshop on stylistic analysis of text for information access, volume 19, pages 321–327, 2005.
- 42. Saeedeh Shelarpour Mohammadreza Rezvan. Harassment-corpus, 2018.
Shinichi Nakagawa and Innes C Cuthill.
Effect size, confidence interval and statistical significance: a practical guide for biologists.Biological reviews, 82(4):591–605, 2007.
- 44. B Sri Nandhini and JI Sheeba. Online social network bullying detection using intelligence techniques. Procedia Computer Science, 45:485–492, 2015.
- 45. Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. Abusive language detection in online user content. In Proceedings of the 25th international conference on world wide web, pages 145–153. International World Wide Web Conferences Steering Committee, 2016.
- 46. James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. The development and psychometric properties of liwc2015. Technical report, 2015.
- 47. Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. Automatic detection of fake news. arXiv preprint arXiv:1708.07104, 2017.
- 48. Michal Ptaszynski, Pawel Dybala, Tatsuaki Matsuba, Fumito Masui, Rafal Rzepka, and Kenji Araki. Machine learning and affect analysis against cyber-bullying. the 36th AISB, pages 7–16, 2010.
- 49. Elaheh Raisi and Bert Huang. Cyberbullying identification using participant-vocabulary consistency. arXiv preprint arXiv:1606.08084, 2016.
- 50. Elaheh Raisi and Bert Huang. Cyberbullying detection with weakly supervised machine learning. In Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, pages 409–416. ACM, 2017.
- 51. Elaheh Raisi and Bert Huang. Weakly supervised cyberbullying detection using co-trained ensembles of embedding models. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 479–486. IEEE, 2018.
- 52. Mohammadreza Rezvan, Saeedeh Shekarpour, Lakshika Balasuriya, Krishnaprasad Thirunarayan, Valerie L Shalin, and Amit Sheth. A quality type-aware annotated corpus and lexicon for harassment research. In Proceedings of the 10th ACM Conference on Web Science, pages 33–36. ACM, 2018.
- 53. Huascar Sanchez and Shreyas Kumar. Twitter bullying detection. ser. NSDI, 12:15–15, 2011.
- 54. Sajedul Talukder and Bogdan Carbunar. Abusniff: Automatic detection and defenses against abusive facebook friends. In Twelfth International AAAI Conference on Web and Social Media, pages 385–394, 2018.
- 55. Ryoko Tokuhisa, Kentaro Inui, and Yuji Matsumoto. Emotion classification using massive examples extracted from the web. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, pages 881–888. Association for Computational Linguistics, 2008.
- 56. Internet Users. The racial slur database, 1999.
- 57. Internet Users. Macmillan dictionary, 2009-2019.
- 58. Internet Users. Twitter, 2019.
- 59. Cynthia Van Hee, Gilles Jacobs, Chris Emmery, Bart Desmet, Els Lefever, Ben Verhoeven, Guy De Pauw, Walter Daelemans, and Véronique Hoste. Automatic detection of cyberbullying in social media text. PloS one, 13(10):e0203794, 2018.
- 60. Luis von Ahn’s Research Group. Offensive/profane word list.
- 61. Wenbo Wang, Lu Chen, Krishnaprasad Thirunarayan, and Amit P Sheth. Harnessing twitter" big data" for automatic emotion identification. In 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, pages 587–592. IEEE, 2012.
- 62. Carrie Wittmer. Roseanne barr is under fire after a racist tweet about former obama adviser valerie jarrett that compares her to an ape, 2018.
- 63. Ho Chung Wu, Robert Wing Pong Luk, Kam Fai Wong, and Kui Lam Kwok. Interpreting tf-idf term weights as making relevance decisions. ACM Transactions on Information Systems (TOIS), 26(3):13, 2008.
- 64. Dawei Yin, Zhenzhen Xue, Liangjie Hong, Brian D Davison, April Kontostathis, and Lynne Edwards. Detection of harassment on web 2.0. Proceedings of the Content Analysis in the WEB, 2:1–7, 2009.
- 65. Arjumand Younus, M Atif Qureshi, Josephine Griffith, Colm O’Riordan, and Gabriella Pasi. A study into the correlation between narcissism and facebook communication patterns. In 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), volume 1, pages 511–514. IEEE, 2015.
- 66. Daphney-Stavroula Zois, Angeliki Kapodistria, Mengfan Yao, and Charalampos Chelmis. Optimal online cyberbullying detection. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2017–2021. IEEE, 2018.