Automatic Detection of Cyberbullying in Social Media Text

01/17/2018 ∙ by Cynthia Van Hee, et al. ∙ 0

While social media offer great communication opportunities, they also increase the vulnerability of young people to threatening situations online. Recent studies report that cyberbullying constitutes a growing problem among youngsters. Successful prevention depends on the adequate detection of potentially harmful messages and the information overload on the Web requires intelligent systems to identify potential risks automatically. The focus of this paper is on automatic cyberbullying detection in social media text by modelling posts written by bullies, victims, and bystanders of online bullying. We describe the collection and fine-grained annotation of a training corpus for English and Dutch and perform a series of binary classification experiments to determine the feasibility of automatic cyberbullying detection. We make use of linear support vector machines exploiting a rich feature set and investigate which information sources contribute the most for this particular task. Experiments on a holdout test set reveal promising results for the detection of cyberbullying-related posts. After optimisation of the hyperparameters, the classifier yields an F1-score of 64 respectively, and considerably outperforms baseline systems based on keywords and word unigrams.

READ FULL TEXT VIEW PDF
POST COMMENT

Comments

There are no comments yet.

Authors

page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Web 2.0 has had a substantial impact on communication and relationships in today’s society. Children and teenagers go online more frequently, at younger ages, and in more diverse ways (e.g. smartphones, laptops and tablets). Although most of teenagers’ Internet use is harmless and the benefits of digital communication are evident, the freedom and anonymity experienced online makes young people vulnerable, with cyberbullying being one of the major threats (Livingstone ., 2010; Tokunaga, 2010; Livingstone ., 2013).

Bullying is not a new phenomenon, and cyberbullying has manifested itself as soon as digital technologies have become primary communication tools. On the positive side, social media like blogs, social networking sites (e.g. Facebook) and instant messaging platforms (e.g. WhatsApp) make it possible to communicate with anyone and at any time. Moreover, they are a place where people engage in social interaction, offering the possibility to establish new relationships and maintain existing friendships (Gross ., 2002; Mckenna  Bargh, 1999). On the negative side however, social media increase the risk of children being confronted with threatening situations including grooming or sexually transgressive behaviour, signals of depression and suicidal thoughts, and cyberbullying. Users are reachable 24/7 and are often able to remain anonymous if desired: this makes social media a convenient way for bullies to target their victims outside the school yard.

With regard to cyberbullying, a number of national and international initiatives have been launched over the past few years to increase children’s online safety. Examples include KiVa111http://www.kivaprogram.net/, a Finnish cyberbullying prevention programme, the ‘Non au harcèlement’ campaign in France, Belgian governmental initiatives and helplines (e.g. clicksafe.be, veiligonline.be, mediawijs.be) that provide information about online safety, and so on.

In spite of these efforts, a lot of undesirable and hurtful content remains online. Tokunaga (2010) analysed a body of quantitative research on cyberbullying and observed cybervictimisation rates among teenagers between 20% and 40%.  Juvonen  Gross (2008) focused on 12 to 17 year olds living in the United States and found that no less than 72% of them had encountered cyberbullying at least once within the year preceding the questionnaire. Hinduja  Patchin (2006) surveyed 9 to 26 year olds in the United States, Canada, the United Kingdom and Australia, and found that 29% of the respondents had ever been victimised online. A study among 2,000 Flemish secondary school students (age 12 to 18) revealed that 11% of them had been bullied online at least once in the six months preceding the survey (Van Cleemput ., 2013). Finally, the 2014 large-scale EU Kids Online Report (Online, 2014) published that 20% of 11 to 16 year olds had been exposed to hate messages online. In addition, youngsters were 12% more likely to be exposed to cyberbullying as compared to 2010, clearly demonstrating that cyberbullying is a growing problem.

The prevalence of cybervictimisation depends on the conceptualisation used in describing cyberbullying, but also on research variables such as location and the number and age span of its participants. Nevertheless, the above-mentioned studies demonstrate that online platforms are increasingly used for bullying, which is a cause for concern given its impact. As shown by Cowie (2013); Fekkes . (2006); O’Moore  Kirkham (2001), cyberbullying can have a negative impact on the victim’s self-esteem, academic achievement and emotional well-being. Price  Dalgleish (2010) found that self-reported effects of cyberbullying include negative effects on school grades, feelings like sadness, anger, fear, and depression and in extreme cases, cyberbullying could even lead to self-harm and suicidal thoughts.

The above studies demonstrate that cyberbullying is a serious problem the consequences of which can be dramatic. Successful early detection of cyberbullying attempts is therefore of key importance to youngsters’ mental well-being. However, the amount of information on the Web makes it practically unfeasible for moderators to monitor all user-generated content manually. To tackle this problem, intelligent systems are required that process this information in a fast way and automatically signal potential threats. This way, moderators can respond quickly and prevent threatening situations from escalating. According to recent research, teenagers are generally in favour of such automatic monitoring, provided that effective follow-up strategies are formulated, and that privacy and autonomy are guaranteed (Van Royen ., 2014).

Parental control tools (e.g. NetNanny222https://www.netnanny.com/) already block unsuited or undesirable content and some social networks make use of keyword-based moderation tools (i.e., using lists of profane and insulting words to flag harmful content). However, such approaches typically fail to detect implicit or subtle forms of cyberbullying in which no explicit vocabulary is used. There is therefore a need for intelligent and self-learning systems that can go beyond keyword spotting and hence improve recall of cyberbullying detection.

The ultimate goal of this sort of research is to develop models which could improve manual monitoring for cyberbullying on social networks. We explore the automatic detection of textual signals of cyberbullying, in which it is approached as a complex phenomenon that can be realised in various ways (see Section 3.5 for a detailed overview). While a lot of the related research focuses on the detection of cyberbullying ‘attacks’, the present study takes into account a broader range of textual signals of cyberbullying, including posts written by bullies, as well as by victims and bystanders.

We propose a machine learning method to cyberbullying detection by making use of a linear SVM classifier 

(Chang  Lin, 2011; Cortes  Vapnik, 1995) exploiting a varied set of features. To the best of our knowledge, this is the first approach to the annotation of fine-grained text categories related to cyberbullying and the detection of signals of cyberbullying events. It is also the first elaborate research on automatic cyberbullying detection on Dutch social media. For the present experiments, we focus on an English and Dutch ASKfm333https://ask.fm/ corpus, but the methodology adopted is language and genre independent, provided there is annotated data available.

The remainder of this paper is structured as follows: the next section presents a theoretic overview and gives an overview of the state of the art in cyberbullying detection, whereas Section 3 describes the corpus. Next, we present the experimental setup and discuss our experimental results. Finally, Section 6 concludes this paper and provides perspectives for further research.

2 Related Research

Cyberbullying is a widely covered topic in the realm of social sciences and psychology. A fair amount of research has been done on the definition and prevalence of the phenomenon (Hinduja  Patchin, 2012; Livingstone ., 2010; Slonje  Smith, 2008), the identification of different forms of cyberbullying (O’Sullivan  Flanagin, 2003; Vandebosch  Van Cleemput, 2009; Willard, 2007), and its consequences (Cowie, 2013; Price  Dalgleish, 2010; Smith ., 2008). In contrast to the efforts made in defining and measuring cyberbullying, the number of studies that focus on its annotation and automatic detection, is limited (Nadali ., 2013). Nevertheless, some important advances have been made in the domain over the past few years.

2.1 A Definition of Cyberbullying

Many social and psychological studies have worked towards a definition of cyberbullying. A common starting point for conceptualising cyberbullying are definitions of traditional (or offline) bullying. Seminal work has been published by (Olweus, 1993; Nansel ., 2001; Salmivalli ., 1999; Wolak ., 2007), who describe bullying based on three main criteria, including i) intention (i.e., a bully intends to inflict harm on the victim), ii) repetition (i.e., bullying acts take place repeatedly over time) and iii) a power imbalance between the bully and the victim (i.e., a more powerful bully attacks a less powerful victim). With respect to cyberbullying, a number of definitions are based on the above-mentioned criteria. A popular definition is that of Smith . (2008, p. 376) which describes cyberbullying as “an aggressive, intentional act carried out by a group or individual, using electronic forms of contact, repeatedly and over time, against a victim who cannot easily defend him or herself”.

Nevertheless, some studies have underlined the differences between offline and online bullying, and have therefore questioned the relevance of the three criteria to the latter. Besides theoretical objections, a number of practical limitations have been observed. Firstly, while Olweus (1993) claims intention to be inherent to traditional bullying, this is much harder to ascertain in an online environment. Online conversations lack the signals of a face-to-face interaction like intonation, facial expressions and gestures, which makes them more ambiguous than real-life conversations. The receiver may therefore get the wrong impression that they are being offended or ridiculed (Vandebosch  Van Cleemput, 2009). Another criterion for bullying that might not hold in online situations, is the power imbalance between bully and victim. Although this can be evident in real life (e.g. the bully is larger, stronger, older than the victim), it is hard to conceptualise or measure in an online environment. It may be related to technological skills, anonymity or the inability of the victim to get away from the bullying Dooley  Cross (2009); Slonje  Smith (2008); Vandebosch  Van Cleemput (2008). Empowering for the bully are also inherent characteristics of the Web: once defamatory or confidential information about a person is made public through the Internet, it is hard, if not impossible, to remove.

Finally, while arguing that repetition is a criterion to distinguish cyberbullying from single acts of aggression,  Olweus (1993) himself states that such a single aggressive action can be considered bullying under certain circumstances, although it is not entirely clear what these circumstances involve. Accordingly, Dooley  Cross (2009) claim that repetition in cyberbullying is problematic to operationalise, as it is unclear what the consequences are of a single derogatory message on a public page. A single act of aggression or humiliation may result in continued distress and humiliation for the victim if it is shared or liked by multiple perpetrators or read by a large audience. Slonje . (2013, p. 26) compare this with a ‘snowball effect’: one post may be repeated or distributed by other people so that it becomes out of the control of the initial bully and has larger effects than was originally intended.

Given these arguments, a number of less ‘strict’ definitions of cyberbullying were postulated by among others (Hinduja  Patchin, 2006; Juvonen  Gross, 2008; Tokunaga, 2010), where a power imbalance and repetition are not deemed necessary conditions for cyberbullying.

The above paragraphs demonstrate that defining cyberbullying is far from trivial, and varying prevalence rates (cf. Section 1) confirm that a univocal definition of the phenomenon is still lacking in the literature (Tokunaga, 2010). Based on existing conceptualisations, we define cyberbullying as content that is published online by an individual and that is aggressive or hurtful against a victim. Based on this definition, an annotation scheme was developed (see Van Hee, Verhoeven . (2015)) to signal textual characteristics of cyberbullying, including posts from bullies, as well as reactions by victims and bystanders.

2.2 Detecting and Preventing Cyberbullying

As mentioned earlier, although research on cyberbullying detection is more limited than social studies on the phenomenon, some important advances have been made in recent years. In what follows, we present a brief overview of the most important natural language processing approaches to cyberbullying detection.

Although some studies have investigated the effectiveness of rule-based modelling (Reynolds ., 2011), the dominant approach to cyberbullying detection involves machine learning. Most machine learning approaches are based on supervised (Dadvar, 2014; Dinakar ., 2011; Yin ., 2009)

or semi-supervised learning  

(Nahar ., 2014). The former involves the construction of a classifier based on labeled training data, whereas semi-supervised approaches rely on classifiers that are built from a training corpus containing a small set of labeled and a large set of unlabelled instances (a method that is often used to handle data sparsity). As cyberbullying detection essentially involves the distinction between bullying and non-bullying posts, the problem is generally approached as a binary classification task where the positive class is represented by instances containing (textual) cyberbullying, while the negative class includes instances containing non-cyberbullying or ‘innocent’ text.

A key challenge in cyberbullying research is the availability of suitable data, which is necessary to develop models that characterise cyberbullying. In recent years, only a few datasets have become publicly available for this particular task, such as the training sets provided in the context of the CAW 2.0 workshop444http://caw2.barcelonamedia.org/ and more recently, the Twitter Bullying Traces dataset (Sui, 2015). As a result, several studies have worked with the former or have constructed their own corpus from social media websites that are prone to bullying content, such as YouTube (Dadvar, 2014; Dinakar ., 2011), Formspring555https://spring.me/ (Dinakar ., 2011), and ASKfm  (Van Hee, Lefever ., 20152) (the latter two are social networking sites where users can send each other questions or respond to them). Despite the bottleneck of data availability, existing approaches to cyberbullying detection have shown its potential, and the relevance of automatic text analysis techniques to ensure child safety online has been recognised (Desmet, 2014; Royen ., 2016).

Among the first studies on cyberbullying detection are Yin . (2009); Reynolds . (2011); Dinakar . (2011), who explored the predictive power of

-grams (with and without tf-idf weighting), part-of-speech information (e.g. first and second pronouns), and sentiment information based on profanity lexicons for this task. Similar features were also exploited for the detection of cyberbullying events and fine-grained text categories related to cyberbullying 

(Van Hee, Lefever ., 20152, 20151). More recent studies have demonstrated the added value of combining such content-based features with user-based information, such as including users’ activities on a social network (i.e., the number of posts), their age, gender, location, number of friends and followers, and so on (Dadvar, 2014; Nahar ., 2014; Al-garadi ., 2016). Moreover, semantic features have been explored to further improve classification performance of the task. To this end, topic model information (Xu ., 2012), as well as semantic relations between -grams (according to a Word2Vec model (Zhao ., 2016)) have been integrated.

As mentioned earlier, data collection remains a bottleneck in cyberbullying research. Although cyberbullying has been recognised as a serious problem (cf. Section 1), real-world examples are often hard to find in public platforms. Naturally, the vast majority of communications do not contain traces of verbal aggression or transgressive behaviour. When constructing a corpus for machine learning purposes, this results in imbalanced datasets, meaning that one class (e.g. cyberbullying posts) is much less represented in the corpus than the other (e.g. non-cyberbullying posts). To tackle this problem, several studies have adopted resampling techniques (Nahar ., 2014; Al-garadi ., 2016; Reynolds ., 2011) that create synthetic minority class examples or reduce the number of negative class examples (i.e., minority class oversampling and majority class undersampling (Chawla ., 2002)).

Table 1 presents a number of recent studies on cyberbullying detection, providing insight into the state of the art in cyberbullying research and the contribution of the current research to the domain.

The studies discussed in this section have demonstrated the feasibility of automatic cyberbullying detection in social media data by making use of a varied set of features. Most of them have, however, focussed on cyberbullying ‘attacks’, or posts written by a bully. Moreover, it is not entirely clear if different forms of cyberbullying have been taken into account (e.g. sexual intimidation or harassment, or psychological threats), in addition to derogatory language or insults.

In the research described in this paper, cyberbullying is considered a complex phenomenon consisting of different forms of harmful behaviour online, which are described in more detail in our annotation scheme (Van Hee, Verhoeven ., 2015). Purposing to facilitate manual monitoring efforts on social networks, we develop a system that automatically detects signals of cyberbullying, including attacks from bullies, as well as victim and bystander reactions. Similarly, Xu . (2012) investigated bullying traces posted by different author roles (accuser, bully, reporter, victim). However, they collected tweets by using specific keywords (i.e., bully, bullied and bullying). As a result, their corpus contains many reports or testimonials of a cyberbullying incident (example 1), instead of actual signals that cyberbullying is going on. Moreover, their method implies that cyberbullying-related content devoid of such keywords will not be part of the training corpus.

  1. ‘Some tweens got violent on the n train, the one boy got off after blows 2 the chest… Saw him cryin as he walkd away :( bullying not cool’ (Xu ., 2012, p. 658)

For this research, English and Dutch social media data were annotated for different forms of cyberbullying, based on the actors involved in a cyberbullying incident. After preliminary experiments for Dutch (Van Hee, Lefever ., 20152, 20151), we currently explore the viability of detecting cyberbullying-related posts in Dutch and English social media. To this end, binary classification experiments are performed exploiting a rich feature set and optimised hyperparameters.

Data source Balanced? Classifier Features Result
(Reynolds ., 2011) Formspring       ✓ J48 lexical Recall=62%, Acc=82%
(Dadvar, 2014) YouTube       ✗ SVM (linear) lexical + semantic + user-based F=64%
(Dinakar ., 2012) YouTube       ✓ SVM (polynomial) lexical F=77%
(Yin ., 2009) Kongregate, Slashdot, MySpace       ✗ SVM (linear) lexical F=48%
(Nahar ., 2014) Kongregate, Slashdot, MySpace       ✗ semi-supervised fuzzy SVM lexical + user-based F=47%
(Van Hee, Lefever ., 20152) ASKfm       ✗ SVM (linear) lexical F=55%
(Al-garadi ., 2016) Twitter       ✗ Random Forest + SMOTE lexical + semantic + user-based + Twitter-based F=94% AUC=94%
(Xu ., 2012) Twitter      ✓ SVM (linear) lexical F=77%
(Zhao ., 2016) Twitter      ✗ SVM (linear) lexical + semantic F=78%
Table 1: State-of-the-art approaches to cyberbullying detection.

3 Data Collection and Annotation

To be able to build representative models for cyberbullying, a suitable dataset is required. This section describes the construction of two corpora, English and Dutch, containing social media posts that are manually annotated for cyberbullying according to our fine-grained annotation scheme. This allows us to develop a detection system covering different forms and participants (or roles) involved in a cyberbullying event.

3.1 Data Collection

Two corpora were constructed by collecting data from the social networking site ASKfm, where users can create profiles and ask or answer questions, with the option of doing so anonymously. ASKfm data typically consists of question-answer pairs published on a user’s profile. The data were retrieved by crawling a number of seed profiles using the GNU Wget software666https://www.gnu.org/software/wget/ in April and October, 2013. After language filtering (i.e., non-English or non-Dutch content was removed), the experimental corpora comprised 113,698 and 78,387 posts for English and Dutch, respectively.

3.2 Data Annotation

Cyberbullying has been a widely covered research topic recently and studies have shed light on direct and indirect types of cyberbullying, implicit and explicit forms, verbal and non-verbal cyberbullying, and so on. This is important from a sociolinguistic point of view, but knowing what cyberbullying involves is also crucial to build models for automatic cyberbullying detection. In the following paragraphs, we present our data annotation guidelines(Van Hee, Verhoeven ., 2015) and focus on different types and roles related to the phenomenon.

3.3 Types of Cyberbullying

Cyberbullying research is mainly centered around the conceptualisation, occurrence and prevention of the phenomenon (Hinduja  Patchin, 2012; Livingstone ., 2010; Slonje  Smith, 2008). Additionally, different forms of cyberbullying have been identified (O’Sullivan  Flanagin, 2003; Price  Dalgleish, 2010; Willard, 2007) and compared with forms of traditional or offline bullying (Vandebosch  Van Cleemput, 2009). Like traditional bullying, direct and indirect forms of cyberbullying have been identified. Direct cyberbullying refers to actions in which the victim is directly involved (e.g. sending a virus-infected file, excluding someone from an online group, insulting and threatening), whereas indirect cyberbullying can take place without awareness of the victim (e.g. outing or publishing confidential information, spreading gossip, creating a hate page on social networking sites) (Vandebosch  Van Cleemput, 2009).

The present annotation scheme describes some specific textual categories related to cyberbullying, including threats, insults, defensive statements from a victim, encouragements to the harasser, etc. (see Section 3.5 for a complete overview). All of these forms were inspired by social studies on cyberbullying(Van Cleemput ., 2013; Vandebosch  Van Cleemput, 2009) and manual inspection of cyberbullying examples.

3.4 Roles in Cyberbullying

Similarly to traditional bullying, cyberbullying involves a number of participants that adopt well-defined roles. Researchers have identified several roles in (cyber)bullying interactions. Although traditional studies on bullying have mainly concentrated on bullies and victims (Salmivalli ., 1996), the importance of bystanders in a bullying episode has been acknowledged (Bastiaensens ., 2014; Salmivalli, 2010). Bystanders can support the victim and mitigate the negative effects caused by the bullying (Salmivalli, 2010), especially on social networking sites, where they hold higher intentions to help the victim than in real life conversations (Bastiaensens ., 2015). While Salmivalli . (1996) distinguish four different bystanders, Vandebosch . (2006) distinguish three main types: i) bystanders who participate in the bullying, ii) who help or support the victim and iii) those who ignore the bullying. Given that passive bystanders are hard to recognise in online text, only the former two are included in our annotation scheme.

3.5 Annotation Guidelines

To operationalise the task of automatic cyberbullying detection, we developed and tested a fine-grained annotation scheme and applied it to our corpora. While a detailed overview of the guidelines is presented in our technical report (Van Hee, Verhoeven ., 2015), we briefly present the categories and main annotation steps below.

  • Threat/Blackmail: expressions containing physical or psychological threats or indications of blackmail.

  • Insult: expressions meant to hurt or offend the victim.

    • General insult: general expressions containing abusive, degrading or offensive language that are meant to insult the addressee.

    • Attacking relatives: insulting expressions towards relatives or friends of the victim.

    • Discrimination: expressions of unjust or prejudicial treatment of the victim. Two types of discrimination are distinguished (i.e., sexism and racism). Other forms of discrimination should be categorised as general insults.

  • Curse/Exclusion: expressions of a wish that some form of adversity or misfortune will befall the victim and expressions that exclude the victim from a conversation or a social group.

  • Defamation: expressions that reveal confident or defamatory information about the victim to a large public.

  • Sexual Talk: expressions with a sexual meaning or connotation. A distinction is made between innocent sexual talk and sexual harassment.

  • Defense: expressions in support of the victim, expressed by the victim himself or by a bystander.

    • Bystander defense: expressions by which a bystander shows support for the victim or discourages the harasser from continuing his actions.

    • Victim defense: assertive or powerless reactions from the victim.

  • Encouragement to the harasser: expressions in support of the harasser.

  • Other: expressions that contain any other form of cyberbullying-related behaviour than the ones described here.

Based on the literature on role-allocation in cyberbullying episodes (Salmivalli ., 2011; Vandebosch ., 2006), four roles are distinguished, including victim, bully, and two types of bystanders.

  1. Harasser or Bully: person who initiates the bullying.

  2. Victim: person who is harassed.

  3. Bystander-defender: person who helps the victim and discourages the harasser from continuing his actions.

  4. Bystander-assistant: person who does not initiate, but helps or encourages the harasser.

Essentially, the annotation scheme describes two levels of annotation. Firstly, the annotators were asked to indicate, at the post level, whether the post under investigation was related to cyberbullying. If the post was considered a signal of cyberbullying, annotators identified the author’s role. Secondly, at the subsentence level, the annotators were tasked with the identification of a number of fine-grained text categories related to cyberbullying. More concretely, they identified all text spans corresponding to one of the categories described in the annotation scheme. To provide the annotators with some context, all posts were presented within their original conversation when possible. All annotations were done using the Brat rapid annotation tool (Stenetorp ., 2012), some examples of which are presented in Table 2.

Annotation Annotation example
category
Threat/blackmail [I am going to find out who you are & I swear you are going to regret it.]
Insult [Kill yourself] [you fucking mc slut !!!!] [NO ONE LIKES YOU !!!!!] [You are an ugly useless little whore !!!!]
Curse/Exclusion [Fuck you.] [Now shush I don’t wanna hear anything.]
Defamation [She slept with her ex behind his girlfiends back and she and him had broken up.]
Sexual Talk [Naked pic of you now.]
Defense [I would appreciate if you dindn’t talk shit about my bestfriend.] He has enough to deal with already.
Encour. to har. [She is a massive slut] [i agree with you @user she is!] [LOL AT HER mate, im on your side]
Table 2: Definitions and brat annotation examples of more fine-grained text
categories related to cyberbullying.

3.6 Annotation Statistics

The English and Dutch corpora were independently annotated for cyberbullying by trained linguists. All were Dutch native speakers and English second-language speakers. To demonstrate the validity of our guidelines, inter-annotator agreement scores were calculated using Kappa on a subset of each corpus. Inter-rater agreement for Dutch (2 raters) is calculated using Cohen’s Kappa (Cohen, 1960). Fleiss’ Kappa (Fleiss, 1971) is used for the English corpus ( 2 raters). Kappa scores for the identification of cyberbullying are = 0.69 (Dutch) and = 0.59 (English).

As shown in Table 3, inter-annotator agreement for the identification of the more fine-grained categories for English varies from fair to substantial (McHugh, 2012), except for defamation, which appears to be more difficult to recognise. No encouragements to the harasser were present in this subset of the corpus. For Dutch, the inter-annotator agreement is fair to substantial, except for curse and defamation. Analysis revealed that one of both annotators often annotated the latter as an insult, and in some cases even did not consider it as cyberbullying-related.

Threat Insult Defense Sexual Curse/ Defamation Encouragements
Talk Exclusion to the harasser
English 0.65 0.63 0.45 0.38 0.58 0.15 N/A
Dutch 0.52 0.66 0.63 0.53 0.19 0.00 0.21
Table 3: Inter-annotator agreement on the fine-grained categories related to cyberbullying.

In short, the inter-rater reliability study shows that the annotation of cyberbullying is not trivial and that more fine-grained categories like defamation, curse and encouragements are sometimes hard to recognise. It appears that defamations were sometimes hard to distinguish from insults, whereas curses and exclusions were sometimes considered insults or threats. The analysis further reveals that encouragements to the harasser are subject to interpretation. Some are straightforward (e.g. ‘I agree we should send her hate’), whereas others are subject to the annotator’s judgement and interpretation (e.g. ‘hahaha’, ‘LOL’).

4 Experimental Setup

In this paper, we explore the feasibility of automatically recognising signals of cyberbullying. A crucial difference with state-of-the-art approaches to cyberbullying detection is that we aim to model bullying attacks, as well as reactions from victims and bystanders (i.e., all under one binary label ‘signals of cyberbullying’), since these could likewise indicate that cyberbullying is going on. The experiments described in this paper focus on the detection of such posts, which are signals of a potential cyberbullying event to be further investigated by human moderators.

The English and Dutch corpus contain 113,698 and 78,387 posts, respectively. As shown in Table 4, the experimental corpus features a heavily imbalanced class distribution with the large majority of posts not being part of cyberbullying. In classification, this class imbalance can lead to decreased performance. We apply cost-sensitive SVM as a possible hyperparameter in optimisation to counter this. The cost-sensitive SVM reweighs the penalty parameter of the error term by the inverse class-ratio. This means that misclassifications of the minority positive class are penalised more than classification errors on the majority negative class. Other pre-processing methods to handle data imbalance in classification include feature filtering metrics and data resampling (He  Garcia, 2009). These methods were omitted as they were found to be too computationally expensive given our high-dimensional dataset.

Corpus size Number (ratio) of
bullying posts
English 113,698 5,375 (4.73%)
Dutch 78,387 5,106 (6.97%)
Table 4: Statistics of the English and Dutch cyberbullying corpus.

For the automatic detection of cyberbullying, we performed binary classification experiments using a linear kernel support vector machine (SVM) implemented in LIBLINEAR (Fan ., 2008) by making use of Scikit-learn (Pedregosa ., 2011), a machine learning library for Python. The motivation behind this is twofold: i) support vector machines (SVMs) have proven to work well for tasks similar to the ones under investigation (Desmet, 2014) and ii) LIBLINEAR allows fast training of large-scale data which allow for a linear mapping (which was confirmed after a series of preliminary experiments using LIBSVM with linear, RBF and polynomial kernels).

The classifier was optimised for feature type (cf. Section 4.1) and hyperparameter combinations (cf. Table 5). Model selection was done using -fold cross validation in grid search over all possible feature types (i.e., groups of similar features, like different orders of -gram bag-of-words features) and hyperparameter configurations. The best performing hyperparameters are selected by F-score on the positive class. The winning model is then retrained on all held-in data and subsequently tested on a hold-out test set to assess whether the classifier is over- or under-fitting. The holdout represents a random sample (

) of all data. The folds were randomly stratified splits over the hold-in class distribution. Testing all feature type combinations is a rudimentary form of feature selection and provides insight into which types of features work best for this particular task.

Feature selection over all individual features was not performed because of the large feature space (NL: 795,072 and EN: 871,296 individual features). Hoste (2005), among other researchers, demonstrated the importance of joint optimisation, where feature selection and hyperparameter optimisation are performed simultaneously, since the techniques mutually influence each other.

The optimised models are evaluated against two baseline systems: i) an unoptimised linear-kernel SVM (configured with default parameter settings) based on word -grams only and, ii) a keyword-based system that marks posts as positive for cyberbullying if they contain a word from existing vocabulary lists composed by aggressive language and profanity terms.

Hyperparameter Values
Penalty of error term
Loss function Hinge, squared hinge
Penalty: norm used in penalisation ‘l1’ (‘least absolute deviations’) or ‘l2’ (‘least squares’)
Class weight (sets penalty of class to weight*) None or ‘balanced’, i.e., weight inversely proportional to class frequencies
Table 5: Hyperparameters in grid-search model selection.

4.1 Pre-processing and Feature Engineering

As pre-processing, we applied tokenisation, PoS-tagging and lemmatisation to the data using the LeTs Preprocess Toolkit (van de Kauter ., 2013). In supervised learning, a machine learning algorithm takes a set of training instances (of which the label is known) and seeks to build a model that generates a desired prediction for an unseen instance. To enable the model construction, all instances are represented as a vector of features (i.e., inherent characteristics of the data) that contain information that is potentially useful to distinguish cyberbullying from non-cyberbullying content.

We experimentally tested whether cyberbullying events can be recognised automatically by lexical markers in a post. To this end, all posts were represented by a number of information sources (or features

) including lexical features like bags-of-words, sentiment lexicon features and topic model features, which are described in more detail below. Prior to feature extraction, some data cleaning steps were executed, such as the replacement of hyperlinks and @-replies, removal of superfluous white spaces, and the replacement of abbreviations by their full form (based on an existing mapping dictionary

777http://www.chatslang.com/terms/abbreviations/). Additionally, tokenisation was applied before -gram extraction and sentiment lexicon matching, and stemming was applied prior to extracting topic model features.

After pre-processing of the corpus, the following feature types were extracted:

  • Word -gram bag-of-words: binary features indicating the presence of word unigrams, bigrams and trigrams.

  • Character -gram bag-of-words: binary features indicating the presence of character bigrams, trigrams and fourgrams (without crossing word boundaries). Character -grams provide some abstraction from the word level and provide robustness to the spelling variation that characterises social media data.

  • Term lists: one binary feature derived for each one out of six lists, indicating the presence of an item from the list in a post: proper names, ‘allness’ indicators (e.g. always, everybody), diminishers (e.g. slightly, relatively), intensifiers (e.g. absolutely, amazingly), negation words and aggressive language and profanity words. Person alternation is a binary feature indicating whether the combination of a first and second person pronoun occurs in order to capture interpersonal intent.

  • Subjectivity lexicon features: positive and negative opinion word ratios, as well as the overall post polarity were calculated using existing sentiment lexicons. For Dutch, we made use of the Duoman (Jijkoun  Hofmann, 2009) and Pattern (De Smedt  Daelemans, 2012) lexicons. For English, we included the Hu and Liu opinion lexicon (Hu  Liu, 2004), the MPQA lexicon (Wilson ., 2005), General Inquirer Sentiment Lexicon (Stone ., 1966), AFINN (Nielsen, 2011), and MSOL (Mohammad ., 2009). For both languages, we included the relative frequency of all 68 psychometric categories in the Linguistic Inquiry and Word Count (LIWC) dictionary for English(Pennebaker ., 2001) and Dutch (Zijlstra ., 2004).

  • Topic model features: by making use of the Gensim topic modelling library (Rehurek  Sojka, 2010), several LDA (Blei ., 2003) and LSI (Deerwester ., 1990) topic models with varying granularity ( = 20, 50, 100 and 200) were trained on data corresponding to each fine-grained category of a cyberbullying event (e.g. threats, defamations, insults, defenses). The topic models were based on a background corpus (EN: tokens, NL: tokens) scraped with the BootCAT (Baroni  Bernardini, 2004) web-corpus toolkit. BootCaT collects ASKfm user profiles using lists of manually determined seed words that are characteristic of the cyberbullying categories.

When applied to the training data, this resulted in and features for English and Dutch, respectively.

5 Results

In this section, we present the results of our experiments on the automatic detection of cyberbullying-related posts in an English (EN) and Dutch (NL) corpus of ASKfm posts. Ten-fold cross-validation was performed in exhaustive grid-search over different feature type and hyperparameter combinations (see Section 4). The unoptimised word -gram-based classifier and keyword-matching system serve as baselines for comparison. Precision, Recall and F

performance metrics were calculated on the positive class (i.e., ‘binary averaging’). We also report Area Under the ROC curve (AUC) scores, a performance metric that is more robust to data imbalance than precision, recall and micro-averaged F-score 

(Fawcett, 2006).

Feature combination Cross-validation scores Holdout scores
F P R Acc AUC F P R Acc AUC
EN
B + C + D + E 64.26 73.32 57.19 96.97 78.07 63.69 74.13 55.82 97.21 77.47
A + B + C 64.24 73.22 57.23 96.96 78.09 64.32 74.08 56.83 97.24 77.96
A + C + E 63.84 73.21 56.59 96.94 77.78 62.94 72.82 55.42 97.14 77.24
word -gram baseline 58.17 67.55 51.07 96.54 74.93 59.63 69.57 52.17 96.57 75.50
profanity baseline 17.17 9.61 80.14 63.73 71.53 17.61 9.90 78.51 63.79 71.34
NL
A + B + C + E 61.20 56.76 66.40 94.47 81.42 58.13 54.03 62.90 94.58 79.75
A + B + C + D + E 61.03 71.55 53.20 95.53 75.86 58.72 67.40 52.03 95.62 75.21
A + C + E 60.82 71.66 52.84 95.53 75.68 58.15 67.71 50.96 95.61 74.71
word -gram baseline 50.39 67.80 40.09 94.81 69.38 49.54 64.29 40.30 95.09 69.44
profanity baseline 28.46 19.24 54.66 81.99 69.28 25.13 16.73 50.53 81.99 67.26
Table 6: Cross-validated and holdout scores (%) according to different
metrics (F, precision, recall, accuracy and area under the curve)
for the English and Dutch top 3 combined feature type systems.
A word -grams
B subjectivity lexicons
C character -grams
D term lists
E topic models
Table 7: Feature group mapping (Table 6)

Table 6 gives us an indication of which feature type combinations score best and hence contribute most to this task. A total of 31 feature type combinations, each with 28 different hyperparameter sets have been tested. Table 6 shows the results for the three best scoring systems by included feature types with optimised hyperparameters. The maximum attained F-score in cross-validation is 64.26% for English and 61.20% for Dutch and shows that the classifier benefits from a variety of feature types. The results on the holdout test set show that the trained systems generalise well on unseen data, indicating little under- or overfitting. The simple keyword-matching baseline system has the lowest performance for both languages even though it obtains high recall for English, suggesting that profane language characterises many cyberbullying-related posts. Feature group and hyperparameter optimisation provides a considerable performance increase over the unoptimised word -gram baseline system. The top-scoring systems for each language do not differ a lot in performance, except the best system for Dutch, which trades recall for precision when compared to the runner-ups.

Table 8 presents the scores of the (hyperparameter-optimised) single feature type systems, to gain insight into the performance of these feature types when used individually. Analysis of the combined and single feature type sets reveals that word -grams, character -grams, and subjectivity lexicons prove to be strong features for this task. In effect, adding character -grams always improved classification performance for both languages. They likely provide robustness to lexical variation in social media text, as compared to word -grams. While subjectivity lexicons appear to be discriminative features, term lists perform badly on their own as well as in combinations for both languages. This shows once again (cf. profanity baseline) that cyberbullying detection requires more sophisticated information sources than profanity lists. Topic models seem to do badly for both languages on their own, but in combination, they improve Dutch performance consistently. A possible explanation for their varying performance in both languages would be that the topic models trained on the Dutch background corpus are of better quality than the English ones. In effect, a random selection of background corpus texts reveals that the English scrape contains more noisy data (i.e., low word-count posts and non-English posts) than the Dutch data.

Feature type Cross-validation scores holdout scores
F P R Acc AUC F P R Acc AUC
EN
word -grams 60.09 60.49 59.69 96.22 78.87 58.35 57.12 59.64 96.27 78.79
subjectivity lexicons 56.82 73.32 46.38 96.64 72.77 56.16 72.61 45.78 96.87 72.50
character -grams 52.69 58.70 47.80 95.91 73.06 53.33 62.37 46.59 96.43 72.65
term lists 40.48 38.98 42.12 94.10 69.41 39.56 39.56 39.56 94.71 68.39
topic models 17.35 9.73 79.91 63.72 71.41 15.70 8.72 78.51 63.07 70.44
NL
word -grams 55.53 72.64 44.94 95.27 71.88 54.99 70.20 45.20 95.57 71.99
subjectivity lexicons 54.34 54.12 54.56 93.97 75.65 51.82 50.61 53.09 94.09 74.90
character -grams 51.70 67.58 41.86 94.86 70.22 50.46 65.20 41.15 95.17 69.88
term lists 28.65 19.36 55.10 81.97 69.48 25.13 16.73 50.53 81.99 67.26
topic models 24.74 21.24 29.61 88.16 60.94 17.99 23.15 14.71 91.98 55.80
Table 8: Cross-validated and holdout scores (%) according to different
metrics (F, precision, recall, accuracy and area under the ROC curve)
for English and Dutch single feature type systems.

A shallow qualitative analysis of the classification output provided insight into some of the classification mistakes.

Table 9 gives an overview of the error rates per cyberbullying category of the best performing and baseline systems. This could give an indication of which types of bullying the current system has trouble classifying. All categories are always considered positive for cyberbullying (i.e., the error rate equals the false negative rate), except for Sexual and Insult which can also be negative (in case of harmless sexual talk and ‘socially acceptable’ insulting language like ‘hi bitches, in for a movie?’ the corresponding category was indicated, but the post itself was not annotated as cyberbullying) and Not cyberbullying, which is always negative. Error rates often being lowest for the profanity baseline confirms that it performs particularly well in terms of recall (at the expense of precision, see Table 8) When looking at the best system for both languages, we see that Defense is the hardest category to correctly classify. This should not be a surprise as the category comprises defensive posts from bystanders and victims, which contain less aggressive language than cyberbullying attacks and are often shorter in length than the latter. Assertive defensive posts (i.e., a subcategory of Defense) that attack the bully) are, however, more often correctly classified. There are not enough instances of Encouragement for either language in the holdout to be representative. In both languages, threats, curses and incidences of sexual harassment are most easily recognisable, showing (far) lower error rates than the categories Defamation, Defense, Encouragements to the harasser, and Insult.

Qualitative error analysis of the English and Dutch predictions reveals that false positives often contain aggressive language directed at a second person, often denoting personal flaws or containing sexual and profanity words. We see that misclassifications are often short posts containing just a few words and that false negatives often lack explicit verbal signs of cyberbullying (e.g. insulting or profane words) or are ironic (examples 2 and 3). Additionally, we see that cyberbullying posts containing misspellings or grammatical errors and incomplete words are also hard to recognise as such (examples 4 and 5). The Dutch and English data are overall similar with respect to qualitative properties of classification errors.

  1. You might want to do some sports ahah x

  2. Look who is there… my thousandth anonymous hater, congratulations!

  3. ivegot 1 word foryou… yknow whatit is? slut

  4. One word for you: G - A - …

Category Nr. occurrences Profanity Word -gram Best system
in holdout baseline baseline
EN
Curse =109 14.68 30.28 24.77
Defamation =21 23.81 47.62 38.10
Defense =165 22.42 52.12 43.64
Encouragement =1 0.00 100.00 100.00
Insult =345 26.67 41.74 35.94
Sexual =165 63.80 21.47 21,47
Threat =12 8.33 41.67 25.00
Not cyberbullying =10,714 36.94 1.10 0.76
NL
Curse =96 39.58 50.00 22.92
Defamation =6 100.00 66.67 33.33
Defense =200 52.50 63.50 46.00
Encouragement =5 40.00 60.00 40.00
Insult =355 43.38 47.89 28.17
Sexual =37 37.84 21.62 27.03
Threat =15 33.33 46.67 20.00
Not cyberbullying =7,295 15.63 1.23 3.07
Table 9: Error rates (%) per cyberbullying category on holdout for
English and Dutch systems.

In short, the experiments show that our classifier clearly outperforms both a keyword-based and word -gram baseline. However, analysis of the classifier output reveals that false negatives often lack explicit clues that cyberbullying is going on, indicating that our system might benefit from irony recognition and integrating world knowledge to capture such implicit realisations of cyberbullying.

Given that we present the first elaborate research on detecting signals of cyberbullying regardless of the author role instead of bully posts alone, crude comparison with the state of the art would be irrelevant. We observe, however, that our classifier obtains competitive results compared to Dadvar (2014); Dinakar . (2011); Nahar . (2014); Yin . (2009); Van Hee, Lefever . (20152).

6 Conclusions and Future Research

The goal of the current research was to investigate the automatic detection of cyberbullying-related posts on social media. Given the information overload on the web, manual monitoring for cyberbullying has become unfeasible. Automatic detection of signals of cyberbullying would enhance moderation and allow to respond quickly when necessary.

Cyberbullying research has often focused on detecting cyberbullying ‘attacks’, hence overlooking posts written by victims and bystanders. However, these posts could just as well indicate that cyberbullying is going on. The main contribution of this paper is that it presents a system for detecting signals of cyberbullying on social media, including posts from bullies, victims and bystanders. A manually annotated cyberbullying dataset was created for two languages, which will be made available for public scientific use. Moreover, while a fair amount of research has been done on cyberbullying detection for English, we believe this is one of the first papers that focus on Dutch as well.

A set of binary classification experiments were conducted to explore the feasibility of automatic cyberbullying detection on social media. In addition, we sought to determine which information sources contribute to this task. Two classifiers were trained on English and Dutch ASKfm data and evaluated on a holdout test of the same genre. Our experiments reveal that the current approach is a promising strategy for detecting signals of cyberbullying in social media data automatically. After feature selection and hyperparameter optimisation, the classifiers achieved an F-score of 64.32% and 58.72% for English and Dutch, respectively. The systems hereby significantly outperformed a keyword and an (unoptimised) -gram baseline. Analysis of the results revealed that false positives often include implicit cyberbullying or offenses through irony, the challenge of which will constitute an important area for future work.

Another interesting direction for future work would be the detection of fine-grained cyberbullying-related categories such as threats, curses and expressions of racism and hate. When applied in a cascaded model, the system could find severe cases of cyberbullying with high precision. This would be particularly interesting for monitoring purposes, since it would allow to prioritise signals of bullying that are in urgent need for manual inspection and follow-up.

Finally, future work will focus on the detection of participants (or roles) typically involved in cyberbullying. This would allow to analyse the context of a cyberbullying incident and hence evaluate its severity. When applied as moderation support on online platforms, such a system would allow to provide feedback in function of the recipient (i.e., a bully, victim, or bystander).

7 Acknowledgment

The work presented in this paper was carried out in the framework of the AMiCA IWT SBO-project 120007 project, funded by the government Flanders Innovation & Entrepreneurship (VLAIO) agency.

References

  • Al-garadi . (2016) Algaradi2016Al-garadi, MA., Varathan, KD.  Ravana, SD.  2016. Cybercrime detection in online communications: The experimental case of cyberbullying detection in the Twitter network Cybercrime detection in online communications: The experimental case of cyberbullying detection in the Twitter network. Computers in Human Behavior63433–443. http://dx.doi.org/10.1016/j.chb.2016.05.051
  • Baroni  Bernardini (2004) Baroni2004Baroni, M.  Bernardini, S.  2004. BootCaT: Bootstrapping Corpora and Terms from the Web BootCaT: Bootstrapping Corpora and Terms from the Web. Proceedings of the Fourth International Conference on Language Resources and Evaluation Proceedings of the Fourth International Conference on Language Resources and Evaluation ( 1313–1316).
  • Bastiaensens . (2014) Bastiaensens2014Bastiaensens, S., Vandebosch, H., Poels, K., Van Cleemput, K., DeSmet, A.  De Bourdeaudhuij, I.  2014. Cyberbullying on social network sites. An experimental study into bystanders’ behavioural intentions to help the victim or reinforce the bully Cyberbullying on social network sites. An experimental study into bystanders’ behavioural intentions to help the victim or reinforce the bully. Computers in Human Behavior31259–271.
  • Bastiaensens . (2015) Bastiaensens2015Bastiaensens, S., Vandebosch, H., Poels, K., Van Cleemput, K., DeSmet, A.  De Bourdeaudhuij, I.  2015. ‘Can I afford to help?’ How affordances of communication modalities guide bystanders’ helping intentions towards harassment on social network sites ‘Can I afford to help?’ How affordances of communication modalities guide bystanders’ helping intentions towards harassment on social network sites. Behaviour & Information Technology344425–435. 10.1080/0144929X.2014.983979
  • Blei . (2003) Blei2003Blei, DM., Ng, AY.  Jordan, MI.  2003. Latent dirichlet allocation Latent dirichlet allocation. Journal of Machine Learning Research3993–1022.
  • Chang  Lin (2011) Chang2011Chang, CC.  Lin, CJ.  2011. LIBSVM: A Library for Support Vector Machines LIBSVM: A Library for Support Vector Machines. ACM Transactions on Intelligent Systems and Technology (TIST)2327:1–27:27. 10.1145/1961189.1961199
  • Chawla . (2002) Chawla2002Chawla, NV., Bowyer, KW., Hall, LO.  Kegelmeyer, PW.  2002. SMOTE: Synthetic Minority Over-sampling TEchnique SMOTE: Synthetic Minority Over-sampling TEchnique.

    Journal of Artificial Intelligence Research (JAIR)16321–357.

  • Cohen (1960) Cohen1960Cohen, J.  1960. A coefficient of agreement for nominal scales A coefficient of agreement for nominal scales. Educational and Psychological Measurement20137–46.
  • Cortes  Vapnik (1995) Cortes1995Cortes, C.  Vapnik, V.  1995. Support-Vector Networks Support-Vector Networks. Machine Learning203273–297. 10.1023/A:1022627411411
  • Cowie (2013) Cowie2013Cowie, H.  2013. Cyberbullying and its impact on young people’s emotional health and well-being Cyberbullying and its impact on young people’s emotional health and well-being. The Psychiatrist375167–170. 10.1192/pb.bp.112.040840
  • Dadvar (2014) Dadvar2014Dadvar, M.  2014.  Experts and machines united against cyberbullying Experts and machines united against cyberbullying PhD thesis.  University of Twente.
  • Deerwester . (1990) Deerwester1990Deerwester, S., Dumais, ST., Furnas, GW., Landauer, TK.  Harshman, R.  1990. Indexing by latent semantic analysis Indexing by latent semantic analysis. Journal of the American Society for Information Science41391–407.
  • De Smedt  Daelemans (2012) DeSmedt2012De Smedt, T.  Daelemans, W.  2012. “Vreselijk mooi!” (“Terribly Beautiful!”): A Subjectivity Lexicon for Dutch Adjectives “Vreselijk mooi!” (“Terribly Beautiful!”): A Subjectivity Lexicon for Dutch Adjectives. Proceedings of the Eight International Conference on Language Resources and Evaluation Proceedings of the Eight International Conference on Language Resources and Evaluation ( 3568-3572). Istanbul, Turkey.
  • Desmet (2014) Desmet2014Desmet, B.  2014.  Finding the online cry for help: automatic text classification for suicide prevention Finding the online cry for help: automatic text classification for suicide prevention PhD thesis.  Ghent University.
  • Dinakar . (2012) Dinakar2012Dinakar, K., Jones, B., Havasi, C., Lieberman, H.  Picard, R.  2012. Common Sense Reasoning for Detection, Prevention, and Mitigation of Cyberbullying Common Sense Reasoning for Detection, Prevention, and Mitigation of Cyberbullying. ACM Transactions on Interactive Intelligent Systems2318:1–18:30.
  • Dinakar . (2011) Dinakar2011Dinakar, K., Reichart, R.  Lieberman, H.  2011. Modeling the Detection of Textual Cyberbullying Modeling the Detection of Textual Cyberbullying. The Social Mobile Web The Social Mobile Web ( WS-11-02,  11–17). AAAI.
  • Dooley  Cross (2009) Dooley2009Dooley, JJ.  Cross, D.  2009. Cyberbullying versus face-to-face bullying: A review of the similarities and differences Cyberbullying versus face-to-face bullying: A review of the similarities and differences. Journal of Psychology217182–188.
  • Fan . (2008) Fan2008Fan, RE., Chang, KW., Hsieh, CJ., Wang, XR.  Lin, CJ.  2008. LIBLINEAR: A Library for Large Linear Classification LIBLINEAR: A Library for Large Linear Classification. Journal of Machine Learning Research91871–1874.
  • Fawcett (2006) fawcett2006introductionFawcett, T.  2006. An introduction to ROC analysis An introduction to roc analysis. Pattern recognition letters278861–874.
  • Fekkes . (2006) Fekkes2006Fekkes, M., Pijpers, FI., Fredriks, AM., Vogels, T.  Verloove-Vanhorick, SP.  2006. Do Bullied Children Get Ill, or Do Ill Children Get Bullied? A Prospective Cohort Study on the Relationship Between Bullying and Health-Related Symptoms Do Bullied Children Get Ill, or Do Ill Children Get Bullied? A Prospective Cohort Study on the Relationship Between Bullying and Health-Related Symptoms. Pediatrics11751568–1574. 10.1542/peds.2005-0187
  • Fleiss (1971) Fleiss1971Fleiss, JL.  1971. Measuring nominal scale agreement among many raters Measuring nominal scale agreement among many raters. Psychological Bulletin765378–382.
  • Gross . (2002) Gross2002Gross, EF., Juvonen, J.  Gable, SL.  2002. Internet Use and Well-Being in Adolescence Internet Use and Well-Being in Adolescence. Journal of Social Issues58175–90.
  • He  Garcia (2009) he2009learningHe, H.  Garcia, EA.  2009. Learning from imbalanced data Learning from imbalanced data. IEEE Transactions on knowledge and data engineering2191263–1284.
  • Hinduja  Patchin (2006) Hinduja2006Hinduja, S.  Patchin, JW.  2006. Bullies Move Beyond the Schoolyard: A Preliminary Look at Cyberbullying Bullies Move Beyond the Schoolyard: A Preliminary Look at Cyberbullying. Youth Violence And Juvenile Justice42148–169.
  • Hinduja  Patchin (2012) Hinduja2012Hinduja, S.  Patchin, JW.  2012. Cyberbullying: Neither an epidemic nor a rarity Cyberbullying: Neither an epidemic nor a rarity. European Journal of Developmental Psychology95539–543.
  • Hoste (2005) Hoste2005Hoste, V.  2005.  Optimization Issues in Machine Learning of Coreference Resolution Optimization Issues in Machine Learning of Coreference Resolution PhD thesis.  Antwerp University.
  • Hu  Liu (2004) Liu2004Hu, M.  Liu, B.  2004. Mining and summarizing customer reviews Mining and summarizing customer reviews. Proceedings of the 10th ACM SIGKDD international conference on Knowledge discovery and data mining Proceedings of the 10th ACM SIGKDD international conference on Knowledge discovery and data mining ( 168–177). ACM.
  • Jijkoun  Hofmann (2009) Jijkoun2009Jijkoun, V.  Hofmann, K.  2009. Generating a Non-English Subjectivity Lexicon: Relations That Matter Generating a Non-English Subjectivity Lexicon: Relations That Matter. Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics ( 398-405). Stroudsburg, PA, USA.
  • Juvonen  Gross (2008) Juvonen2008Juvonen, J.  Gross, EF.  2008. Extending the school grounds? – Bullying experiences in cyberspace Extending the school grounds? – Bullying experiences in cyberspace. Journal of School Health789496–505.
  • Livingstone . (2010) Livingstone2010Livingstone, S., Haddon, L., Görzig, A.  Ólafsson, K.  2010. Risks and safety on the internet: The perspective of European children. Initial Findings. Risks and safety on the internet: The perspective of European children. Initial Findings. London: EU Kids Online.
  • Livingstone . (2013) Livingstone2013Livingstone, S., Kirwil, L., Ponte, C.  Staksrud, E.  2013. In their own words: what bothers children online? In their own words: what bothers children online? London: EU Kids Online.
  • McHugh (2012) McHugh2012McHugh, ML.  2012. Interrater reliability: the kappa statistic Interrater reliability: the kappa statistic. Biochemia Medica223276–282.
  • Mckenna  Bargh (1999) Mckenna1999Mckenna, KY.  Bargh, JA.  1999. Plan 9 From Cyberspace: The Implications of the Internet for Personality and Social Psychology. Plan 9 From Cyberspace: The Implications of the Internet for Personality and Social Psychology. Personality & Social Psychology Review4157–75.
  • Mohammad . (2009) Mohammad2009Mohammad, S., Dunne, C.  Dorr, B.  2009. Generating High-coverage Semantic Orientation Lexicons from Overtly Marked Words and a Thesaurus Generating High-coverage Semantic Orientation Lexicons from Overtly Marked Words and a Thesaurus. Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2 ( 599–608). Stroudsburg, PA, USAAssociation for Computational Linguistics.
  • Nadali . (2013) Nadali2013Nadali, S., Azmi Murad, MA., Sharef, NM., Mustapha, A.  Shojaee, S.  2013. A review of cyberbullying detection: An overview A review of cyberbullying detection: An overview. 13th International Conference on Intellient Systems Design and Applications 13th International Conference on Intellient Systems Design and Applications ( 325–330). Salangor, Malaysia,. 10.1109/ISDA.2013.6920758
  • Nahar . (2014) Nahar2014Nahar, V., Al-Maskari, S., Li, X.  Pang, C.  2014. Semi-supervised Learning for Cyberbullying Detection in Social Networks Semi-supervised Learning for Cyberbullying Detection in Social Networks. ADC.Databases Theory and Applications ADC.Databases Theory and Applications ( 160–171).
  • Nansel . (2001) Nansel2001Nansel, TR., Overpeck, M., Pilla, RS., Ruan, JW., Morton, BS.  Scheidt, P.  2001. Bullying behaviors among US youth: prevalence and association with psychosocial adjustment Bullying behaviors among US youth: prevalence and association with psychosocial adjustment. JAMA285162094–2100.
  • Nielsen (2011) Afinn2011Nielsen, FÅ.  2011.

    A New ANEW: Evaluation of a Word List for Sentiment Analysis in Microblogs A New ANEW: Evaluation of a Word List for Sentiment Analysis in Microblogs.

    M. Rowe, M. Stankovic, AS. Dadzie  M. Hardey (), Proceedings of the ESWC2011 Workshop on ‘Making Sense of Microposts’: Big things come in small packages Proceedings of the ESWC2011 Workshop on ‘Making Sense of Microposts’: Big things come in small packages ( 718,  93–98). CEUR-WS.org.
  • Olweus (1993) Olweus1993Olweus, D.  1993. Bullying at School: What We Know and What We Can Do Bullying at School: What We Know and What We Can Do (2nd ). Wiley.
  • O’Moore  Kirkham (2001) Moore2001O’Moore, M.  Kirkham, C.  2001. Self-esteem and its relationship to bullying behaviour Self-esteem and its relationship to bullying behaviour. Aggressive Behavior274269–283.
  • Online (2014) EUKidsOnline2014Online, EK.  2014. EU Kids Online: findings, methods, recommendations. EU Kids Online, LSE, London, UK. http://eprints.lse.ac.uk/60512/. EU Kids Online: findings, methods, recommendations. EU Kids Online, LSE, London, UK. http://eprints.lse.ac.uk/60512/. London: EU Kids Online. http://eprints.lse.ac.uk/60512/
  • O’Sullivan  Flanagin (2003) Osullivan2003O’Sullivan, PB.  Flanagin, AJ.  2003. Reconceptualizing ‘flaming’ and other problematic messages Reconceptualizing ‘flaming’ and other problematic messages. New Media & Society5169-94.
  • Pedregosa . (2011) Pedregosa2011Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O.Duchesnay, E.  2011. Scikit-learn: Machine Learning in Python Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research122825–2830.
  • Pennebaker . (2001) Pennebaker2001Pennebaker, JW., Francis, ME.  Booth, RJ.  2001. Linguistic Inquiry and Word Count: LIWC 2001 Linguistic Inquiry and Word Count: LIWC 2001. Mahwah, NJLawrence Erlbaum Associates.
  • Price  Dalgleish (2010) Price2010Price, M.  Dalgleish, J.  2010. Cyberbullying: Experiences, Impacts and Coping Strategies as Described by Australian Young People Cyberbullying: Experiences, Impacts and Coping Strategies as Described by Australian Young People. Youth Studies Australia29251–59.
  • Rehurek  Sojka (2010) Rehurek2010Rehurek, R.  Sojka, P.  2010. Software framework for topic modelling with large corpora Software framework for topic modelling with large corpora. The LREC 2010 Workshop on new Challenges for NLP Frameworks The LREC 2010 Workshop on new Challenges for NLP Frameworks ( 45–50).
  • Reynolds . (2011) Reynolds2011Reynolds, K., Kontostathis, A.  Edwards, L.  2011. Using Machine Learning to Detect Cyberbullying Using Machine Learning to Detect Cyberbullying. Proceedings of the 2011 10th International Conference on Machine Learning and Applications and Workshops Proceedings of the 2011 10th International Conference on Machine Learning and Applications and Workshops ( 241–244). Washington, DC, USAIEEE Computer Society.
  • Royen . (2016) VanRoyen2016Royen, KV., Poels, K.  Vandebosch, H.  2016. Harmonizing freedom and protection: Adolescents’ voices on automatic monitoring of social networking sites Harmonizing freedom and protection: Adolescents’ voices on automatic monitoring of social networking sites. Children and Youth Services Review6435 - 41. http://dx.doi.org/10.1016/j.childyouth.2016.02.024
  • Salmivalli (2010) Salmivalli2010Salmivalli, C.  2010. Bullying and the peer group: A review Bullying and the peer group: A review. Aggression and Violent Behavior152112–120.
  • Salmivalli . (1999) Salmivalli1999Salmivalli, C., Kaukiainen, A., Kaistaniemi, L.  Lagerspetz, KMJ.  1999. Self-Evaluated Self-Esteem, Peer-Evaluated Self-Esteem, and Defensive Egotism as Predictors of Adolescents’ Participation in Bullying Situations Self-Evaluated Self-Esteem, Peer-Evaluated Self-Esteem, and Defensive Egotism as Predictors of Adolescents’ Participation in Bullying Situations. Personality and Social Psychology Bulletin25101268–1278. 10.1177/0146167299258008
  • Salmivalli . (1996) Salmivalli1996Salmivalli, C., Lagerspetz, K., Björkqvist, K., Österman, K.  Kaukiainen, A.  1996. Bullying as a group process: Participant roles and their relations to social status within the group Bullying as a group process: Participant roles and their relations to social status within the group. Aggressive Behavior2211–15.
  • Salmivalli . (2011) Salmivalli2011Salmivalli, C., Voeten, M.  Poskiparta, E.  2011. Bystanders Matter: Associations Between Reinforcing, Defending, and the Frequency of Bullying Behavior in Classrooms Bystanders Matter: Associations Between Reinforcing, Defending, and the Frequency of Bullying Behavior in Classrooms. Journal of Clinical Child & Adolescent Psychology405668-676. 10.1080/15374416.2011.597090
  • Slonje  Smith (2008) Slonje2008Slonje, R.  Smith, PK.  2008. Cyberbullying: Another main type of bullying? Cyberbullying: Another main type of bullying? Scandinavian Journal of Psychology492147–154.
  • Slonje . (2013) Slonje2013Slonje, R., Smith, PK.  Frisén, A.  2013. The Nature of Cyberbullying, and Strategies for Prevention The Nature of Cyberbullying, and Strategies for Prevention. Compututers in Human Behavior29126–32.
  • Smith . (2008) Smith2008Smith, PK., Mahdavi, J., Carvalho, M., Fisher, S., Russell, S.  Tippett, N.  2008. Cyberbullying: its nature and impact in secondary school pupils Cyberbullying: its nature and impact in secondary school pupils. Journal of Child Psychology and Psychiatry494376–385.
  • Stenetorp . (2012) Stenetorp2012Stenetorp, P., Pyysalo, S., Topić, G., Ohta, T., Ananiadou, S.  Tsujii, J.  2012. brat: a Web-based Tool for NLP-Assisted Text Annotation brat: a Web-based Tool for NLP-Assisted Text Annotation. Proceedings of the Demonstrations Session at EACL 2012 Proceedings of the Demonstrations Session at EACL 2012 ( 102–107). Avignon, France.
  • Stone . (1966) Stone1966Stone, PJ., Dunphy, DCD., Smith, MS.  Ogilvie, DM.  1966. The General Inquirer: A Computer Approach to Content Analysis The General Inquirer: A Computer Approach to Content Analysis. The MIT Press.
  • Sui (2015) Sui2015Sui, J.  2015.  Understanding and Fighting Bullying with Machine Learning Understanding and Fighting Bullying with Machine Learning PhD thesis.  Department of Computer Sciences, University of Wisconsin-Madison.
  • Tokunaga (2010) Tokunaga2010Tokunaga, RS.  2010. Following You Home from School: A Critical Review and Synthesis of Research on Cyberbullying Victimization Following You Home from School: A Critical Review and Synthesis of Research on Cyberbullying Victimization. Computers in Human Behavior263277–287.
  • Van Cleemput . (2013) VanCleemput2013Van Cleemput, K., Bastiaensens, S., Vandebosch, H., Poels, K., Deboutte, G., DeSmet, A.  De Bourdeaudhuij, I.  2013. Zes jaar onderzoek naar cyberpesten in Vlaanderen, België en daarbuiten: een overzicht van de bevindingen. (Six years of research on cyberbullying in Flanders, Belgium and beyond: an overview of the findings.) (White Paper) Zes jaar onderzoek naar cyberpesten in Vlaanderen, België en daarbuiten: een overzicht van de bevindingen. (Six years of research on cyberbullying in Flanders, Belgium and beyond: an overview of the findings.) (White Paper) . University of Antwerp & Ghent University.
  • Vandebosch  Van Cleemput (2008) Vandebosch2008Vandebosch, H.  Van Cleemput, K.  2008. Defining cyberbullying: a qualitative research into the perceptions of youngsters Defining cyberbullying: a qualitative research into the perceptions of youngsters. Cyberpsychology and behavior: the impact of the Internet, multimedia and virtual reality on behavior and society114499–503.
  • Vandebosch  Van Cleemput (2009) Vandebosch2009Vandebosch, H.  Van Cleemput, K.  2009. Cyberbullying among youngsters: profiles of bullies and victims. Cyberbullying among youngsters: profiles of bullies and victims. New Media & Society1181349–1371.
  • Vandebosch . (2006) Vandebosch2006Vandebosch, H., Van Cleemput, K., Mortelmans, D.  Walrave, M.  2006. Cyberpesten bij jongeren in Vlaanderen: Een studie in opdracht van het viWTA (Cyberbullying among youngsters in Flanders: a study commissoned by the viWTA). Brussels: viWTA Cyberpesten bij jongeren in Vlaanderen: Een studie in opdracht van het viWTA (Cyberbullying among youngsters in Flanders: a study commissoned by the viWTA). Brussels: viWTA . http://www.samenlevingentechnologie.be/ists/nl/pdf/rapporten
    /rapportcyberpesten.pdf
  • van de Kauter . (2013) vandeKauter2013van de Kauter, M., Coorman, G., Lefever, E., Desmet, B., Macken, L.  Hoste, V.  2013. LeTs Preprocess: The multilingual LT3 linguistic preprocessing toolkit LeTs Preprocess: The multilingual LT3 linguistic preprocessing toolkit. Computational Linguistics in the Netherlands Journal3103–120.
  • Van Hee, Lefever . (20151) VanHee2015bVan Hee, C., Lefever, E., Verhoeven, B., Mennes, J., Desmet, B., De Pauw, G.Hoste, V.  20151. Automatic detection and prevention of cyberbullying Automatic detection and prevention of cyberbullying. P. Lorenz  C. Bourret (), International Conference on Human and Social Analytics, Proceedings International conference on human and social analytics, proceedings ( 13–18). IARIA.
  • Van Hee, Lefever . (20152) VanHee2015aVan Hee, C., Lefever, E., Verhoeven, B., Mennes, J., Desmet, B., De Pauw, G.Hoste, V.  20152. Detection and fine-grained classification of cyberbullying events Detection and fine-grained classification of cyberbullying events. G. Angelova, K. Bontcheva  R. Mitkov (), Proceedings of Recent Advances in Natural Language Processing, Proceedings Proceedings of recent advances in natural language processing, proceedings ( 672–680).
  • Van Hee, Verhoeven . (2015) VanHee2015cVan Hee, C., Verhoeven, B., Lefever, E., De Pauw, G., Daelemans, W.  Hoste, V.  2015. Guidelines for the Fine-Grained Analysis of Cyberbullying, version 1.0 Guidelines for the Fine-Grained Analysis of Cyberbullying, version 1.0   LT3 15-01. LT3, Language and Translation Technology Team–Ghent University.
  • Van Royen . (2014) VanRoyen2014Van Royen, K., Poels, K., Daelemans, W.  Vandebosch, H.  2014. Automatic monitoring of cyberbullying on social networking sites: From technological feasibility to desirability Automatic monitoring of cyberbullying on social networking sites: From technological feasibility to desirability. Telematics and Informatics. 10.1016/j.tele.2014.04.002
  • Willard (2007) Willard2007Willard, NE.  2007. Cyberbullying and Cyberthreats: Responding to the Challenge of Online Social Aggression, Threats, and Distress Cyberbullying and Cyberthreats: Responding to the Challenge of Online Social Aggression, Threats, and Distress (2nd ). Research Publishers LLC.
  • Wilson . (2005) Wilson2005Wilson, T., Wiebe, J.  Hoffmann, P.  2005. Recognizing Contextual Polarity in Phrase-level Sentiment Analysis Recognizing Contextual Polarity in Phrase-level Sentiment Analysis. Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing ( 347–354). Association for Computational Linguistics. 10.3115/1220575.1220619
  • Wolak . (2007) Wolak2007Wolak, J., Mitchell, KJ.  Finkelhor, D.  2007. Does Online Harassment Constitute Bullying? An Exploration of Online Harassment by Known Peers and Online-Only Contacts Does Online Harassment Constitute Bullying? An Exploration of Online Harassment by Known Peers and Online-Only Contacts. Journal of Adolescent Health416, SupplementS51–S58.
  • Xu . (2012) Xu2012Xu, JM., Jun, KS., Zhu, X.  Bellmore, A.  2012. Learning from Bullying Traces in Social Media Learning from Bullying Traces in Social Media. Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( 656–666). Stroudsburg, PA, USAAssociation for Computational Linguistics.
  • Yin . (2009) Yin2009Yin, D., Davison, BD., Xue, Z., Hong, L., Kontostathis, A.  Edwards, L.  2009. Detection of Harassment on Web 2.0 Detection of Harassment on Web 2.0. Proceedings of the Content Analysis in the Web 2.0 (CAW2.0). Proceedings of the Content Analysis in the Web 2.0 (CAW2.0). Madrid, Spain.
  • Zhao . (2016) Zhao2016Zhao, R., Zhou, A.  Mao, K.  2016. Automatic Detection of Cyberbullying on Social Networks Based on Bullying Features Automatic Detection of Cyberbullying on Social Networks Based on Bullying Features. Proceedings of the 17th International Conference on Distributed Computing and Networking Proceedings of the 17th International Conference on Distributed Computing and Networking ( 43:1–43:6). New York, NY, USAACM. 10.1145/2833312.2849567
  • Zijlstra . (2004) Zijlstra2004Zijlstra, H., Van Meerveld, T., Van Middendorp, H., Pennebaker, JW.  Geenen, R.  2004. De Nederlandse versie van de ‘linguistic inquiry and word count’ (LIWC) De Nederlandse versie van de ‘linguistic inquiry and word count’ (LIWC). Gedrag Gezond32271–281.