Solutions to MLWARE1 organised by Analytics Vidhya (https://datahack.analyticsvidhya.com/contest/mlware-1/)
This paper makes a simple increment to state-of-the-art in sarcasm detection research. Existing approaches are unable to capture subtle forms of context incongruity which lies at the heart of sarcasm. We explore if prior work can be enhanced using semantic similarity/discordance between word embeddings. We augment word embedding-based features to four feature sets reported in the past. We also experiment with four types of word embeddings. We observe an improvement in sarcasm detection, irrespective of the word embedding used or the original feature set to which our features are augmented. For example, this augmentation results in an improvement in F-score of around 4% for three out of these four feature sets, and a minor degradation in case of the fourth, when Word2Vec embeddings are used. Finally, a comparison of the four embeddings shows that Word2Vec and dependency weight-based features outperform LSA and GloVe, in terms of their benefit to sarcasm detection.READ FULL TEXT VIEW PDF
Do word embeddings converge to learn similar things over different
While word embeddings have been shown to implicitly encode various forms...
We analyze the process of creating word embedding feature representation...
This paper presents an approach for investigating the nature of semantic...
A lot of prior work on event extraction has exploited a variety of featu...
We consider probabilistic topic models and more recent word embedding
We introduce language-driven image generation, the task of generating an...
Solutions to MLWARE1 organised by Analytics Vidhya (https://datahack.analyticsvidhya.com/contest/mlware-1/)
Sarcasm is a form of verbal irony that is intended to express contempt or ridicule. Linguistic studies show that the notion of context incongruity is at the heart of sarcasm [Ivanko and Pexman2003]. A popular trend in automatic sarcasm detection is semi-supervised extraction of patterns that capture the underlying context incongruity [Davidov et al.2010, Joshi et al.2015, Riloff et al.2013]. However, techniques to extract these patterns rely on sentiment-bearing words and may not capture nuanced forms of sarcasm. Consider the sentence ‘With a sense of humor like that, you could make a living as a garbage man anywhere in the country.111All examples in this paper are actual instances from our dataset.
’ The speaker makes a subtle, contemptuous remark about the sense of humor of the listener. However, absence of sentiment words makes the sarcasm in this sentence difficult to capture as features for a classifier.
In this paper, we explore use of word embeddings to capture context incongruity in the absence of sentiment words. The intuition is that word vector-based similarity/discordance is indicative of semantic similarity which in turn is a handle for context incongruity
word vector-based similarity/discordance is indicative of semantic similarity which in turn is a handle for context incongruity. In the case of the ‘sense of humor’ example above, the words ‘sense of humor’ and ‘garbage man’ are semantically dissimilar and their presence together in the sentence provides a clue to sarcasm. Hence, our set of features based on word embeddings aim to capture such semantic similarity/discordance. Since such semantic similarity is but one of the components of context incongruity and since existing feature sets rely on sentiment-based features to capture context incongruity, it is imperative that the two be combined for sarcasm detection. Thus, our paper deals with the question:
Can word embedding-based features when augmented to features reported in prior work improve the performance of sarcasm detection?
To the best of our knowledge, this is the first attempt that uses word embedding-based features to detect sarcasm. In this respect, the paper makes a simple increment to state-of-the-art but opens up a new direction in sarcasm detection research. We establish our hypothesis in case of four past works and four types of word embeddings, to show that the benefit of using word embedding-based features holds across multiple feature sets and word embeddings.
In our literature survey of sarcasm detection [Joshi et al.2016], we observe that a popular trend is semi-supervised extraction of patterns with implicit sentiment. One such work is by riloff who give a bootstrapping algorithm that discovers a set of positive verbs and negative/undesirable situations. However, this simplification (of representing sarcasm merely as positive verbs followed by negative situation) may not capture difficult forms of context incongruity. Consider the sarcastic sentence ‘A woman needs a man like a fish needs bicycle’222This quote is attributed to Irina Dunn, an Australian writer (https://en.wikipedia.org/wiki/Irina_Dunn. The sarcasm in this sentence is understood from the fact that a fish does not need bicycle - and hence, the sentence ridicules the target ‘a man’. However, this sentence does not contain any sentiment-bearing word. Existing sarcasm detection systems relying on sentiment incongruity (as in the case of our past work reported as 22) may not work well in such cases of sarcasm.
To address this, we use semantic similarity as a handle to context incongruity. To do so, we use word vector similarity scores. Consider similarity scores (as given by Word2Vec) between two pairs of words in the sentence above:
similarity(‘man’,’woman’) = 0.766
similarity(‘fish’,’bicycle’) = 0.131
Words in one part of this sentence (‘man’ and ‘woman’) are lot more similar than words in another part of the sentence (‘fish’ and ‘bicycle’). This semantic discordance can be a clue to presence of context incongruity. Hence, we propose features based on similarity scores between word embeddings of words in a sentence. In general, we wish to capture the most similar and most dissimilar word pairs in the sentence, and use their scores as features for sarcasm detection.
We augment our word embedding-based features to the following four feature sets that have been reported:
liebrecht: They consider unigrams, bigrams and trigrams as features.
gonzalez: They propose two sets of features: unigrams and dictionary-based. The latter are words from a lexical resource called LIWC. We use words from LIWC that have been annotated as emotion and psychological process words, as described in the original paper.
buschmeier: In addition to unigrams, they propose features such as: (a) Hyperbole (captured by three positive or negative words in a row), (b) Quotation marks and ellipsis, (c) Positive/Negative Sentiment words followed by an exclamation mark or question mark, (d) Positive/Negative Sentiment Scores followed by ellipsis (represented by a ‘…’), (e) Punctuation, (f) Interjections, and (g) Laughter expressions.
22: In addition to unigrams, they use features based on implicit and explicit incongruity. Implicit incongruity features are patterns with implicit sentiment as extracted in a pre-processing step. Explicit incongruity features consist of number of sentiment flips, length of positive and negative sub-sequences and lexical polarity.
In this section, we now describe our word embedding-based features. We reiterate that these features will be augmented to features from prior works (described in Section 3).
As stated in Section 2
, our word embedding-based features are based on similarity scores between word embeddings. The similarity score is the cosine similarity between vectors of two words. To illustrate our features, we use our example ‘A woman needs a man like a fish needs a bicycle’. The scores for all pairs of words in this sentence are given in Table 1.
Using these similarity scores, we compute two sets of features:
Unweighted similarity features (S): We first compute similarity scores for all pairs of words (except stop words). We then return four feature values per sentence.333These feature values consider all words in the sentence, i.e., the ‘maximum’ is computed over all words:
Maximum score of most similar word pair
Minimum score of most similar word pair
Maximum score of most dissimilar word pair
Minimum score of most dissimilar word pair
For example, in case of the first feature, we consider the most similar word to every word in the sentence, and the corresponding similarity scores. These most similar word scores for each word are indicated in bold in Table 1. Thus, the first feature in case of our example would have the value 0.766 derived from the man-woman pair and the second feature would take the value 0.078 due to the needs-man pair. The other features are computed in a similar manner.
Distance-weighted similarity features (WS): Like in the previous case, we first compute similarity scores for all pairs of words (excluding stop-words). For all similarity scores, we divide them by square of distance between the two words. Thus, the similarity between terms that are close in the sentence is weighted higher than terms which are distant from one another. Thus, for all possible word pairs, we compute four features:
Maximum distance-weighted score of most similar word pair
Minimum distance-weighted score of most similar word pair
Maximum distance-weighted score of most dissimilar word pair
Minimum distance-weighted score of most dissimilar word pair
These are computed similar to unweighted similarity features.
We create a dataset consisting of quotes on GoodReads 444www.goodreads.com. GoodReads describes itself as ‘the world’s largest site for readers and book recommendations.
’ The website also allows users to post quotes from books. These quotes are snippets from books labeled by the user with tags of their choice. We download quotes with the tag ‘sarcastic’ as sarcastic quotes, and the ones with ‘philosophy’ as non-sarcastic quotes. Our labels are based on these tags given by users. We ensure that no quote has both these tags. This results in a dataset of 3629 quotes out of which 759 are labeled as sarcastic. This skew is similar to skews observed in datasets on which sarcasm detection experiments have been reported in the past[Riloff et al.2013].
We report five-fold cross-validation results on the above dataset. We use by svmperf with as 20,
as 3, and loss function as F-score optimization. This allows SVM to be learned while optimizing the F-score.
As described above, we compare features given in prior work alongside the augmented versions. This means that for each of the four papers, we experiment with four configurations:
Features given in paper X
Features given in paper X + unweighted similarity features (S)
Features given in paper X + weighted similarity features (WS)
Features given in paper X + S+WS (i.e., weighted and unweighted similarity features)
We experiment with four types of word embeddings:
LSA: This approach was reported in Landauer97asolution. We use pre-trained word embeddings based on LSA555http://www.lingexp.uni-tuebingen.de/z2/LSAspaces/. The vocabulary size is 100,000.
GloVe: We use pre-trained vectors avaiable from the GloVe project666http://nlp.stanford.edu/projects/glove/. The vocabulary size in this case is 2,195,904.
Dependency Weights: We use pre-trained vectors777 https://levyomer.wordpress.com/2014/04/25/dependency-based-word-embeddings/ weighted using dependency distance, as given in depwords. The vocabulary size is 174,015.
Word2Vec: use pre-trained Google word vectors. These were trained using Word2Vec tool 888https://code.google.com/archive/p/Word2Vec/ on the Google News corpus. The vocabulary size for Word2Vec is 3,000,000. To interact with these pre-trained vectors, as well as compute various features, we use gensim library [Řehůřek and Sojka2010].
To interact with the first three pre-trained vectors, we use scikit library [Pedregosa et al.2011].
Table 2 shows performance of sarcasm detection when our word embedding-based features are used on their own i.e, not as augmented features. The embedding in this case is Word2Vec. The four rows show baseline sets of features: unigrams, unweighted similarity using word embeddings (S), weighted similarity using word embeddings (WS) and both (i.e., unweighted plus weighted similarities using word embeddings). Using only unigrams as features gives a F-score of 72.53%, while only unweighted and weighted features gives F-score of 69.49% and 58.26% respectively. This validates our intuition that word embedding-based features alone are not sufficient, and should be augmented with other features.
Following this, we show performance using features presented in four prior works: buschmeier, liebrecht, 22 and gonzalez, and compare them with augmented versions in Table 3.
Table 3 shows results for four kinds of word embeddings. All entries in the tables are higher than the simple unigrams baseline, i.e., F-score for each of the four is higher than unigrams - highlighting that these are better features for sarcasm detection than simple unigrams. Values in bold indicate the best F-score for a given prior work-embedding type combination. In case of liebrecht for Word2Vec, the overall improvement in F-score is 4%. Precision increases by 8% while recall remains nearly unchanged. For features given in gonzalez, there is a negligible degradation of 0.91% when word embedding-based features based on Word2Vec are used. For buschmeier for Word2Vec, we observe an improvement in F-score from 76.61% to 78.09%. Precision remains nearly unchanged while recall increases. In case of 22 and Word2Vec, we observe a slight improvement of 0.20% when unweighted (S) features are used. This shows that word embedding-based features are useful, across four past works for Word2Vec.
Table 3 also shows that the improvement holds across the four word embedding types as well. The maximum improvement is observed in case of liebrecht. It is around 4% in case of LSA, 5% in case of GloVe, 6% in case of Dependency weight-based and 4% in case of Word2Vec. These improvements are not directly comparable because the four embeddings have different vocabularies (since they are trained on different datasets) and vocabulary sizes, their results cannot be directly compared.
Therefore, we take an intersection of the vocabulary (i.e., the subset of words present in all four embeddings) and repeat all our experiments using these intersection files. The vocabulary size of these intersection files (for all four embeddings) is 60,252. Table 4 shows the average increase in F-score when a given word embedding and a word embedding-based feature is used, with the intersection file as described above. These gain values are lower than in the previous case. This is because these are the values in case of the intersection versions - which are subsets of the complete embeddings. Each gain value is averaged over the four prior works. Thus, when unweighted similarity (+S) based features computed using LSA are augmented to features from prior work, an average increment of 0.835% is obtained over the four prior works. The values allow us to compare the benefit of using these four kinds of embeddings. In case of unweighted similarity-based features, dependency-based weights give the maximum gain (0.978%). In case of weighted similarity-based features and ‘+S+WS’, Word2Vec gives the maximum gain (1.411%). Table 5 averages these values over the three types of word embedding-based features. Using Dependency-based and Word2Vec embeddings results in a higher improvement in F-score (1.048% and 1.143% respectively) as compared to others.
|Word Embedding||Average F-score Gain|
Some categories of errors made by our system are:
Embedding issues due to incorrect senses: Because words may have multiple senses, some embeddings lead to error, as in ‘Great. Relationship advice from one of America’s most wanted.’.
Contextual sarcasm: Consider the sarcastic quote ‘Oh, and I suppose the apple ate the cheese’. The similarity score between ‘apple’ and ‘cheese’ is 0.4119. This comes up as the maximum similar pair. The most dissimilar pair is ‘suppose’ and ‘apple’ with similarity score of 0.1414. The sarcasm in this sentence can be understood only in context of the complete conversation that it is a part of.
Metaphors in non-sarcastic text: Figurative language may compare concepts that are not directly related but still have low similarity. Consider the non-sarcastic quote ‘Oh my love, I like to vanish in you like a ripple vanishes in an ocean - slowly, silently and endlessly’. Our system incorrectly predicts this as sarcastic.
Early sarcasm detection research focused on speech [Tepperman et al.2006] and lexical features [Kreuz and Caucci2007]. Several other features have been proposed [Kreuz and Caucci2007, Joshi et al.2015, Khattri et al.2015, Liebrecht et al.2013, González-Ibánez et al.2011a, Rakov and Rosenberg2013, Wallace2015, Wallace et al.2014, Veale and Hao2010, González-Ibánez et al.2011b, Reyes et al.2012]. Of particular relevance to our work are papers that aim to first extract patterns relevant to sarcasm detection. 4 use a semi-supervised approach that extracts sentiment-bearing patterns for sarcasm detection. 22 extract phrases corresponding to implicit incongruity i.e. the situation where sentiment is expressed without use of sentiment words. riloff describe a bootstrapping algorithm that iteratively discovers a set of positive verbs and negative situation phrases, which are later used in a sarcasm detection algorithm. 3 also perform semi-supervised extraction of patterns for sarcasm detection. The only prior work which uses word embeddings for a related task of sarcasm detection is by ghoshsarcastic. They model sarcasm detection as a word sense disambiguation task, and use embeddings to identify whether a word is used in the sarcastic or non-sarcastic sense. Two sense vectors for every word are created: one for literal sense and one for sarcastic sense. The final sense is determined based on the similarity of these sense vectors with the sentence vector.
This paper shows the benefit of features based on word embedding for sarcasm detection. We experiment with four past works in sarcasm detection, where we augment our word embedding-based features to their sets of features. Our features use the similarity score values returned by word embeddings, and are of two categories: similarity-based (where we consider maximum/minimum similarity score of most similar/dissimilar word pair respectively), and weighted similarity-based (where we weight the maximum/minimum similarity scores of most similar/dissimilar word pairs with the linear distance between the two words in the sentence). We experiment with four kinds of word embeddings: LSA, GloVe, Dependency-based and Word2Vec. In case of Word2Vec, for three of these past feature sets to which our features were augmented, we observe an improvement in F-score of at most 5%. Similar improvements are observed in case of other word embeddings. A comparison of the four embeddings shows that Word2Vec and dependency weight-based features outperform LSA and GloVe.
This work opens up avenues for use of word embeddings for sarcasm classification. Our word embedding-based features may work better if the similarity scores are computed for a subset of words in the sentence, or using weighting based on syntactic distance instead of linear distance as in the case of our weighted similarity-based features.
Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, volume 2, pages 757–762.
Scikit-learn: Machine learning in python.The Journal of Machine Learning Research, 12:2825–2830.
Data & Knowledge Engineering, 74:1–12.
Humans require context to infer ironic intent (so computers probably do, too).In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 512–516.