Ambiguity is common in natural language. Word Sense Disambiguation (WSD) deals with lexical ambiguity, i.e. polysemes in sentences. An effective WSD tool can benefit various downstream tasks, such as Information Retrieval Zhong and Ng (2012) and Machine Translation Neale et al. (2016).
Recently, the pre-trained language models, such as ELMo Peters et al. (2018), GPT Radford et al. (2018), and BERT Devlin et al. (2019), have been proven to be effective to extract features from plain text. They pre-train language models on large corpora, then add the pre-trained word representation into task-specific models, or directly fine-tune the language model on downstream tasks. Peters et al. (2018) tried to incorporate the pre-trained ELMo embeddings as WSD features, but there are currently no studies which fine-tune language models on WSD task.
There have been lots of works on WSD, and they can mainly be divided into two groups: supervised methods and knowledge-based methods. Supervised methods need large sense annotated corpora. They train a classifier using features extracted from the context of the polyseme. These methods can also be divided into two subgroups according to the features they use. Feature-based supervised methods use many conventional features like surrounding words, PoS tags of surrounding words, local word collectionsZhong and Ng (2010), word embeddings, and PoS tag embeddings Iacobacci et al. (2016)
. Neural-based supervised methods use a neural network encoder like BiLSTM to extract featuresMelamud et al. (2016); Yuan et al. (2016); Raganato et al. (2017a).
Knowledge-based methods rely on the structure and content of knowledge bases, for instance, sense definitions Lesk (1986); Basile et al. (2014) and semantic networks which provide the relationship and similarity between two senses Agirre and Soroa (2009); Moro et al. (2014). Supervised methods perform better than knowledge-based methods Raganato et al. (2017b), but knowledge-based methods are usually unsupervised and require no sense annotated data. Some recent studies have explored ways of using knowledge like sense definitions to enhance supervised methods Luo et al. (2018b, a).
In this paper, we fine-tune BERT on the WSD task for the first time and compare the performance of different polyseme features output by the BERT encoder. The fine-tuned model beats baselines by a large margin. Many polysemes are rare, and the sense distribution of a polyseme is usually unbalanced. So the sense annotated corpora usually lack annotations for some polysemes or particular senses. To address this issue, we consider using sense definitions because we can obtain definitions for unseen sense from lexical databases easily. We find that the incorporation of sense definition improves the performance significantly.
The contributions of this paper are: (1) We fine-tune the BERT on WSD task and achieve the state-of-the-art results. (2) We prove that external knowledge is still useful for WSD with pre-trained language models and improves the performance.
Given a sentence and some polysemes in the sentence, the WSD task aims to identify the correct senses of the polysemes in the sentence. Our model consists of an encoder and a classifier. The encoder extracts the polysemes’ features from sentences, and the classifier uses the features to predict senses. Figure 1 gives an overview of our model. We represent the sentence as a word sequence , and assume that is the polyseme. We denote the sense set of as . The definition of in the lexical databases is a word sequence . The hidden size of BERT is .
We use BERT Devlin et al. (2019) as the encoder. BERT uses the WordPiece Wu et al. (2016) embeddings as a part of inputs. The WordPiece embeddings have a fixed size of vocabulary which includes some words and some word pieces, and segments out-of-vocabulary words. For example, the word “loveliest” is segmented into two word pieces “love” and “##liest”. Then a multi-layer Transformer reads the embeddings of the word piece sequence and outputs a hidden state for every token in the sequence.
As WSD is a single sentence tagging task, the most simple way of using BERT is to insert [CLS] at the start of the input sentence, and use the hidden state output by BERT to predict the label of the -th token. However, we cannot directly apply this simple method on WSD because some polysemes (about 15% in the training dataset) may be segmented into word pieces. These polysemes correspond to at least two final hidden states. We assume that the polyseme corresponds to a hidden state list
. To obtain a fixed-sized feature vector for the classifier, we average these hidden states:
is the feature used by the classifier. Max pooling is another way of merging these hidden states:
We can also get features by concatenating and the first final hidden state . corresponds to [CLS] and encodes global information of the sentence.
We use a 2-layer MLP to predict the correct sense. The MLP outputs the sense distribution of the polyseme in the given context:
where are fully-connected linear layers, , and . The layer is specific for every polyseme.
This MLP classifier has poor performance on unseen or infrequent words and senses because of lack of data. So we introduce the sense definitions to address the issue of data scarcity. We use another BERT to encode the definition of a sense into its sense vector. Then this sense vector is used as the parameter when predicting this sense. Formally, we replace with , where is the concatenation of all sense vectors of the polyseme. The sense vectors map senses into a unified space and encode the semantic similarities into the distance between vectors. So the classification for unseen or infrequent senses can benefit from similar senses which have more training instances. We name the model without and with definitions Bert and Bert respectively.
In the training process, parameters are updated by minimizing the cross-entropy loss between the true label and the sense distribution :
where is the number of instances in the dataset, and is a one-hot vector which represents the true label of .
We evaluate our model on the English all-words tasks. We use the evaluation framework proposed by Raganato et al. (2017b). It provides five all-words fine-grained WSD datasets for evaluation: Senseval-2 (Edmonds and Cotton, 2001, SE2), Senseval-3 task 1 (Snyder and Palmer, 2004, SE3), SemEval-07 task 17 (Pradhan et al., 2007, SE7), SemEval-13 task 12 (Navigli et al., 2013, SE13), SemEval-15 task 13 (Moro and Navigli, 2015, SE15). Following previous works, we use SE7 as the validation dataset, and use the SE2, SE3, SE13, and SE15 as test datasets. The framework provides two annotated corpora for training: Semcor Miller et al. (1994) and OMSTI Taghipour and Ng (2015). We choose SemCor as our training dataset. Table 1 illustrates the statistics of these datasets. All of the datasets are annotated with WordNet Miller (1995) 3.0.
3.2 Experiment Setup
We use the BERT as the encoder. The number of Transformer layers is , the hidden layer size is , and the number of attention heads is . We tried to use the BERT, but the F1-score is nearly the same as BERT. We use Dropout in every layer of the classifier and the dropout rate is . The optimizer is Adam Kingma and Ba (2014). We reduce the learning rate during the training process. In the
-th epoch, the learning rate is. The parameters of the BERT encoder are fixed in the first epochs. We train the model for epochs and choose the model which has the best F1-score on the validation set.
We compare our model with the following baselines: the simple MFS baseline which always outputs the most common sense in the training dataset, the knowledge-based methods Lesk Basile et al. (2014) and Babelfy Moro et al. (2014), the feature-based supervised methods IMS Zhong and Ng (2010) and IMS Iacobacci et al. (2016), and the neural based methods Bi-LSTM Kågebäck and Salomonsson (2016); Raganato et al. (2017a), GAS Luo et al. (2018b), CAN and HCAN Luo et al. (2018a).
3.3 English All-word Task Results
|System||Test Datasets||Concatenation of Test Datasets||All|
Table 2 shows the F1-score of our models and baselines on the standard English All-words WSD benchmark. The Bert and Bert in the table use the average operation to merge hidden states. They don’t use the sentence vectors. Our best model improves the state-of-the-art results by 5.2%, which indicates that BERT encoder is quite powerful in the WSD task. Moreover, our models outperform previous models on all of the four datasets and PoS types. Introducing sense definitions can significantly improve the performance of the Bert model. The F1-score of Bert is better than Bert on almost all datasets and PoS types except on the adverbs, which reveals the efficiency of introducing sense definitions.
We compare the performance of Bert and Bert on words with different occurrence numbers in the training dataset. Table 3 shows the results. Compared with the Bert model, Bert achieves the largest performance improvement on unseen words ( F1-score). The improvement decreases as the word becomes frequent because the Bert model can be trained better with more instances. So we can conclude that utilizing sense definitions can enhance the model on infrequent polysemes. In addition, both models have better performance on low-frequency words. The reason is that high-frequency words usually have many senses.
We compare two variations of the BERT encoder: the way of merging hidden states of polysemes, and whether concatenating the hidden states of polysemes with the sentence vector. The results of these variations are presented in Table 4. All of the models are based on Bert. We can find that the performance of using average or max operation is nearly the same for Bert. But Concatenating the sentence vector will impair the F1-score. The reason is: the sentence vector contains the global semantic information of the entire sentence. The sense of polyseme is determined by its local context. So using the sentence vector brings too much irrelevant information.
In this paper, we fine-tune the pre-trained language models like BERT on WSD tasks for the first time. We find that our BERT-based models achieve the state-of-the-art results on the standard evaluation. We also utilize sense definitions to enhance the model on infrequent polysemes. In the future works, we will consider using the relations between senses, like hypernym and hyponym, to provide more accurate sense representations.
- Personalizing pagerank for word sense disambiguation. In Proceedings of EACL, Cited by: §1.
- An enhanced lesk word sense disambiguation algorithm through a distributional semantic model. In Proceedings of COLING, Cited by: §1, §3.2.
- Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, Cited by: §1, §2.1.
- SENSEVAL-2: overview. In Proceedings of SENSEVAL, Cited by: §3.1.
- Embeddings for word sense disambiguation: an evaluation study. In Proceedings of ACL, Cited by: §1, §3.2.
Word sense disambiguation using a bidirectional lstm.
Proceedings of the Workshop on Cognitive Aspects of the Lexicon, Cited by: §3.2.
- Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §3.2.
- Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone. In Proceedings of the International Conference on Systems Documentation, Cited by: §1.
- Leveraging gloss knowledge in neural word sense disambiguation by hierarchical co-attention. In Proceedings of EMNLP, Cited by: §1, §3.2.
- Incorporating glosses into neural word sense disambiguation. In Proceedings of ACL, Cited by: §1, §3.2.
- Context2vec: learning generic context embedding with bidirectional lstm. In Proceedings of SIGNLL, Cited by: §1.
- Using a semantic concordance for sense identification. In Proceedings of the workshop on Human Language Technology, Cited by: §3.1.
- WordNet: a lexical database for english. Communications of the ACM, pp. 39–41. Cited by: §3.1.
- Semeval-2015 task 13: multilingual all-words sense disambiguation and entity linking. In Proceedings of SemEval, Cited by: §3.1.
- Entity linking meets word sense disambiguation: a unified approach. Transactions of the Association for Computational Linguistics 2, pp. 231–244. Cited by: §1, §3.2.
- Semeval-2013 task 12: multilingual word sense disambiguation. In Proceedings of SemEval, Cited by: §3.1.
- Word sense-aware machine translation: including senses as contextual features for improved translation models.. In Proceedings of LREC, Cited by: §1.
- Deep contextualized word representations. In Proceedings of NAACL, Cited by: §1.
- Semeval-2007 task-17: english lexical sample, srl and all words. In Proceedings of SemEval, Cited by: §3.1.
- Improving language understanding by generative pre-training. Cited by: §1.
- Neural sequence learning models for word sense disambiguation. In Proceedings of EMNLP, Cited by: §1, §3.2.
- Word sense disambiguation: a unified evaluation framework and empirical comparison. In Proceedings of EACL, Cited by: §1, §3.1.
- The english all-words task. In Proceedings of SENSEVAL, Cited by: §3.1.
- One million sense-tagged instances for word sense disambiguation and induction. In Proceedings of CoNLL, Cited by: §3.1.
Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. Cited by: §2.1.
- Semi-supervised word sense disambiguation with neural models. In Proceedings of COLING, Cited by: §1.
- It makes sense: a wide-coverage word sense disambiguation system for free text. In Proceedings of ACL, Cited by: §1, §3.2.
- Word sense disambiguation improves information retrieval. In Proceedings of ACL, Cited by: §1.