in this paper we trained our model which is participated in NSURL-2019 task 7 competition Taghizadeh et al. (2019) which associated with NER for the Persian language.
Named Entity Recognition (NER) is one of the important and basic tasks in natural language processing, assigning different parts of a text to suitable named entity categories.
There are several sets of named entity (NE) categories introduced and used in different NE tagged corpora as their tagsets. For example, Peyma’s Shahshahani et al. (2018) tagset consists of person, organization, location, date, money, percent, and time, while the Arman tagset Poostchi et al. contains person, organization, location, facility, product, and event.
NER is one of the key parts of many downstream applications in NLP, such as question answering Aliod et al. (2006), information retrieval Guo et al. (2009), and machine translation Babych and Hartley (2003). As a result, the performance of NER can affect the quality of a variety of downstream applications. Furthermore, this effect is more obvious in low-resource languages because in these languages due to lack of data and tagged corpora, usually applications are implemented in pipe-line architecture unlike other languages like English which prefer to use End-to-End solutions.
Preparing basic tools in under-resourced languages by high performance can be a good solution to such languages while we counter with lack of data issue for training such tools.
We have trained conditional random field on the top of pre-trained bidirectional transformer BERT. Delvin et al. Devlin et al. (2019) introduced BERT as a pre-trained Bidirectional Transformer model for language understanding tasks. BERT achieved state of the art results in many tasks like question answering, language inference, and Named entity recognition.Devlin et al. (2019)
The need for large tagged data is the main problem with the recent supervised methods such as deep learning. Transfer learning can help this problem for under-resourced languages. Word embeddings approachMikolov et al. (2013),Bojanowski et al. (2016), Joulin et al. (2017) and Peters et al. (2018) are the first kind of transfer learning solutions. We use word embeddings for supervised tasks after we trained them unsupervised on large raw corpora of texts. By this, they can reduce the need for huge labeled data. They defer by BERT usually in many aspects like the fine-tuning step. After pretraining BERT on large row corpora, we fine-tune it for our specific supervised problem. While BERT tokenizes text by itself, it extracts contextualized embeddings for each token. BERT is pre-trained on 104 languages like Persian, and this is one of the big advantages of this model. Vaswani et al. Vaswani et al. (2017)
introduced transformer architecture and self-attention as an alternative for encoder-decoder recurrent neural networks architecture which could achieve state of the art results in English to German and English to France machine translation problem. Furthermore, the speed for training transformers is much less than recurrent neural networks in encoder-decoder architecture. CRF as a probabilistic model like hidden Markov model makes it possible to extract and consider structural dependencies among tags in data. While Encoders like BERT try to maximize likelihood by selecting best hidden representation while CRF maximizes likelihood by selecting best output tags. We achieved 88.4% CONLL F1 score in word-level and 83.5% CONLL F1 score in phrase-level evaluation on Peyma dataset. We won second place in NSURL-2019 task 7Taghizadeh et al. (2019) competition for NER task.
In section 2, we talk about previous methods for NER and solutions like transfer learning to deal with under-resourced languages. Section 3 describes BERT. Section 4 explains our model in more details, discussing the training and test phases. In section 5, we show the achieved results on experiments like evaluating our model on different datasets. Section 6 concludes the paper.
2 Related Work
This paper describes a deep learning method based on word embedding and transfer learning, for named entity recognition in Persian language. Thus, in this section we first discuss some related work on Persian NER, then some recent work on English NER, and then talk about some word embedding models which can be used in NER tasks via transfer learning.
Mortazavi and Shamsfard Mortazavi and Shamsfard (2009)
used a rule-based system to extract named entities for Persian languages. It was one of the first implementations for NER in Persian while no datasets existed in that time for evaluation. Poostchi et al.Poostchi et al. introduced new annotated Persian named-entity recognition dataset named Arman. Arman contains 250,015 tokens and 7,682 sentences. Set of entity categories consists of person, organization (like banks and ministries), location, facility, product, and event. They also trained conditional random field with bidirectional LSTM on this dataset as a base-line. Shahshahani et al. Shahshahani et al. (2018) introduced new annotated Persian named-entity recognition dataset called Peyma. Peyma contains 7,145 sentences, 302,530 tokens and 41,148 tokens with entity tags collected from 709 documents. Class distribution for both Peyma and Arman datasets are presented respectively in Fig.1 and Fig.2.
Bokaei and Mahmoudi Bokaei and Mahmoudi (2018)
trained recurrent and convolutional neural networks with CRF on Arman dataset.
Baevski et al. Baevski et al. (2019) used a novel method for training bidirectional transformer which could over perform previous work and achieved state of the art result in English NER.
Akbik et al.Akbik et al. (2018) used contextualized word embeddings extracted from character-level language model to solve the NER problem.
Delvin et al. Devlin et al. (2019) introduced BERT as a pre-trained bidirectional transformer. They used and evaluated BERT on many tasks, including NER.
Using unsupervised methods can be a promising way because the most important issue for low resource languages is the lack of labeled data while but the access to a large amount of raw texts is more probable and feasible. Today, word embeddings such as GlovePennington et al. (2014), word2vec Mikolov et al. (2013) , and fastText Joulin et al. (2017) are essential parts of many methods in NLP. These models give continuous representations in n-dimensional space for each word which contain semantic information and features about that word.
Elmo Peters et al. (2018) introduced deep contextualized word embedding by considering the context of words. Which means words have different embedding in different contexts. Delvin et al. Devlin et al. (2019) and Radford et al.Radford et al. proposed a new method with transfer self-attention blocks without the need to change in architecture for a specific problem. They suggest fine-tuning pre-trained bidirectional transformers for specific problems.
Radford et al. Radford et al. introduced a new language model called GPT.2, which could reach 55% F1 score on the CoQA dataset without any labeled data. This approach tries to remove the need for labeled data and gives a general model to solve problems against BERT, which tries just to give a general model.
best performing models before us for NER in Persian are LSTM based models which usually come with CRF and pre-trained non-contextualized embedding layers. these models are evaluated on two common datasets for NER: PEYMA and ARMAN. Bokaei and Mahmoudi Bokaei and Mahmoudi (2018) and Shahshahani et al. Shahshahani et al. (2018) had reported the best results which you can see in Table 3
BERT (Bidirectional Encoder Representations from Transformer) is a language model representation based on self-attention blocks. BERT is pre-trained in different language model tasks on raw unlabeled texts. The pre-trained deep bidirectional model with one output layer can reach state-of-the-art results in many tasks such as question answering and Multi-Genre Natural Language Inference. The idea is to have a general architecture which fits many problems and a pre-trained model which minimize the need for labeled data. For example, in Fig. 3 You see how BERT can be used in different tasks like question answering, sentences pair classification, single sentence classification, and single sentence tagging task. While each task has a different format of inputs and outputs. As mentioned before, one of the big advantages of BERT is that it was trained in 104 languages and Persian is one of those. Which motivate us to use it for NER in Persian.
4 Our Proposed Model
In this paper, we propose a method for Persian NER. In this method, we use BERT pre-trained model. As in NER task, we need to assign the most suitable tag to each token, and suitable tokenization is an important step.
While BERT has its tokenization with Byte-Pair encoding and it will assign tags to its extracted tokens, we should take care of this issue. BERT extracted tokens are always equal to or smaller than our main tokens (that taken from the Step-1 Shamsfard et al., 2010) because BERT takes tokens of our dataset one by one. As a result, we will have intra-tokens which take X tag (meaning don’t mention). We trained a conditional random field and fully connected layer after output representation of tokens extracted by BERT. It Is a fine-tuning step to make the entire model ready for NER task. You can see a schema of the model in Fig.4 .
We have trained and tested our model on two different datasets: Peyma Shahshahani et al. (2018) and Arman Poostchi et al. .We split PEYMA dataset into 5 equal subsets (Peyma contains 7145 sentences thus each subset contains 1429 sentences) and use 5-fold cross-validation. we repeated training phase 5 times separately, Each time, one of the 5 subsets is used as the test set and the remaining 4 subsets are put together to form a training set. In all experiments, CONLL F1 score is calculated in two levels: word and phrase as a metric for evaluating the performance of model. Results of our model on Peyma and Arman datasets are given respectively in Table 1 And Table 2.
On Peyma dataset We can reach 90.59% CONLL F1 score in phrase-level and 87.62% F1 score in word level. Best results are seen for Percent class and worst for Time.
On Arman dataset, We reached 79.93% CONLL F1 score in phrase-level and 84.03% F1 score on word-level. Best results are seen for Person class and worst for Event
One of the causes for achieving different results in each class is the amount of named entities in the datasets. As can be seen in Fig.1 and Fig.2, the number of phrases for Time and Event classes are much lower than others.
As you see in Table 3 in both word and phrase levels, our model outperform other NER approaches for the Persian language. Unfortunately previous works reported their results just on one of the datasets. Shahshahani et al.Shahshahani et al. (2018) reported their results just in word level evaluation on Peyma dataset.Table 3 shows that our results are 10 percent better than Shahshahani and colleagues on the same platform. On the other hand Bokaei and Mahmoudi Bokaei and Mahmoudi (2018) reported their results on Arman dataset Which is lower than ours in both word and phrase levels according to Table 3.
|Bokaei and Mahmoudi Bokaei and Mahmoudi (2018)||81.50||76.79||-||-|
|Shahshahani et al.Shahshahani et al. (2018)||-||-||80.0||-|
|Beheshti-NER (Our Model)||84.03||79.93||90.59||87.62|
The results of NSURL task-7 competition is reported in two levels of evaluation, namely word and phrase levels for two subtasks: A) NER for 3- classes (Person, Location, Organization) and B) NER for all classes. for the competition, we have trained our model on PEYMA corpus in addition to another corpus which was prepared by Iran Telecommunication Research Center (ITRC). The organizers also used two kinds of in-domain and out-domain test data. Our model won second place in all of these evaluation types.
Tables 4, 5, 6, 7 and 8 show the results of evaluation reported by competition for all teams which participated in the challenge. Our method is mentioned as Beheshti-NER-1111Code is available at https://github.com/sEhsanTaher/Beheshti-NER. Table 4 and 5 show the results for subtask A. according to the tables, we reached to 84.0% and 87.9% F1 score respectively for phrase and word level evaluations.
|Team||Test Data 1|
|In Domain||Out Domain||Total|
|Team||Test Data 1|
|In Domain||Out Domain||Total|
|Team||Test Data 1|
|In Domain||Out Domain||Total|
|Team||Test Data 1|
|In Domain||Out Domain||Total|
details of evaluation for each class in subtask B is given in Table 7. as you see all teams have higher scores in Percent class and the worst score for many teams is for Time class.
|Team||Test Data 1|
in this work we fine-tuned the pre-trained BERT model with a CRF layer in NER task for Persian language. our trained model achieved best results compared to the previous ones and ranked as the second team in NSURL competition. this work present BERT as a good transfer learning solution for solving low resource problems.
results show that our model could outperform previous methods with a dramatic difference. the reason for this could be using a big pre-trained model, BERT, which achieved state of the art results in English and proved to perform well with a less amount of data for training.
- Contextual String Embeddings for Sequence Labeling. In Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, USA, pp. 1638–1649. Cited by: §2.
- Named entity recognition for question answering. In ALTA, Cited by: §1.
- Improving machine translation quality with automatic named entity recognition. Cited by: §1.
- Cloze-driven Pretraining of Self-attention Networks. arXiv:1903.07785 [cs] (en). Note: arXiv: 1903.07785 Cited by: §2.
Enriching Word Vectors with Subword Information. arXiv:1607.04606 [cs]. Note: arXiv: 1607.04606 Cited by: §1.
- Improved Deep Persian Named Entity Recognition. In 2018 9th International Symposium on Telecommunications (IST), Tehran, Iran, pp. 381–386 (en). External Links: Cited by: §2, §2, Table 3, §5.
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), pp. 4171–4186. External Links: Cited by: §1, §2, §2, Figure 3.
- Named entity recognition in query. In Proceedings of the 32Nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, New York, NY, USA, pp. 267–274. External Links: Cited by: §1.
- Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 427–431. Cited by: §1, §2.
Efficient Estimation of Word Representations in Vector Space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, Y. Bengio and Y. LeCun (Eds.), Cited by: §1, §2.
- Named entity recognition in persian texts. Cited by: §2.
- Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, pp. 1532–1543 (en). External Links: Cited by: §2.
- ELMO: Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, pp. 2227–2237. External Links: Cited by: §1, §2.
-  BiLSTM-CRF for Persian Named-Entity Recognition. pp. 5 (en). Cited by: §1, §2, §5.
-  Language Models are Unsupervised Multitask Learners. pp. 24 (en). Cited by: §2, §2.
- PEYMA: A Tagged Corpus for Persian Named Entities. arXiv:1801.09936 [cs]. Note: arXiv: 1801.09936 Cited by: §1, §2, §2, Table 3, §5, §5.
- STeP-1: a set of fundamental tools for Persian text processing. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. External Links: Cited by: §4.
- NSURL-2019 task 7: named entity recognition (ner) in farsi. In Proceedings of the first International Workshop on NLP Solutions for Under Resourced Languages, NSURL ’19, Trento, Italy. Cited by: §1, §1.
- Attention Is All You Need. arXiv:1706.03762 [cs]. Note: arXiv: 1706.03762 External Links: Cited by: §1.