Source code and dataset for ACL 2019 paper "ERNIE: Enhanced Language Representation with Informative Entities"
Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding. We argue that informative entities in KGs can enhance language representation with external knowledge. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. The experimental results have demonstrated that ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks. The source code of this paper can be obtained from https://github.com/thunlp/ERNIE.READ FULL TEXT VIEW PDF
Neural language representation models such as Bidirectional Encoder
Pre-trained language representation models (PLMs) learn effective langua...
Pre-trained models such as BERT are widely used in NLP tasks and are
Pre-trained language representation models, such as BERT, capture a gene...
Large-scale pre-trained language models such as BERT have brought signif...
Language representation models such as BERT could effectively capture
We introduce an efficient algorithm for mining informative combinations ...
Source code and dataset for ACL 2019 paper "ERNIE: Enhanced Language Representation with Informative Entities"
Pre-trained language representation models, including feature-based Mikolov et al. (2013); Pennington et al. (2014); Peters et al. (2017, 2018) and fine-tuning Dai and Le (2015); Howard and Ruder (2018); Radford et al. (2018); Devlin et al. (2019) approaches, can capture rich language information from text and then benefit many NLP applications. BERT Devlin et al. (2019)
, as one of the most recently proposed models, obtains the state-of-the-art results on various NLP applications by simple fine-tuning, including named entity recognitionSang and De Meulder (2003), question answering Rajpurkar et al. (2016); Zellers et al. (2018), natural language inference Bowman et al. (2015), and text classification Wang et al. (2018).
Although pre-trained language representation models have achieved promising results and worked as a routine component in many NLP tasks, they neglect to incorporate knowledge information for language understanding. As shown in Figure 1, without knowing Blowin’ in the Wind and Chronicles: Volume One are song and book respectively, it is difficult to recognize the two occupations of Bob Dylan, i.e., songwriter and writer, on the entity typing task. Furthermore, it is nearly impossible to extract the fine-grained relations, such as composer and author on the relation classification task. For the existing pre-trained language representation models, these two sentences are syntactically ambiguous, like “UNK wrote UNK in UNK”. Hence, considering rich knowledge information can lead to better language understanding and accordingly benefits various knowledge-driven applications, e.g. entity typing and relation classification.
For incorporating external knowledge into language representation models, there are two main challenges. (1) Structured Knowledge Encoding: regarding to the given text, how to effectively extract and encode its related informative facts in KGs for language representation models is an important problem; (2) Heterogeneous Information Fusion
: the pre-training procedure for language representation is quite different from the knowledge representation procedure, leading to two individual vector spaces. How to design a special pre-training objective to fuse lexical, syntactic, and knowledge information is another challenge.
To overcome the challenges mentioned above, we propose Enhanced Language RepresentatioN with Informative Entities (ERNIE), which pre-trains a language representation model on both large-scale textual corpora and KGs:
(1) For extracting and encoding knowledge information, we firstly recognize named entity mentions in text and then align these mentions to their corresponding entities in KGs. Instead of directly using the graph-based facts in KGs, we encode the graph structure of KGs with knowledge embedding algorithms like TransE Bordes et al. (2013), and then take the informative entity embeddings as input for ERNIE. Based on the alignments between text and KGs, ERNIE integrates entity representations in the knowledge module into the underlying layers of the semantic module.
(2) Similar to BERT, we adopt the masked language model and the next sentence prediction as the pre-training objectives. Besides, for the better fusion of textual and knowledge features, we design a new pre-training objective by randomly masking some of the named entity alignments in the input text and asking the model to select appropriate entities from KGs to complete the alignments. Unlike the existing pre-trained language representation models only utilizing local context to predict tokens, our objectives require models to aggregate both context and knowledge facts for predicting both tokens and entities, and lead to a knowledgeable language representation model.
We conduct experiments on two knowledge-driven NLP tasks, i.e., entity typing and relation classification. The experimental results show that ERNIE significantly outperforms the state-of-the-art model BERT on these knowledge-driven tasks, by taking full advantage of lexical, syntactic, and knowledge information. We also evaluate ERNIE on other common NLP tasks, and ERNIE still achieves comparable results.
Many efforts are devoted to pre-training language representation models for capturing language information from text and then utilizing the information for specific NLP tasks. These pre-training approaches can be divided into two classes, i.e., feature-based approaches and fine-tuning approaches.
focuses on adopting feature-based approaches to transform words into distributed representations. As these pre-trained word representations capture syntactic and semantic information in textual corpora, they are often used as input embeddings and initialization parameters for various NLP models, and offer significant improvements over random initialization parametersTurian et al. (2010). Since these word-level models often suffer from the word polysemy, peters2018deep further adopt the sequence-level model (ELMo) to capture complex word features across different linguistic contexts and use ELMo to generate context-aware word embeddings.
Different from the above-mentioned feature-based language approaches only using the pre-trained language representations as input features, dai2015semi train auto-encoders on unlabeled text, and then use the pre-trained model architecture and parameters as a starting point for other specific NLP models. Inspired by dai2015semi, more pre-trained language representation models for fine-tuning have been proposed. howard2018universal present AWD-LSTM Merity et al. (2018) to build a universal language model (ULMFiT). radford2018improving propose a generative pre-trained Transformer Vaswani et al. (2017) (GPT) to learn language representations. devlin2018bert propose a deep bidirectional model with multiple-layer Transformers (BERT), which achieves the state-of-the-art results for various NLP tasks.
Though both feature-based and fine-tuning language representation models have achieved great success, they ignore the incorporation of knowledge information. As demonstrated in recent work, injecting extra knowledge information can significantly enhance original models, such as reading comprehension Mihaylov and Frank (2018); Zhong et al. (2018), machine translation Zaremoodi et al. (2018), natural language inference Chen et al. (2018), knowledge acquisition Han et al. (2018a), and dialog systems Madotto et al. (2018). Hence, we argue that extra knowledge information can effectively benefit existing pre-training models. In fact, some work has attempted to joint representation learning of words and entities for effectively leveraging external KGs and achieved promising results Wang et al. (2014); Toutanova et al. (2015); Han et al. (2016); Yamada et al. (2016); Cao et al. (2017, 2018). sun2019ernie propose the knowledge masking strategy for masked language model to enhance language representation by knowledge 111It is a coincidence that both sun2019ernie and we chose ERNIE as the model names, which follows the interesting naming habits like ELMo and BERT. sun2019ernie released their code on March 16th and submitted their paper to Arxiv on April 19th while we submitted our paper to ACL whose deadline is March 4th.. In this paper, we further utilize both corpora and KGs to train an enhanced language representation model based on BERT.
In this section, we present the overall framework of ERNIE and its detailed implementation, including the model architecture in Section 3.2, the novel pre-training task designed for encoding informative entities and fusing heterogeneous information in Section 3.4, and the details of the fine-tuning procedure in Section 3.5.
We denote a token sequence as 222In this paper, tokens are at the subword level., where is the length of the token sequence. Meanwhile, we denote the entity sequence aligning to the given tokens as , where is the length of the entity sequence. Note that is not equal to in most cases, as not every token can be aligned to an entity in KGs. Furthermore, we denote the whole vocabulary containing all tokens as , and the entity list containing all entities in KGs as . If a token has a corresponding entity , their alignment is defined as . In this paper, we align an entity to the first token in its named entity phrase, as shown in Figure 2.
As shown in Figure 2, the whole model architecture of ERNIE consists of two stacked modules: (1) the underlying textual encoder (T-Encoder) responsible to capture basic lexical and syntactic information from the input tokens, and (2) the upper knowledgeable encoder (K-Encoder) responsible to integrate extra token-oriented knowledge information into textual information from the underlying layer, so that we can represent heterogeneous information of tokens and entities into a united feature space. Besides, we denote the number of T-Encoder layers as , and the number of K-Encoder layers as .
To be specific, given a token sequence and its corresponding entity sequence , the textual encoder firstly sums the token embedding, segment embedding, positional embedding for each token to compute its input embedding, and then computes lexical and syntactic features as follows,
where is a multi-layer bidirectional Transformer encoder. As is identical to its implementation in BERT and BERT is prevalent, we exclude a comprehensive description of this module and refer readers to devlin2018bert and vaswani2017attention.
After computing , ERNIE adopts a knowledgeable encoder K-Encoder to inject the knowledge information into language representation. To be specific, we represent with their entity embeddings , which are pre-trained by the effective knowledge embedding model TransE Bordes et al. (2013). Then, both and are fed into K-Encoder for fusing heterogeneous information and computing final output embeddings,
and will be used as features for specific tasks. More details of the knowledgeable encoder K-Encoder will be introduced in Section 3.3.
As shown in Figure 2, the knowledgeable encoder K-Encoder consists of stacked aggregators, which are designed for encoding both tokens and entities as well as fusing their heterogeneous features. In the -th aggregator, the input token embeddings and entity embeddings from the preceding aggregator are fed into two multi-head self-attentions (MH-ATTs) Vaswani et al. (2017) respectively,
Then, the -th aggregator adopts an information fusion layer for the mutual integration of the token and entity sequence, and computes the output embedding for each token and entity. For a token and its aligned entity , the information fusion process is as follows,
where is the inner hidden state integrating the information of both the token and the entity.
is the non-linear activation function, which usually is the GELU functionHendrycks and Gimpel (2016). For the tokens without corresponding entities, the information fusion layer computes the output embeddings without integration as follows,
For simplicity, the -th aggregator operation is denoted as follows,
The output embeddings of both tokens and entities computed by the top aggregator will be used as the final output embeddings of the knowledgeable encoder K-Encoder.
In order to inject knowledge into language representation by informative entities, we propose a new pre-training task for ERNIE, which randomly masks some token-entity alignments and then requires the system to predict all corresponding entities based on aligned tokens. As our task is similar to training a denoising auto-encoder Vincent et al. (2008), we refer to this procedure as a denoising entity auto-encoder (dEA). Considering that the size of
is quite large for the softmax layer, we thus only require the system to predict entities based on the given entity sequence instead of all entities in KGs. Given the token sequenceand its corresponding entity sequence , we define the aligned entity distribution for the token as follows,
where is a linear layer. Eq. 7
will be used to compute the cross-entropy loss function for dEA.
Considering that there are some errors in token-entity alignments, we perform the following operations for dEA: (1) In of the time, for a given token-entity alignment, we replace the entity with another random entity, which aims to train our model to correct the errors that the token is aligned with a wrong entity; (2) In of the time, we mask token-entity alignments, which aims to train our model to correct the errors that the entity alignment system does not extract all existing alignments; (3) In the rest of the time, we keep token-entity alignments unchanged, which aims to encourage our model to integrate the entity information into token representations for better language understanding.
Similar to BERT, ERNIE also adopts the masked language model (MLM) and the next sentence prediction (NSP) as pre-training tasks to enable ERNIE to capture lexical and syntactic information from tokens in text. More details of these pre-training tasks can be found from devlin2018bert. The overall pre-training loss is the sum of the dEA, MLM and NSP loss.
As shown in Figure 3, for various common NLP tasks, ERNIE can adopt the fine-tuning procedure similar to BERT. We can take the final output embedding of the first token, which corresponds to the special [CLS] token, as the representation of the input sequence for specific tasks. For some knowledge-driven tasks (e.g., relation classification and entity typing), we design special fine-tuning procedure:
For relation classification, the task requires systems to classify relation labels of given entity pairs based on context. The most straightforward way to fine-tune ERNIE for relation classification is to apply the pooling layer to the final output embeddings of the given entity mentions, and represent the given entity pair with the concatenation of their mention embeddings for classification. In this paper, we design another method, which modifies the input token sequence by adding two mark tokens to highlight entity mentions. These extra mark tokens play a similar role like position embeddings in the conventional relation classification modelsZeng et al. (2015). Then, we also take the [CLS] token embedding for classification. Note that we design different tokens [HD] and [TL] for head entities and tail entities respectively.
The specific fine-tuning procedure for entity typing is a simplified version of relation classification. As previous typing models make full use of both context embeddings and entity mention embeddings Shimaoka et al. (2016); Yaghoobzadeh and Schütze (2017); Xin et al. (2018), we argue that the modified input sequence with the mention mark token [ENT] can guide ERNIE to combine both context information and entity mention information attentively.
In this section, we present the details of pre-training ERNIE and the fine-tuning results on five NLP datasets, which contain both knowledge-driven tasks and the common NLP tasks.
The pre-training procedure primarily acts in accordance with the existing literature on pre-training language models. For the large cost of training ERNIE from scratch, we adopt the parameters of BERT released by Google333https://github.com/google-research/bert to initialize the Transformer blocks for encoding tokens. Since pre-training is a multi-task procedure consisting of NSP, MLM, and dEA, we use English Wikipedia as our pre-training corpus and align text to Wikidata. After converting the corpus into the formatted data for pre-training, the annotated input has nearly M subwords and M entities, and discards the sentences having less than entities.
Before pre-training ERNIE, we adopt the knowledge embeddings trained on Wikidata444https://www.wikidata.org/ by TransE as the input embeddings for entities. To be specific, we sample part of Wikidata which contains entities and fact triples. The entity embeddings are fixed during training and the parameters of the entity encoding modules are all initialized randomly.
In this work, we denote the hidden dimension of token embeddings and entity embeddings as , respectively, and the number of self-attention heads as , respectively. In detail, we have the following model size: . The total parameters are about M.
The total amount of parameters of is about
M, which means the knowledgeable module of ERNIE is much smaller than the language module and has little impact on the run-time performance. And, we only pre-train ERNIE on the annotated corpus for one epoch. To accelerate the training process, we reduce the max sequence length fromto as the computation of self-attention is a quadratic function of the length. To keep the number of tokens in a batch as same as BERT, we double the batch size to . Except for setting the learning rate as , we largely follow the pre-training hyper-parameters used in BERT. For fine-tuning, most hyper-parameters are the same as pre-training, except batch size, learning rate, and number of training epochs. We find the following ranges of possible values work well on the training datasets with gold annotations, i.e., batch size: 32, learning rate (Adam): , , , number of epochs ranging from 3 to 10.
We also evaluate ERNIE on the distantly supervised dataset, i.e., FIGER Ling et al. (2015). As the powerful expression ability of deeply stacked Transformer blocks, we found small batch size would lead the model to overfit the training data. Hence, we use a larger batch size and less training epochs to avoid overfitting, and keep the range of learning rate unchanged, i.e., batch size: 2048, number of epochs: 2, 3.
As most datasets do not have entity annotations, we use TAGME Ferragina and Scaiella (2010) to extract the entity mentions in the sentences and link them to their corresponding entities in KGs.
Given an entity mention and its context, entity typing requires systems to label the entity mention with its respective semantic types. To evaluate performance on this task, we fine-tune ERNIE on two well-established datasets FIGER Ling et al. (2015) and Open Entity Choi et al. (2018). The training set of FIGER is labeled with distant supervision, and its test set is annotated by human. Open Entity is a completely manually-annotated dataset. The statistics of these two datasets are shown in Table 1. We compare our model with the following baseline models for entity typing:
NFGEC is a hybrid model proposed by shimaoka2016attentive. NFGEC combines the representations of entity mention, context and extra hand-craft features as input, and is the state-of-the-art model on FIGER. As this paper focuses on comparing the general language representation abilities of various neural models, we thus do not use the hand-craft features in this work.
For Open Entity, we add a new hybrid model UFET Choi et al. (2018) for comparison. UFET is proposed with the Open Entity dataset, which uses a Bi-LSTM for context representation instead of two Bi-LSTMs separated by entity mentions in NFGEC.
Besides NFGEC and UFET, we also report the result of fine-tuning BERT with the same input format introduced in Section 3.5 for fair comparison. Following the same evaluation criteria used in the previous work, we compare NFGEC, BERT, ERNIE on FIGER, and adopt strict accuracy, loose macro, loose micro scores for evaluation. We compare NFGEC, BERT, UFET, ERNIE on Open Entity, and adopt precision, recall, micro-F1 scores for evaluation.
The results on FIGER are shown in Table 2. From the results, we observe that: (1) BERT achieves comparable results with NFGEC on the macro and micro metrics. However, BERT has lower accuracy than the best NFGEC model. As strict accuracy is the ratio of instances whose predictions are identical to human annotations, it illustrates some wrong labels from distant supervision are learned by BERT due to its powerful fitting ability. (2) Compared with BERT, ERNIE significantly improves the strict accuracy, indicating the external knowledge regularizes ERNIE to avoid fitting the noisy labels and accordingly benefits entity typing.
The results on Open Entity are shown in Table 3. From the table, we observe that: (1) BERT and ERNIE achieve much higher recall scores than the previous entity typing models, which means pre-training language models make full use of both the unsupervised pre-training and manually-annotated training data for better entity typing. (2) Compared to BERT, ERNIE improves the precision by and the recall by , which means the informative entities help ERNIE predict the labels more precisely.
In summary, ERNIE effectively reduces the noisy label challenge in FIGER, which is a widely-used distantly supervised entity typing dataset, by injecting the information from KGs. Besides, ERNIE also outperforms the baselines on Open Entity which has gold annotations.
Relation classification aims to determine the correct relation between two entities in a given sentence, which is an important knowledge-driven NLP task. To evaluate performance on this task, we fine-tune ERNIE on two well-established datasets FewRel Han et al. (2018c) and TACRED Zhang et al. (2017). The statistics of these two datasets are shown in Table 4. As the original experimental setting of FewRel is few-shot learning, we rearrange the FewRel dataset for the common relation classification setting. Specifically, we sample instances from each class for the training set, and sample instances for the development and test respectively. There are classes in FewRel, and there are classes (including a special relation “no relation”) in TACRED. We compare our model with the following baseline models for relation classification:
With a convolution layer, a max-pooling layer, and a non-linear activation layer, CNN gets the output sentence embedding, and then feeds it into a relation classifier. To better capture the position of head and tail entities, position embeddings are introduced into CNNZeng et al. (2015); Lin et al. (2016); Wu et al. (2017); Han et al. (2018b).
zhang2017position propose PA-LSTM introducing a position-aware attention mechanism over an LSTM network, which evaluates the relative contribution of each word in the sequence for the final sentence representation.
zhang2018graph adopt the graph convolution operations to model dependency trees for relation classification. To encode the word order and reduce the side effect of errors in dependency parsing, Contextualized GCN (C-GCN) firstly uses Bi-LSTM to generate contextualized representations as input for GCN models.
In addition to these three baselines, we also fine-tune BERT with the same input format introduced in Section 3.5 for fair comparison.
As FewRel does not have any null instance where there is not any relation between entities, we adopt macro averaged metrics to present the model performances. Since FewRel is built by checking whether the sentences contain facts in Wikidata, we drop the related facts in KGs before pre-training for fair comparison. From Table 5, we have two observations: (1) As the training data does not have enough instances to train the CNN encoder from scratch, CNN just achieves an F1 score of . However, the pre-training models including BERT and ERNIE increase the F1 score by at least . (2) ERNIE achieves an absolute F1 increase of over BERT, which means fusing external knowledge is very effective.
In TACRED, there are nearly null instances so that we follow the previous work Zhang et al. (2017) to adopt micro averaged metrics to represent the model performances instead of the macro. The results of CNN, PA-LSTM, and C-GCN come from the paper by zhang2018graph, which are the best results of CNN, RNN, and GCN respectively. From Table 5, we observe that: (1) The C-GCN model outperforms the strong BERT model by an F1 increase of , as C-GCN utilizes the dependency trees and the entity mask strategy. The entity mask strategy refers to replacing each subject (and object similarly) entity with a special NER token, which is similar to our proposed pre-training task dEA. (2) ERNIE achieves the best recall and F1 scores, and increases the F1 of BERT by nearly , which proves the effectiveness of the knowledgeable module for relation classification.
In conclusion, we find that the pre-trained language models can provide more information for relation classification than the vanilla encoder CNN and RNN. And ERNIE outperforms BERT on both of the relation classification datasets, especially on the FewRel which has a much smaller training set. It demonstrates extra knowledge helps the model make full use of small training data, which is important for most NLP tasks as large-scale annotated data is unavailable.
The General Language Understanding Evaluation (GLUE) benchmark Wang et al. (2018) is a collection of diverse natural language understanding tasks Warstadt et al. (2018); Socher et al. (2013); Dolan and Brockett (2005); Agirre et al. (2007); Williams et al. (2018); Rajpurkar et al. (2016); Dagan et al. (2006); Levesque et al. (2011), which is the main benchmark used in devlin2018bert. To explore whether our knowledgeable module degenerates the performance on common NLP tasks, we evaluate ERNIE on datasets of GLUE and compare it with BERT.
In Table 6, we report the results of our evaluation submissions and those of BERT from the leaderboard. We notice that ERNIE is consistent with on big datasets like MNLI, QQP, QNLI, and SST-2. The results become more unstable on small datasets, that is, ERNIE is better on CoLA and RTE, but worse on STS-B and MRPC.
In short, ERNIE achieves comparable results with on GLUE. On the one hand, it means GLUE does not require external knowledge for language representation. On the other hand, it illustrates ERNIE does not lose the textual information after heterogeneous information fusion.
In this subsection, we explore the effects of the informative entities and the knowledgeable pre-training task (dEA) for ERNIE using FewRel dataset. w/o entities and w/o dEA refer to fine-tuning ERNIE without entity sequence input and the pre-training task dEA respectively. As shown in Table 7, we have the following observations: (1) Without entity sequence input, dEA still injects knowledge information into language representation during pre-training, which increases the F1 score of BERT by . (2) Although the informative entities bring much knowledge information which intuitively benefits relation classification, ERNIE without dEA takes little advantage of this, leading to the F1 increase of .
In this paper, we propose ERNIE to incorporate knowledge information into language representation models. Accordingly, we propose the knowledgeable aggregator and the pre-training task dEA for better fusion of heterogeneous information from both text and KGs. The experimental results demonstrate that ERNIE has better abilities of both denoising distantly supervised data and fine-tuning on limited data than BERT. There are three important directions remain for future research: (1) inject knowledge into feature-based pre-training models such as ELMo Peters et al. (2018); (2) introduce diverse structured knowledge into language representation models such as ConceptNet Speer and Havasi (2012)
which is different from the world knowledge database Wikidata; (3) annotate more real-world corpora heuristically for building larger pre-training data. These directions may lead to more general and effective language understanding.
This work is funded by the Natural Science Foundation of China (NSFC) and the German Research Foundation (DFG) in Project Crossmodal Learning, NSFC 61621136008 / DFG TRR-169, the National Natural Science Foundation of China (NSFC No. 61572273) and China Association for Science and Technology (2016QNRC001).