Meta Fine-Tuning Neural Language Models for Multi-Domain Text Mining

by   Chengyu Wang, et al.
East China Normal University

Pre-trained neural language models bring significant improvement for various NLP tasks, by fine-tuning the models on task-specific training sets. During fine-tuning, the parameters are initialized from pre-trained models directly, which ignores how the learning process of similar NLP tasks in different domains is correlated and mutually reinforced. In this paper, we propose an effective learning procedure named Meta Fine-Tuning (MFT), served as a meta-learner to solve a group of similar NLP tasks for neural language models. Instead of simply multi-task training over all the datasets, MFT only learns from typical instances of various domains to acquire highly transferable knowledge. It further encourages the language model to encode domain-invariant representations by optimizing a series of novel domain corruption loss functions. After MFT, the model can be fine-tuned for each domain with better parameter initializations and higher generalization ability. We implement MFT upon BERT to solve several multi-domain text mining tasks. Experimental results confirm the effectiveness of MFT and its usefulness for few-shot learning.



There are no comments yet.


page 1

page 2

page 3

page 4


How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?

The fine-tuning of pre-trained language models has a great success in ma...

NukeLM: Pre-Trained and Fine-Tuned Language Models for the Nuclear and Energy Domains

Natural language processing (NLP) tasks (text classification, named enti...

Exploring Low-dimensional Intrinsic Task Subspace via Prompt Tuning

How can pre-trained language models (PLMs) learn universal representatio...

Language Models are Few-Shot Learners

Recent work has demonstrated substantial gains on many NLP tasks and ben...

MetaXT: Meta Cross-Task Transfer between Disparate Label Spaces

Albeit the universal representational power of pre-trained language mode...

Plug-Tagger: A Pluggable Sequence Labeling Framework Using Language Models

Plug-and-play functionality allows deep learning models to adapt well to...

From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression

Pre-trained Language Models (PLMs) have achieved great success in variou...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Recent years has witnessed a boom in pre-trained neural language models. Notable works include ELMo Peters et al. (2018), BERT Devlin et al. (2019), Transformer-XL Dai et al. (2019), ALBERT Lan et al. (2019), StructBERT Wang et al. (2019b) and many others. These models revolutionize the learning paradigms of various NLP tasks. After pre-training, only a few fine-tuningepochs are required to train models for these tasks.

The “secrets” behind this phenomenon owe to the models’ strong representation learning power to encode the semantics and linguistic knowledge from massive text corpora Jawahar et al. (2019); Kovaleva et al. (2019); Liu et al. (2019a); Tenney et al. (2019). By simple fine-tuning, models can transfer the universal Natural Language Understanding (NLU) abilities to specific tasks Wang et al. (2019a). However, state-of-the art language models mostly utilize self-supervised tasks during pre-training (for instance, masked language modeling and next sentence prediction in BERT Devlin et al. (2019)). This unavoidably creates a learning gap between pre-training and fine-tuning. Besides, for a group of similar tasks, conventional practices require the parameters of all task-specific models to be initialized from the same pre-trained language model, ignoring how the learning process in different domains is correlated and mutually reinforced.

A basic solution is fine-tuning models by multi-task learning. Unfortunately, multi-task fine-tuning of BERT does not necessarily yield better performance across all the tasks Sun et al. (2019a)

. A probable cause is that learning too much from other tasks may force the model to acquire non-transferable knowledge, which harms the overall performance. A similar finding is presented in 

Bingel and Søgaard (2017); McCann et al. (2018)

on multi-task training of neural networks. Additionally, language models such as BERT do not have the “shared-private” architecture 

Liu et al. (2017) to enable effective learning of domain-specific and domain-invariant features. Other approaches modify the structures of language models to accommodate multi-task learning and mostly focus on specific applications, without providing a unified solution for all the tasks Stickland and Murray (2019); Zhou et al. (2019b); Gulyaev et al. (2020).

A recent study Finn et al. (2017) reveals that meta-learning achieves better parameter initialization for a group of tasks, which improves the models’ generalization abilities in different domains and makes them easier to fine-tune. As pre-trained language models have general NLU abilities, they should also have the ability to learn solving a group of similar NLP tasks. In this work, we propose a separate learning procedure, inserted between pre-training and fine-tuning, named Meta Fine-Tuning (MFT). This work is one of the early attempts for improving fine-tuning of neural language models by meta-learning. Take the review analysis task as an example. MFT only targets at learning the polarity of reviews (positive or negative) in general, ignoring features of specific aspects or domains. After that, the learned model can be adapted to any domains by fine-tuning. The comparison between fine-tuning and MFT is shown in Figure 1. Specifically, MFT first learns the embeddings of class prototypes from multi-domain training sets, and assigns typicality scores to individuals, indicating the transferability of each instance. Apart from minimizing the multi-task classification loss over typical instances, MFT further encourages the language model to learn domain-invariant representations by jointly optimizing a series of novel domain corruption loss functions.

Figure 1: Comparison between fine-tuning and MFT. “LM” refers to pre-trained language models.

For evaluation, we implement the MFT strategy upon BERT Devlin et al. (2019) for three multi-domain text mining tasks: i) natural language inference Williams et al. (2018) (sentence-pair classification), ii) review analysis Blitzer et al. (2007) (sentence classification) and iii) domain taxonomy construction Luu et al. (2016) (word-pair classification). Experimental results show that the effectiveness and superiority of MFT. We also show that MFT is highly useful for multi-domain text mining in the few-shot learning setting. 111Although we focus on MFT for BERT only, MFT is general and can be applied to other language models easily.

Figure 2: The neural architecture of MFT for BERT Devlin et al. (2019)

. Due to space limitation, we only show two corrupted domain classifiers and three layers of transformer encoders, with others omitted.

2 Related Work

We overview recent advances on pre-trained language models, transfer learning and meta-learning.

Pre-trained language models have gained much attention from the NLP community recently. Among these models, ELMo Peters et al. (2018) learns context-sensitive embeddings for each token form both left-to-right and right-to-left directions. BERT Devlin et al. (2019) is usually regarded as the most representative work, employing transformer encoders to learn language representations. The pre-training technique of BERT is improved in Liu et al. (2019b). Follow-up works employ transformer-based architectures, including Transformer-XL Dai et al. (2019), XLNet Yang et al. (2019), ALBERT Lan et al. (2019), StructBERT Wang et al. (2019b)

and many more. They change the unsupervised learning objectives of BERT in pre-training. After language models are pre-trained, they can be fine-tuned for a variety of NLP tasks. The techniques of fine-tuning BERT are summarized in 

Sun et al. (2019a)Cui et al. (2019) improve BERT’s fine-tuning by sparse self-attention. Arase and Tsujii (2019) introduce the concept of “transfer fine-tuning”, which injects phrasal paraphrase relations into BERT. Compared to previous methods, fine-tuning for multi-domain learning has not been sufficiently studied.

Transfer learning aims to transfer the resources or models from one domain (the source domain) to another (the target domain), in order to improve the model performance of the target domain. Due to space limitation, we refer readers to the surveys Pan and Yang (2010); Lu et al. (2015); Zhuang et al. (2019) for an overview. For NLP applications, the “shared-private” architecture Liu et al. (2017) is highly popular, which include sub-networks for learning domain-specific representations and a shared sub-network for knowledge transfer and domain-invariant representation learning. Recently, adversarial training has been frequently applied Shen et al. (2018); Hu et al. (2019); Cao et al. (2018); Li et al. (2019b); Zhou et al. (2019a), where the domain adversarial classifiers are trained to help the models to learn domain-invariant features. Multi-domain learning is a special case of transfer learning whose goal is to transfer knowledge across multiple domains for mutual training reinforcement Pei et al. (2018); Li et al. (2019a); Cai and Wan (2019). Our work also addresses multi-domain learning, but solves the problem from a meta-learning aspect.

Compared to transfer learning, meta-learning is a slightly different learning paradigm. Its goal is to train meta-learners that can adapt to a variety of different tasks with little training data Vanschoren (2018), mostly applied to few-shot learning. In NLP, existing meta-learning models mostly focus on training meta-learners for single applications, such as link prediction Chen et al. (2019), dialog systems Madotto et al. (2019), lexical relation classification Wang et al. (2020) and semantic parsing Guo et al. (2019)Dou et al. (2019) leverage meta-learning for low-resource NLU. Our work is more general and improves the generalization abilities and prediction performance of neural language models in various domains.

3 MFT: The Proposed Framework

In this section, we start with some basic notations and an overview of MFT. After that, we describe the algorithmic techniques of MFT in detail.

3.1 Overview

Denote as the training set of the th domain, where and are the input text and the class label of the th sample, respectively222Note that can be either a single sentence, a sentence pair, or any other possible input texts of the target NLP task.. is the number of total samples in . The goal of MFT is to train a meta-learner initialized from a pre-trained language model, based on the training sets of domains: . The meta-learner provides better parameter initializations, such that it can be easily adapted to each of the domains by fine-tuning the meta-learner over the training set of the th domain separately.

Due to the large parameter space of neural language models, it is computationally challenging to search for the optimal values of the meta-learner’s parameters. As discussed earlier, building a trivial multi-task learner over does not guarantee satisfactory results either Sun et al. (2019a). Here, we set up two design principles for MFT: Learning from Typicality and Learning Domain-invariant Representations, introduced as follows:

Learning from Typicality. To make the meta-learner easier to be fine-tuned to any domains, the encoded knowledge should be highly general and transferable, not biased towards specific domains. Hence, only typical instances w.r.t. all the domains should be the priority learning targets. We first generate multi-domain class prototypes from to summarize the semantics of training data. Based on the prototypes, we compute typicality scores for all training instances, treated as weights for MFT.

Learning Domain-invariant Representations. A good meta-learner should be adapted to any domains quickly. Since BERT has strong representation power, this naturally motivates us to learn domain-invariant representations are vital for fast domain adaptation Shen et al. (2018). In MFT, besides minimizing the classification loss, we jointly minimize new learning objectives to force the language model to have domain-invariant encoders.

Based on the two general principles, we design the neural architecture of MFT, with the example for BERT Devlin et al. (2019) shown in Figure 2. The technical details are introduced subsequently.

3.2 Learning from Typicality

Denote as the class label set of all domains. is the collection of input texts in that have class label , i.e., . As class prototypes can summarize the key characteristics of the corresponding data Yao et al. (2020), we treat the class prototype as the averaged embeddings of all the input texts in . Formally, we have where maps to its -dimensional embeddings. As for BERT Devlin et al. (2019), we utilize the mean pooling of representations of from the last transformer encoder as .

Ideally, we regard a training instance to be typical if it is semantically close to its class prototype , and also is not too far away from class prototypes generated from other domains for high transferability. Therefore, the typicality score  of can be defined as follows:333Here, we assume that the training instance has the class label . Because each instance is associated with only one typicality score, for simplicity, we denote it as , instead of .

where is the pre-defined balancing factor (),

is the cosine similarity function and

is the indicator function that returns 1 if the input boolean function is true and 0 otherwise.

As one prototype may not be insufficient to represent the complicated semantics of the class Cao et al. (2017), we can also generate multiple prototypes by clustering, with the th prototype to be . Here, is extended by the following formula:

where is the cluster membership weight of w.r.t. each class label .

After typicality scores are computed, we discuss how to set up the learning objectives for MFT. The first loss is the multi-task typicality-sensitive label classification loss . It penalizes the text classifier for predicting the labels of typical instances of all domains incorrectly, which is defined as:444For clarity, we omit all the regularization terms in objective functions throughout this paper.

where serves as the weight of each training instance. is the predicted probability of having the class label , with the -dimensional “[CLS]” token embeddings of the last layer of BERT (denoted as ) as features.

3.3 Learning Domain-invariant Representations

Based on previous research of domain-invariant learning Shen et al. (2018); Hu et al. (2019), we could add an additional domain adversarial classifier for MFT to optimize. However, we observe that adding such classifiers to models such as BERT may be sub-optimal. For ease of understanding, we only consider two domains and . The loss of the adversarial domain classifier is:

where if the domain is and 0 otherwise. is the predicated probability of such adversarial domain classifier. In the min-max game of adversarial learning Shen et al. (2018), we need to maximize the loss such that the domain classifier fails to predict the true domain label. The min-max game between encoders and adversarial classifiers is computationally expensive, which is less appealing to MFT over large language models. Additionally, models such as BERT do not have the “shared-private” architecture Liu et al. (2017), frequently used for transfer learning. One can also replace by asking the classifier to predict the flipped domain labels directly Shu et al. (2018); Hu et al. (2019). Hence, we can instead minimize the flipped domain loss :

We claim that, applying to BERT as an auxiliary loss does not necessarily generate domain-invariant features. When is minimized, always tends to predict the wrong domain label (which predicts for and for ). The optimization of still makes the learned features to be domain-dependent, since the domain information is encoded implicitly in , only with domain labels inter-changed. A similar case holds for multiple domains where we only force the classifier to predict the domain of the input text into any one of the reminder domains (excluding ). Therefore, it is necessary to modify which truly guarantee domain invariance and avoid the expensive (and sometimes unstable) computation of adversarial training.

In this work, we propose the domain corruption strategy to address this issue. Given a training instance of the th domain, we generate a corrupt domain label from a corrupted domain distribution . is unrelated to the true domain label , which may or may not be the same as . The goal of the domain classifier is to approximate instead of always predicting the incorrect domains as in Hu et al. (2019). In practice,

can be defined with each domain uniformly distributed, if the

domain datasets are relatively balanced in size. To incorporate prior knowledge of domain distributions into the model,

can also be non-uniform, with domain probabilities estimated from

by maximum likelihood estimation.

Consider the neural architecture of transformer encoders in BERT Devlin et al. (2019). Let be the -dimensional mean pooling of the token embeddings of in the th layer (excluding the “[CLS]” embeddings), i.e.,

where represents the -the layer embedding of the th token in , and is the maximum sequence length. Additionally, we learn a -dimensional domain embedding of the true domain label of , denoted as . The input features are constructed by adding the two embeddings: , with the typicality-sensitive domain corruption loss as:

where is the predicted probability of the input features having the corrupted domain label . We deliberately feed the true domain embedding into the classifier to make sure even if the classifier knows the true domain information from , it can only generate corrupted outputs. In this way, we force the BERT’s representations to hide any domain information from being revealed, making the representations of domain-invariant.

We further notice that neural language models may have deep layers. To improve the level of domain invariance of all layers and create a balance between effectiveness and efficiency, we follow the work Sun et al. (2019b) to train a series of skip-layer classifiers. Denote as the collection of selected indices of layers (for example, one can set for BERT-base). The skip-layer domain corruption loss is defined as the average cross-entropy loss of all classifiers, defined as follows:

In summary, the overall loss for MFT to minimize is a linear combination of and , i.e., , where is a tuned hyper-parameter.

1:  Restore BERT’s parameters from the pre-trained model, with others randomly initialized;
2:  for each domain and each class  do
3:     Compute prototype embeddings ;
4:  end for
5:  for each training instance  do
6:     Compute typicality score ;
7:  end for
8:  while number of training steps do not reach a limit do
9:     Sample a batch from ;
10:     Shuffle domain labels of to generate ;
11:     Estimate model predictions of inputs and compare them against ;
12:     Update all model parameters by back propagation;
13:  end while
Algorithm 1 Learning Algorithm for MFT

3.4 Joint Optimization

We describe how to optimize for MFT. Based on the formation of , it is trivial to derive that:

As seen, MFT can be efficiently optimized via gradient-based algorithms with slight modifications. The procedure is shown in Algorithm 1. It linearly scans the multi-domain training set to compute prototypes and typicality scores . Next, it updates model parameters by batch-wise training. For each batch , as an efficient implementation, we shuffle the domain labels to generate the corrupted labels , as an approximation of sampling from the original corrupted domain distribution . This trick avoids the computation over the whole dataset, and be adapted to the changes of domain distributions if new training data is continuously added to through time. When the iterations stop, we remove all the additional layers that we have added for MFT, and fine-tune BERT for the domains over their respective training sets, separately.

4 Experiments

We conduct extensive experiments to evaluate MFT on multiple multi-domain text mining tasks.

4.1 Datasets and Experimental Settings

We employ the Google’s pre-trained BERT model555We use the uncased, base version of BERT. See: as the language model, with dimension . Three multi-domain NLP tasks are used for evaluation, with statistics of datasets reported in Table 1:

  • Natural language inference: predicting the relation between two sentences as “entailment”, “neutral” or “contradiction”, using the dataset MNLI Williams et al. (2018). MNLI is a large-scale benchmark corpus for evaluating natural language inference models, with multi-domain data divided into five genres.

  • Review analysis: classifying the product review sentiment of the famous Amazon Review Dataset Blitzer et al. (2007) (containing product reviews of four domains crawled from the Amazon website) as positive or negative.

  • Domain taxonomy construction: predicting whether there exists a hypernymy (“is-a”) relation between two terms (words/noun phrases) for taxonomy derivation. Labeled term pairs sampled from three domain taxonomies are used for evaluation. The domain taxonomies are constructed by Velardi et al. (2013). with labeled datasets created and released by Luu et al. (2016) 666Following Luu et al. (2016), in this task, we only do the binary classification of domain term pairs as hypernymy or non-hypernymy and do not consider reconstructing the graph structures of the domain taxonomies. .

Dataset Domain #Train #Dev #Test
MNLI Telephone 83,348 2,000 -
Government 77,350 2,000 -
Slate 77,306 2,000 -
Travel 77,350 2,000 -
Fiction 77,348 2,000 -
Amazon Book 1,763 120 117
DVD 1,752 120 128
Electronics 1,729 144 127
Kitchen 1,756 119 125
Taxonomy Animal 8,650 1,081 1,076
Plant 6,188 769 781
Vehicle 842 114 103
Table 1: Statistical summarization of datasets.

Because MNLI does not contain public labeled test sets that can be used for single-genre evaluation, we randomly sample 10% of the training data for parameter tuning and report the performance of the original development sets. We hold out 2,000 labeled reviews from the Amazon dataset Blitzer et al. (2007) and split them into development and test sets. As for the taxonomy construction task, because BERT does not naturally support word-pair classification, we combine a term pair to form a sequence of tokens separated by the special token “[SEP]” as input. The ratios of training, development and testing sets of the three domain taxonomy datasets are set as 80%:10%:10%.

The default hyper-parameter settings of MFT are as follows: , and . During model training, we run epochs of MFT and further fine-tune the model in epochs for each domain, separately. The initial learning rate is set as in all experiments. The regularization hyper-parameters, the optimizer and the reminder settings are the same as in Devlin et al. (2019). In MFT, only 7K

11.5K additional parameters need to be added (depending on the number of domains), compared to the original BERT model. All the algorithms are implemented with TensorFlow and trained with NVIDIA Tesla P100 GPU. The training time takes less than one hour. For evaluation, we use 


as the evaluation metric for all models trained via MFT and fine-tuning.

Method Telephone Government Slate Travel Fiction Average
BERT (S) 82.5 84.9 78.2 83.1 82.0 82.1
BERT (Mix) 83.1 85.2 79.3 85.1 82.4 83.0
BERT (MTL) 83.8 86.1 80.2 85.2 83.6 83.8
BERT (Adv) 81.9 84.2 79.8 82.0 82.2 82.0
MFT (DC) 84.2 86.3 80.2 85.8 84.0 84.1
MFT (TW) 83.8 86.5 81.3 83.7 84.4 83.9
MFT (Full) 84.6 86.3 81.5 85.4 84.6 84.5
Table 2: Natural language inference results over MNLI (divided into five genres) in terms of accuracy (%).

4.2 General Experimental Results

We report the general testing results of MFT. For fair comparison, we implement following the fine-tuning methods as strong baselines:

  • BERT (S): It fine-tunes BERT models, each with single-domain data.

  • BERT (Mix): It combines all the domain data and fine-tunes a single BERT model only.

  • BERT (MTL): It fine-tunes BERT by multi-task fine-tuning Sun et al. (2019a).

  • BERT (Adv): It fine-tunes BERT by BERT (C) with an additional adversarial domain loss proposed in Hu et al. (2019).

We also evaluate the performance of MFT and its variants under the following three settings:

  • MFT (DC): It is MFT with domain corruption. All the typicality scores in the objective function are removed.

  • MFT (TW): It is MFT with typicality weighting. The domain corruption loss is removed.

  • MFT (Full): It is the full implementation.

The results of three multi-domain NLP tasks are reported in Table 2, Table 3 and Table 4, respectively. Generally, the performance trends of all three tasks are pretty consistent. With MFT, the accuracy of fine-tuned BERT boosts 2.4% for natural language inference, 2.6% for review analysis and 3% for domain taxonomy construction. The simple multi-task fine-tuning methods do not have large improvement, of which the conclusion is consistent with Sun et al. (2019a). Our method has the highest performance in 10 domains (genres) out of a total of 12 domains of the three tasks, outperforming previous fine-tuning approaches. For ablation study, we compare MFT (DC), MFT (TW) and MFT (Full). The results show that domain corruption is more effective than typicality weighting in MNLI, but less effective in Amazon and Taxonomy.

Method Book DVD Elec. Kit. Avg.
BERT (S) 90.7 88.2 89.0 85.7 88.4
BERT (Mix) 91.8 89.4 87.8 88.4 89.3
BERT (MTL) 92.2 89.0 88.3 88.2 89.0
BERT (Adv) 89.3 87.4 86.5 86.7 87.5
MFT (DC) 90.6 89.4 92.5 88.7 90.3
MFT (TW) 90.4 88.9 94.5 89.1 90.7
MFT (Full) 91.2 88.8 94.8 89.2 91.0
Table 3: Review analysis results over Amazon Review Dataset in terms of accuracy (%).
Method Animal Plant Vehicle Avg.
BERT (S) 93.6 91.8 84.2 89.3
BERT (Mix) 93.8 88.2 83.6 88.5
BERT (MTL) 94.2 89.2 82.4 88.6
BERT (Adv) 92.8 86.3 83.2 87.4
MFT (DC) 94.3 91.8 86.8 91.0
MFT (TW) 94.0 92.0 89.2 91.7
MFT (Full) 94.5 92.3 90.2 92.3
Table 4: Domain taxonomy construction results over Taxonomy Dataset in terms of accuracy (%).
Dataset Amazon Taxonomy
90.7 91.3
90.8 92.0
91.4 92.5
91.2 92.8
Table 5: Change of prediction accuracy when the selected layer indices take different values (%).
Typicality Domain Label Review Text
Book NEG More hate books. How could anyone write anything so wrong.
Low Kitchen NEG The spoon handle is crooked and there are marks/damage to the wood. Avoid.
Kitchen NEG The glass is quite fragile. I had two breaks.
Kitchen POS I would recommend them to everyone..and at their price, it’s a HUGE DEAL!
High Electronics NEG What a waste of money. For $300 you shouldn’t HAVE to buy a protection plan for…
Electronics NEG Do not waste your money, this was under recommendations, but I would NOT…
Table 6: Cases of review texts from Amazon Review Dataset with high and low typicality scores.
Genre With MFT? Improvement With MFT? Improvement With MFT? Improvement
No Yes No Yes No Yes
Training data 5% of the original 10% of the original 20% of the original
Telephone 70.5 74.7 4.2% 74.1 76.4 2.3% 75.9 79.8 3.9%
Government 76.5 78.1 1.6% 78.8 81.0 2.2% 80.5 82.9 2.4%
Slate 64.2 69.8 5.7% 67.6 71.8 4.2% 71.8 74.1 2.3%
Travel 71.9 75.4 3.5% 74.8 78.1 3.3% 78.3 80.3 2.0%
Fiction 69.7 73.8 4.1% 73.3 76.6 3.3% 76.2 78.4 2.2%
Average 70.5 74.4 3.9% 73.7 76.8 3.1% 76.5 79.1 2.6%
Table 7: Few-shot natural language inference results over MNLI in terms of accuracy (%).

4.3 Detailed Model Analysis

In this section, we present more experiments on detailed analysis of MFT. We first study how many training steps of MFT we should do before fine-tuning. As datasets of different tasks vary in size, we tune the epochs of MFT instead. In this set of experiments, we fix parameters as default, vary the training epochs of MFT and then run fine-tuning for 2 epochs for all domains. The results of two NLP tasks are shown in Figure 3. It can be seen that too many epochs of MFT can hurt the performance because BERT may learn too much from other domains before fine-tuning on the target domain. We suggest that a small number of MFT epochs are sufficient for most cases. Next, we tune the hyper-parameter from 0 to 0.5, with the results shown in Figure 4. The inverted-V trends clearly reflect the balance between the two types of losses in MFT, with very few exceptions due to the fluctuation of the stochastic learning process.

(a) Dataset: Amazon
(b) Dataset: Taxonomy
Figure 3: Tuning the learning epochs of MFT.
(a) Dataset: Amazon
(b) Dataset: Taxonomy
Figure 4: Tuning the hyper-parameter .

We also vary the number of corrupted domain classifiers by changing . Due to space limitation, we only report averaged accuracy across all domains, shown in Table 5. In a majority of scenarios, adding more corrupted domain classifiers slightly improve the performance, as it poses strong domain-invariance constraints to deeper layers of transformer encoders in BERT.

For more intuitive understanding of MFT, we present some cases from Amazon Review Dataset with relatively extreme (low and high) typicality scores, shown in Table 6. As seen, review texts with low scores are usually related to certain aspects of specific products (for example, “crooked spoon handle” and “fragile glass”), whose knowledge is non-transferable on how to do review analysis in general. In contrast, reviews with high typicality scores may contain expressions on the review polarity that can be frequently found in various domains (for example, “huge deal” and “a waste of money”). From the cases, we can see how MFT can create a meta-learner that is capable of learning to solve NLP tasks in different domains.

4.4 Experiments for Few-shot Learning

Acquiring a sufficient amount of training data for emerging domains often poses challenges for NLP researchers. In this part, we study how MFT can benefit few-shot learning when the size of the training data in a specific domain is small. Because the original MNLI dataset Williams et al. (2018) is large in size, we randomly sample 5%, 10% and 20% of the original training set for each genre, and do MFT and fine-tuning over BERT. For model evaluation, we use the entire development set without sampling. In this set of experiments, we do not tune any hyper-parameters and set them as default. Due to the small sizes of our few-shot training sets, we run MFT for only one epoch, and then fine-tune BERT for 2 epochs per genre.

In Table 7, we report the few-shot learning results, and compare them against fine-tuned BERT without MFT. From the experimental results, we can safely come to the following conclusions. i) MFT improves the performance for text mining of all genres in MNLI, regardless of the percentages of the original training sets we use. ii) MFT has a larger impact on smaller training sets (a 3.9% improvement in accuracy for 5% few-shot learning, compared to a 2.6% improvement for 20%). iii) The improvement of applying MFT before fine-tuning is almost the same as doubling the training data size. Therefore, MFT is highly useful for few-shot learning when the training data of other domains are available.

5 Conclusion and Future Work

In this paper, we propose a new training procedure named Meta Fine-Tuning (MFT) used in neural language models for multi-domain text mining. Experimental results show the effectiveness of MFT from various aspects. In the future, we plan to apply MFT to other language models (e.g., Transformer-XL Dai et al. (2019) and ALBERT Lan et al. (2019)) and for other NLP tasks.


  • Arase and Tsujii (2019) Yuki Arase and Jun’ichi Tsujii. 2019. Transfer fine-tuning: A BERT case study. In EMNLP-IJCNLP, pages 5392–5403.
  • Bingel and Søgaard (2017) Joachim Bingel and Anders Søgaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. In EACL, pages 164–169.
  • Blitzer et al. (2007) John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In ACL.
  • Cai and Wan (2019) Yitao Cai and Xiaojun Wan. 2019. Multi-domain sentiment classification based on domain-aware embedding and attention. In IJCAI, pages 4904–4910.
  • Cao et al. (2018) Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, and Shengping Liu. 2018.

    Adversarial transfer learning for chinese named entity recognition with self-attention mechanism.

    In EMNLP, pages 182–192.
  • Cao et al. (2017) Yixin Cao, Lifu Huang, Heng Ji, Xu Chen, and Juanzi Li. 2017. Bridge text and knowledge by learning multi-prototype entity mention embedding. In ACL, pages 1623–1633.
  • Chen et al. (2019) Mingyang Chen, Wen Zhang, Wei Zhang, Qiang Chen, and Huajun Chen. 2019.

    Meta relational learning for few-shot link prediction in knowledge graphs.

    In EMNLP-IJCNLP, pages 4216–4225.
  • Cui et al. (2019) Baiyun Cui, Yingming Li, Ming Chen, and Zhongfei Zhang. 2019. Fine-tune BERT with sparse self-attention mechanism. In EMNLP-IJCNLP, pages 3546–3551.
  • Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In ACL, pages 2978–2988.
  • Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171–4186.
  • Dou et al. (2019) Zi-Yi Dou, Keyi Yu, and Antonios Anastasopoulos. 2019. Investigating meta-learning algorithms for low-resource natural language understanding tasks. In EMNLP-IJCNLP, pages 1192–1197.
  • Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, pages 1126–1135.
  • Gulyaev et al. (2020) Pavel Gulyaev, Eugenia Elistratova, Vasily Konovalov, Yuri Kuratov, Leonid Pugachev, and Mikhail Burtsev. 2020. Goal-oriented multi-task bert-based dialogue state tracker. CoRR, abs/2002.02450.
  • Guo et al. (2019) Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. 2019. Coupling retrieval and meta-learning for context-dependent semantic parsing. In ACL, pages 855–866.
  • Hu et al. (2019) Mengting Hu, Yike Wu, Shiwan Zhao, Honglei Guo, Renhong Cheng, and Zhong Su. 2019. Domain-invariant feature distillation for cross-domain sentiment classification. In EMNLP-IJCNLP, pages 5558–5567.
  • Jawahar et al. (2019) Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In ACL, pages 3651–3657.
  • Kovaleva et al. (2019) Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In EMNLP-IJCNLP, pages 4364–4373.
  • Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A lite BERT for self-supervised learning of language representations. CoRR, abs/1909.11942.
  • Li et al. (2019a) Yitong Li, Timothy Baldwin, and Trevor Cohn. 2019a. Semi-supervised stochastic multi-domain learning using variational inference. In ACL, pages 1923–1934. Association for Computational Linguistics.
  • Li et al. (2019b) Zheng Li, Xin Li, Ying Wei, Lidong Bing, Yu Zhang, and Qiang Yang. 2019b.

    Transferable end-to-end aspect-based sentiment analysis with selective adversarial learning.

    In EMNLP-IJCNLP, pages 4589–4599.
  • Liu et al. (2019a) Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In NAACL-HLT, pages 1073–1094.
  • Liu et al. (2017) Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In ACL, pages 1–10.
  • Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  • Lu et al. (2015) Jie Lu, Vahid Behbood, Peng Hao, Hua Zuo, Shan Xue, and Guangquan Zhang. 2015. Transfer learning using computational intelligence: A survey. Knowl. Based Syst., 80:14–23.
  • Luu et al. (2016) Anh Tuan Luu, Yi Tay, Siu Cheung Hui, and See-Kiong Ng. 2016. Learning term embeddings for taxonomic relation identification using dynamic weighting neural network. In EMNLP, pages 403–413.
  • Madotto et al. (2019) Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, and Pascale Fung. 2019. Personalizing dialogue agents via meta-learning. In ACL, pages 5454–5459.
  • McCann et al. (2018) Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. CoRR, abs/1806.08730.
  • Pan and Yang (2010) Sinno Jialin Pan and Qiang Yang. 2010. A survey on transfer learning. IEEE Trans. Knowl. Data Eng., 22(10):1345–1359.
  • Pei et al. (2018) Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. 2018. Multi-adversarial domain adaptation. In AAAI, pages 3934–3941.
  • Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL-HLT, pages 2227–2237.
  • Shen et al. (2018) Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. 2018. Wasserstein distance guided representation learning for domain adaptation. In AAAI, pages 4058–4065.
  • Shu et al. (2018) Rui Shu, Hung H. Bui, Hirokazu Narui, and Stefano Ermon. 2018. A DIRT-T approach to unsupervised domain adaptation. In ICLR.
  • Stickland and Murray (2019) Asa Cooper Stickland and Iain Murray. 2019. BERT and pals: Projected attention layers for efficient adaptation in multi-task learning. In ICML, pages 5986–5995.
  • Sun et al. (2019a) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019a. How to fine-tune BERT for text classification? In CCL, pages 194–206.
  • Sun et al. (2019b) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019b. Patient knowledge distillation for BERT model compression. In EMNLP-IJCNLP, pages 4322–4331.
  • Tenney et al. (2019) Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In ACL, pages 4593–4601.
  • Vanschoren (2018) Joaquin Vanschoren. 2018. Meta-learning: A survey. CoRR, abs/1810.03548.
  • Velardi et al. (2013) Paola Velardi, Stefano Faralli, and Roberto Navigli. 2013. Ontolearn reloaded: A graph-based algorithm for taxonomy induction. Computational Linguistics, 39(3):665–707.
  • Wang et al. (2019a) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR.
  • Wang et al. (2020) Chengyu Wang, Minghui Qiu, Jun Huang, and Xiaofeng He. 2020. KEML: A knowledge-enriched meta-learning framework for lexical relation classification. CoRR, abs/2002.10903.
  • Wang et al. (2019b) Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Liwei Peng, and Luo Si. 2019b. Structbert: Incorporating language structures into pre-training for deep language understanding. CoRR, abs/1908.04577.
  • Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT, pages 1112–1122.
  • Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS, pages 5754–5764.
  • Yao et al. (2020) Huaxiu Yao, Xian Wu, Zhiqiang Tao, Yaliang Li, Bolin Ding, Ruirui Li, and Zhenhui Li. 2020. Automated relational meta-learning. CoRR, abs/2001.00745.
  • Zhou et al. (2019a) Joey Tianyi Zhou, Hao Zhang, Di Jin, Hongyuan Zhu, Meng Fang, Rick Siow Mong Goh, and Kenneth Kwok. 2019a. Dual adversarial neural transfer for low-resource named entity recognition. In ACL, pages 3461–3471.
  • Zhou et al. (2019b) Junru Zhou, Zhuosheng Zhang, and Hai Zhao. 2019b. LIMIT-BERT : Linguistic informed multi-task BERT. CoRR, abs/1910.14296.
  • Zhuang et al. (2019) Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. 2019. A comprehensive survey on transfer learning. CoRR, abs/1911.02685.