Learning Word Embeddings with Domain Awareness

06/07/2019 ∙ by Guoyin Wang, et al. ∙ Duke University 0

Word embeddings are traditionally trained on a large corpus in an unsupervised setting, with no specific design for incorporating domain knowledge. This can lead to unsatisfactory performances when training data originate from heterogeneous domains. In this paper, we propose two novel mechanisms for domain-aware word embedding training, namely domain indicator and domain attention, which integrate domain-specific knowledge into the widely used SG and CBOW models, respectively. The two methods are based on a joint learning paradigm and ensure that words in a target domain are intensively focused when trained on a source domain corpus. Qualitative and quantitative evaluation confirm the validity and effectiveness of our models. Compared to baseline methods, our method is particularly effective in near-cold-start scenarios.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Word embeddings have shown their effectiveness in many natural language processing (NLP) tasks

[Turney and Pantel2010, Collobert et al.2011, Weston et al.2015]

. To learn embeddings, unsupervised learning is conducted over a large generic corpus, e.g., news text, so as to ensure coverage on various language phenomena and unbiasedness in semantics

[Mikolov et al.2013, Song et al.2018, Song et al.2017]. Consequently, the resulted embeddings are versatile and can be applied to different tasks, however, losing their characteristics in representing domain-specific information.

Practically, when conducting tasks in a certain domain, one often has to use the generic embeddings and rely on fine-tuning in later supervised learning

[Collobert et al.2011]. In this situation, starting with generic embeddings can lead to a tough training process where there exists word distribution mismatch across domains. [Wendlandt et al.2018]

also confirmed that domain variance greatly affects the quality of word embeddings where their stability within domain is greater than that across domains. To obtain domain-specific embeddings, previous work mainly focuses on capturing domain-specific characteristics

[Bollegala et al.2015, Yang et al.2017] .

Although existing approaches show their effectiveness in learning embeddings by considering domain variances, they are still restricted by relying on massive resources from a target domain. On the other hand, obtaining sufficient target domain data can be challenging, which brings a cold-start issue for embedding training. For example, there may be not sufficient training samples that represent different features (e.g., words and contexts etc.) in such a target domain, thus affecting regularization and retrofitting approaches, which rely on the availability of domain-specific data or alternative high quality semantic resources.

To address the aforementioned problem, we propose an approach for minimizing the impact of target domain data availability. In particular, we aim to learn domain-aware word embeddings by directly exploiting target-domain words and contexts in a sparse corpus. Two different algorithms are proposed to this end. The first leverages a domain indicator vector

[Liu et al.2018, Stymne et al.2018] to present domain information in the learning process. The second uses a domain attention mechanism to highlight those training samples in the source domain that can also be present in the target domain. Distinguishing from existing methods based on word frequency, both of our methods utilize pairwise word concurrence information, which potentially avoid unitary invariance problem (rotate two words’ embeddings without changing distance in between) in shifting word embedding [Yin and Shen2018].

Experimental results show that our approaches can effectively learn embeddings with target-specific knowledge, outperforming embeddings learned solely on a source or target domain, as well as different aggregation methods. In these tasks, we also demonstrate that our models achieve better results compared to the state-of-the-art methods in previous studies.

2 Related Work

Word embeddings have been extensively researched in recent years [Collobert et al.2011, Mikolov et al.2013, Pennington et al.2014]. To align embeddings with specific task or application, there are studies focusing on enhancing embeddings, such as using additional data sources to adapt initial embeddings [Maas et al.2011, Yang and Eisenstein2015], incorporating external information and refining objective functions [Wieting et al.2015, Nguyen et al.2016]

, and retrofitting approaches that leverage word relations defined in semantic lexicons or other resources to help adjusting and refining embeddings

[Faruqui et al.2015, Kiela et al.2015]. Although embedding quality can be improved through external guidance, there is a high barrier from the unavailability of annotated data, especially for domain specific scenarios.

When domain variance is considered, bollegala2015-ACL bollegala2015-ACL proposed a joint model to learn unsupervised cross-domain word embeddings, assuming that pivot words frequently shared in both source and target domain have similar behaviors. However, the assumption does not hold if the source and target domains are much different. yang2017-EMNLP yang2017-EMNLP proposed a retrofitting method with frequency driven regularization of shared words across domains. xu2018-IJCAI xu2018-IJCAI further replace simple word frequency with co-occurrence counts of frequent words in the context, which is similar to our motivation. However, their model directly learns word embeddings on each target domain (task) and hence is unable to deal with domains with scarce data, which is also proved to be a common limitation for bollegala2015-ACL bollegala2015-ACL and yang2017-EMNLP yang2017-EMNLP. In comparison, our approach offers an alternative way for learning reliable domain adapted word embeddings when in-domain data is limited.

3 The Approach

To learn word embeddings with domain awareness, it requires to capture domain characteristics effectively and smartly, especially when target domain data is limited. In general, such domain characteristics are carried by word behaviors (such as distribution, usage, etc.) in different domains. When targeting on a particular domain, word embedding approaches are expected to enhance the behaviors of the words from that domain so that domain characteristics are learned accordingly. Conventionally in doing so, words appear in a target domain are emphasized when training them from a source domain, which is confirmed by many previous studies [Bollegala et al.2015, Yang et al.2017]. In this paper, we follow this paradigm so that the learning can be done in an unsupervised manner without other prerequisites.

We propose two models to learn domain aware embeddings based on the widely used skip-gram (SG) and continuous bag-of-words (CBOW) models, which are two algorithms to train word embeddings in word2vec [Mikolov et al.2013]. Our proposed models jointly learn word relations among its context from source and target domain. To deliver domain-specific knowledge, we use word-word pairs extracted from the target domain to depict word relations in it, where such relations are key component utilized in our models. Particularly, we have different adaptation strategies for SG and CBOW, regarding to their nature in cooperating with domain-specific knowledge in different ways. Details are illustrated as follows.

3.1 Learning with Domain Indicator

Figure 1: Illustration of the SG-DI model. “DI” refers to the domain indicator, with solid arrows presenting prediction, dashed arrows indicating word-word pair inspection.

Inspired by blitzer2006-EMNLP blitzer2006-EMNLP and daume2007-ACL daume2007-ACL, words shared in two domains can be leveraged to transfer knowledge across domains. With word-word pairs extracted from a target domain, a domain indicator is introduced in this proposed model, namely, SG-DI, for each word in the source domain context to learn whether it is observed with a given word in the target domain. The proposed model is adapted on the SG model, which is performed based on the assumption that a word can be modeled according to the relations among its neighboring words. Basically, the SG model predicts the context with a given word, formulated as maximizing the likelihood


over a corpus , where and refer to the given word and context word respectively. is corpus size; defines the window size. The

function in this model is conventionally the probability

to predict context word, estimated by


with referring to the vocabulary. Note that for a large vocabulary, SG, as well as CBOW, uses hierarchical softmax or negative sampling [Mikolov et al.2013] to address the computational complexity requiring matrix multiplication, where is the dimension of word embeddings.

Based on the aforementioned fundamentals, in learning with the domain indicator, a softmax function


is designed to measure whether a word is associated with another word in the target domain, by introducing a new vector for each to present its appearance in the target domain with . The function shares an updating paradigm similar to negative sampling:


denotes the sigmoid function and

the discounting learning rate. Particularly, is the target label specifying the relative direction of given , defined as


where represents that and co-occurred in the target domain and for otherwise. The final model is defined as Equation 1 with .

Formally, this model is equal to feature augmentation where the domain indicator plays as the augmented feature specifying the domain variance. Mathematically the validity of doing so was explained by daume2007-ACL daume2007-ACL that each feature has multiple versions for different domains, where the nature of target domain is emphasized when training with such augmented features. Note that in the extreme case where there is no target domain word pairs utilized in this model, SG-DI becomes the ordinary SG model with constantly. Thus in this sense SG-DI can be seen a generalized SG model with considering domain variance.

3.2 Learning with Domain Attention

In addition to the “hard decision” mode carried out by domain indicator, we also propose another model, namely CBOW-DA, that “softly” emphasizes the domain-specific context related to a target domain. The assumption behind this model is that the words from the target domain that appears in the source domain are considered heavy weighted against other words that only appears in the source domain. In doing so, words in the context receive a non-uniformed weighting scheme with regards to their domain-specific contributions to the target word. Particularly, this weighting scheme can be seen as a domain attention mechanism for different words where the ones from the target domain are heavier weighted than others.

Figure 2: Illustration of the CBOW-DA model. “DA” refers to the domain attention, with black solid arrows presenting prediction, dashed arrows indicating word-word pair inspection. is the element-wise production.

CBOW-DA is constructed based on the CBOW structure, to formally illustrate our CBOW-DA, we start from the basics of CBOW, which is different from SG, focuses on maximizing the likelihood that a word is predicted from its context. Particularly, for a given corpus with vocabulary , CBOW can be formulated as maximizing the likelihood


where denotes the context of word in a window with size . A projection layer takes the sum of the embeddings from the context words, resulting in over all context words from to . The probability of predicting with is defined as


where is the embedding for , and and refer to input and output embeddings, respectively.

To adapt CBOW with domain attention, as illustrated in Figure 2, we introduce an function to compute the attention assigning to each word pair , with considering its domain information. Particularly, is formulated as


where it is contributed by two parts: indicates how much attention a word should have in predicting , which is a measurement for the word associations in the source domain; is the domain factor similar to the indicator defined in Eq. 4, reflecting if and co-occurred () in the target domain corpus or not (). The formulation intends to assign words appeared in the target domain with higher weights than others, so that these words contribute more to the learning process. With the domain attention, the projection layer in the updated CBOW model is thus rewritten as

Followed by the common setting in previous work [Bollegala et al.2015, Yang et al.2017] that source and target domain should share some data, CBOW-DA also assumes there exist enough cases that words appear in both domains. However, similar to SG-DI, in extreme cases where no target domain words are observed, the proposed attention mechanism still works for each word where , thus their attention is determined entirely by word associations in the source domain.

4 Experiments

Experiments are done to illustrate the effectiveness of our approach. The corpus in the source domain that we used to train word embeddings is from the latest articles dumped from Wikipedia444https://dumps.wikimedia.org/enwiki/latest/ (Wiki for short), which contains approximately 2 billion words. All the embeddings trained with different approaches share the same hyper-parameters, i.e., 200 dimensions, minimum count of words to 5, a windows size of 5 words, using negative sampling as the learning strategy with 10 negative samples. These setups are used throughout the following experiments.

In addition to the aforementioned common settings, to facilitate our models, word-word tables are built for each target domain to record word co-occurrences. To match our setting for embedding training, co-occurred words are only considered within a context window of 5 words, i.e., all word pairs are extracted in a similar way as training embeddings. In doing so, when word-word pairs from the target domain are found during training on the source domain corpus, they are utilized by SG-DI or CBOW-DA accordingly.

We first conduct qualitative analysis on the generated embeddings when using IMDB corpus [Maas et al.2011] as the target domain dataset, where embeddings learned on different target domain knowledge are compared. Then we employ downstream tasks, i.e., text classification, sequence labeling, to assess the quality of different embeddings. The following baseline models are employed in comparing with ours:

Figure 3: Illustration of embedding difference against word frequency. Embedding difference for each word is measured by cosine distance () between its source and adapted embeddings; all words’ frequencies are normalized by Sørensen-Dice coefficient.
  • Source embeddings (S), CBOW and SG models that trained from the source domain, which is the Wiki corpus in this paper.

  • Target embeddings (T), CBOW and SG models that trained from the target domain, which varies in different tasks.

  • Embeddings from all data (+), CBOW and SG models are trained from the corpus from both the source and target domain.

  • Concatenation (), embeddings that are separately trained from the source and target domain and then concatenated. The concatenation is done between the same typed embeddings, e.g., CBOW ST, SG ST.

  • Averaged (Avg), embeddings that are separately trained from the source and target domain and then averaged. Similar to concatenation, average is also done between the same typed embeddings, e.g., CBOW Avg(S,T), SG Avg(S,T).

  • The cross-domain regularized embedding (CRE) method, which is proposed by yang2017-EMNLP yang2017-EMNLP using a simple regularization algorithm to train cross-domain word embeddings.555It is reported in yang2017-EMNLP yang2017-EMNLP that the “DARep” proposed by bollegala2015-ACL bollegala2015-ACL has a similar setting but significantly worse performance than theirs, we thus do not include DARep in our baselines.

  • CTCB and CTSG

    , the embeddings based on reinforcement learning

    [Song and Shi2018] that uses contexts from the target domain to adapt initial word embeddings learned from the source domain.

Figure 4: Visualization of CBOW embeddings for words spielberg, director, film and movie. From left to right, the embeddings are trained from the source domain (S), cross-domain (S T, i.e., CBOW-DA) and the target domain (T).

Once word pairs are prepared, it is straightforward to use these pairs with some retrofitting methods to adapt pretrained embeddings to a target domain. We add two such methods from faruqui2015-NAACL-HLT faruqui2015-NAACL-HLT and kiela2015-EMNLP kiela2015-EMNLP respectively as additional baselines to illustrate how they are performed in the cross domain setting.

4.1 Qualitative Analysis

To demonstrate the benefit of our models from learning word pair information, we first qualitatively compare adaptation effect between our models and others which only utilize frequency information from single words for adaptation. Herein we choose the state-of-art model, CRE [Yang et al.2017], as the representative of such kind of models for comparison.

In this analysis, we use the movie domain and choose IMDB review dataset [Maas et al.2011] as the in-domain corpus, which is a widely used dataset for sentiment classification with 100k positive and negative movie reviews in total. This dataset contains over 26M word tokens, which can afford training reasonable SG and CBOW embeddings for the target domain.

To effectively quantify embedding shifting, we calculate the cosine distance between embeddings trained on source domain and shifted embeddings from each model. Following the setup in CRE, we use Sørensen-Dice coefficient [Dice1945] to represent the normalized word frequency.

As shown in Figure 3, it is observed that embedding trained by CRE has clear separation in shifting: when the frequency of a word is high, its embedding directly follows source domain embedding; when the frequency is low, its embedding remains far away from the source domain and hence barely influenced by the adaptation process. Moreover, Figure 3 also shows that most words in target domain has low normalized frequency. The CRE method hence does not efficiently utilize source domain information and still suffer from insufficient target domain data. On the contrary, our models shift word embeddings more evenly over different word frequency. Particularly noticed from Figure 3 that, for words with Sørensen-Dice coefficient larger than , our model consistently shift their embeddings with distances around 0.8 to 1. Such findings indicate that, by utilizing word pair information, our model consistently shift embeddings without significant influence from word frequencies.

To further investigate how word pair knowledge improve the embedding quality for the target domain, we project the embedding vectors to a 2D plane using principal components analysis (PCA) and visualize a group of strong correlated movie-related words. Smaller distance between two nodes on the plots indicates higher relationship. As shown in

4, by utilizing word pair information, our model significantly decrease the distance in between and show a more significant cluster. Since the sentence representation on many tasks greatly rely on word embedding aggregation[Joulin et al.2016], better clustering can provide a better prior knowledge and potentially benefit the downstream tasks.

Embeddings  ATIS  IMDB
CBOW S   96.19 90.22
SG S   96.30 90.16
CBOW T   95.74 91.10
SG T   95.30 90.67

  96.19 90.32
SG S + T   96.30 90.28

  96.09 90.55
SG S T   95.86 90.87

  95.86 90.38
SG Avg(S,T)   95.52 90.45

  95.86 91.22

Faruqui et al. faruqui2015-NAACL-HLT
  94.95 89.18

Kiela et al. kiela2015-EMNLP
  95.41 90.45
CTCB   97.42 91.09
CTSG   97.20 90.84
SG-DI   97.64 91.89
CBOW-DA   97.75 92.34

Table 1: Text classification results on ATIS and IMDB.

4.2 Text Classification

The first downstream application to evaluate our embedding models is text classification, where two target domains are considered in the experiment. One task is performed on intent classification in the flight domain, for which we use the ATIS666ATIS is short for Air Travel Information System. [Hemphill et al.1990] corpus for intent classification. The other task is sentiment classification in the movie domain where the IMDB [Maas et al.2011] corpus used in the previous section is employed for this task. The details of datasets are illustrated in Appendix. Note that, in using ATIS and IMDB, we can better evaluate our proposed models with respect to different sized data in the target domain.

Figure 5:

Learning curves of the first five epochs on the IMDB task with using different embeddings.

For both ATIS and IMDB, we use a bi-directional LSTM (biLSTM) model to encode each utterance or review for classification. We have different settings for the hidden states in the biLSTM model for the two tasks. The ATIS task is performed on a biLSTM with the hidden state size of 256. For IMDB, because the reviews are longer than usual sentences, we follow the setting of hidden state size 1024 as suggested in dai2015-NIPS dai2015-NIPS. Embeddings from different models are used as input for the aforementioned classifiers. In training, all models use cross-entropy loss and Adam optimizer

[Kingma and Ba2014] for parameter updating. Other hyper-parameters (e.g, training epoch and learning rate) are determined by using 15% of the training data for validation and we apply early stopping strategy when the error rate on the validation set starts to increase.

Results are reported in Table 1 for the two tasks with different input embeddings. Overall, in-domain word embeddings tend to outperform out-of-domain ones when in-domain data is relatively large and in a contrary when in-domain data is limited. The aggression methods, i.e., ST, Avg(S,T), normally perform in between the embeddings from the source and the target domain; while the embeddings trained on the concatenated corpus (S+T) tend to perform similar to the ones trained in the source domain because the Wiki corpus dominate the in-domain corpora. Against all aggression methods and baselines from other studies, our SG-DI and CBOW-DA constantly outperform them with a large margin. Particularly, no matter whether there is enough in-domain data, SG-DI and CBOW-DA effectively integrate domain knowledge into embeddings and thus result better performance in text classification. When there are more in-domain data, CBOW-DA demonstrates a larger margin on top of other baselines, as well as SG-DI. Especially, the performance of CBOW-DA on ATIS achieve a record score 97.75, which is, to the best of our knowledge, the state-of-the-art result reported on this dataset. To compare between SG-DI and CBOW-DA, the reason leads to the difference of their performance could be that the attention mechanism works more smooth than the indicator.

Embeddings P R F1
CBOW S 78.69 70.75 74.51
SG S 76.79 72.99 74.84
CBOW T 76.14 71.18 73.57
SG T 76.89 71.08 73.87

76.24 72.61 74.38
SG S + T 75.64 72.91 74.25

75.00 72.97 73.97
SG S T 75.47 72.97 74.20

74.81 71.62 73.18
SG Avg(S,T) 74.91 72.13 73.49

77.44 70.47 73.79

Faruqui et al. faruqui2015-NAACL-HLT
72.86 69.15 70.96

Kiela et al. kiela2015-EMNLP
73.86 70.15 71.96

76.52 73.03 74.73
CTSG 75.94 72.31 74.08

77.04 74.18 75.59
CBOW-DA 76.76 73.30 75.51

Table 2: NER results on GENIA.

Consider that most complex tasks rely on tuning embeddings in an end-to-end fashion, it is from another point of view to evaluate our embeddings by measuring how they can provide a better staring point for training models for target domain tasks. In doing so, we take the IMDB task as the case study to show learning curves on test data when training with different embeddings for the first five epochs. Clearly demonstrated in Figure 5, embeddings learned from the target domain have significant better initial performance than that from the source domain. Although learned from the source domain, SG-DI and CBOW-DA perform similar to the target domain embeddings, while other baseline models are inferior to them. As a result, a classifier could benefit from less iterations of training when using such embeddings.

4.3 Sequence Labeling

Embeddings P R F1
CBOW S 85.13 85.29 85.21
SG S 84.96 84.86 84.91
CBOW T 79.51 79.78 79.64
SG T 79.55 80.18 79.86

85.14 85.29 85.21
SG S + T 85.00 84.89 84.94

84.24 84.62 84.43
SG S T 84.02 84.40 84.21

84.38 84.72 84.55
SG Avg(S,T) 84.12 84.58 84.35

81.42 81.42 81.42

Faruqui et al. faruqui2015-NAACL-HLT
81.12 81.52 81.32

Kiela et al. kiela2015-EMNLP
82.04 82.34 82.19

85.25 85.08 85.16
CTSG 85.23 84.85 85.04

85.44 85.42 85.43
CBOW-DA 85.41 85.93 85.67

Table 3: POS tagging results on Twitter.

In addition to text classification, we conduct two sequence labeling tasks to evaluate the quality of our embeddings from another type of downstream tasks. The first one is named entity recognition (NER) on the GENIA v.3.02 dataset

[Ohta et al.2002], which contains 2,000 MEDLINE abstracts, 557,779 tokens, 18,546 sentences and is annotated with 36 fine-grained biological entities. The second task is part-of-speech (POS) tagging in the social media domain, which is performed on the ARK dataset [Gimpel et al.2011], which contains 1,827 tweets with manual POS annotations on English tweets. The preprocessing details of the two datasets are illustrated in Appendix.

Note that, in order to test the effectiveness of our models, we only use the dataset provided in both tasks as the target domain data, which is different from previous studies [Bollegala et al.2013, Yang et al.2017] that external in-domain data are provided. We use a bidirectional LSTM-CRF [Huang et al.2015, Lample et al.2016] model as the labeler taking different pretrained embeddings as input. The LSTM state size is set to 200. All other settings are the same with the model for text classification. Precision (P), recall (R) and F1 score are used to evaluate the performance of this task.

The NER results are reported in Table 2. Overall the embeddings trained in the target domain show worse performance than those trained in the source domain; all the aggregation baselines perform worse than trained on source domain. Such findings indicate that with limited data, the quality of embeddings trained on the target domain is not guaranteed. Interestingly, we find the retrofitting methods obtain the worst results, which show the limitation of such methods in the cold-start scenario with limited data in the target domain. Compared to other embeddings, both our models significantly improve the recall and F1 score. To investigate, we notice that the model with our embeddings extract more entities without greatly reducing precision, while all other embeddings, on the contrary, suffer from precision drop when introducing target domain information. Consider the entities is usually domain-specific terms, high recall is thus an indicator of how the model is performed with target domain knowledge. Particularly, our SG-DI and CBOW-DA even outperform the state-of-art layered-BiLSTM-CRF model [Ju et al.2018] on GENIA NER with a simple BiLSTM-CRF model.

The POS tagging results are reposted in Table 3. Owing to the limited target domain data, the overall trend of the results from the source, target and aggregated embeddings is similar to the NER task, while our SG-DI and CBOW-DA also show improvement over all baselines. Such results illustrate the effectiveness of our models in utilizing and combining the source and target domain information. Compared to other baseline models, the superiority of our models in both NER and POS tagging may be from the modeling of word-pair relationship, while other models are built from word frequency [Yang et al.2017] or context [Song and Shi2018] from the target domain. Especially when data target domain is limited, such word-pair information is more robust than others in depicting word-word relations. Moreover, since our SG-DI and CBOW-DA embedding are trained on the source domain corpus, they do not suffer from the data sparsity issue as that in target domain embeddings and those retrofitting methods.

5 Conclusion

In this paper, we proposed two domain adaptation methods, namely, domain indicator and domain attention, to enhance the widely used SG and CBOW model with domain awareness. The two methods leverage word-word relations from the target domain and use them to adjust embedding learning on the source domain. Experimental results confirm that, even though trained from the source domain, our embeddings outperform all other baselines in the target domain, especially the embeddings directly from the target domain. More importantly, for scenarios that in-domain data is extremely limited, our proposed models still perform robustly, which illustrate their reliability and adaptability than other methods.


  • [Blitzer et al.2006] J. Blitzer, R. McDonald, and F. Pereira. Domain Adaptation with Structural Correspondence Learning. In EMNLP, 2006.
  • [Bollegala et al.2013] Danushka Bollegala, Yutaka Matsuo, and Mitsuru Ishizuka. Minimally supervised novel relation extraction using a latent relational mapping. TKDE, 2013.
  • [Bollegala et al.2015] Danushka Bollegala, Takanori Maehara, and Ken-ichi Kawarabayashi. Unsupervised Cross-Domain Word Representation Learning. In ACL, 2015.
  • [Collobert et al.2011] Ronan Collobert, Jason Weston, Léon Bottou, and etc. Natural Language Processing (Almost) from Scratch. JMLR, 2011.
  • [Dai and Le2015] Andrew M. Dai and Quoc V. Le. Semi-supervised Sequence Learning. In NIPS, 2015.
  • [Daumé III2007] Hal Daumé III. Frustratingly Easy Domain Adaptation. In ACL, 2007.
  • [Dice1945] Lee R Dice. Measures of the amount of ecologic association between species. Ecology, 1945.
  • [Faruqui et al.2015] Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, and etc. Retrofitting Word Vectors to Semantic Lexicons. In NAACL-HLT, 2015.
  • [Gimpel et al.2011] Kevin Gimpel, Nathan Schneider, Brendan O’Connor, and etc. Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments. In ACL, 2011.
  • [Hemphill et al.1990] Charles T. Hemphill, John J. Godfrey, and George R. Doddington. The ATIS Spoken Language Systems Pilot Corpus. In Workshop on Speech and Natural Language, 1990.
  • [Huang et al.2015] Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv preprint, abs/1508.01991, 2015.
  • [Joulin et al.2016] Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of Tricks for Efficient Text Classification. arXiv preprint, 2016.
  • [Ju et al.2018] Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. A neural layered model for nested named entity recognition. In NAACL, 2018.
  • [Kiela et al.2015] Douwe Kiela, Felix Hill, and Stephen Clark. Specializing word embeddings for similarity or relatedness. In EMNLP, 2015.
  • [Kingma and Ba2014] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv preprint, abs/1412.6980, 2014.
  • [Lample et al.2016] Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, and etc. Neural Architectures for Named Entity Recognition. In NAACL, 2016.
  • [Liu et al.2018] Qi Liu, Yue Zhang, and Jiangming Liu. Learning Domain Representation for Multi-Domain Sentiment Classification. In NAACL, 2018.
  • [Maas et al.2011] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts.

    Learning Word Vectors for Sentiment Analysis.

    In ACL, 2011.
  • [Mikolov et al.2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed Representations of Words and Phrases and their Compositionality. In NIPS. 2013.
  • [Muis and Lu2017] Aldrian Obaja Muis and Wei Lu. Labeling Gaps Between Words: Recognizing Overlapping Mentions with Mention Separators. In EMNLP, 2017.
  • [Nguyen et al.2016] Kim Anh Nguyen, Sabine Schulte im Walde, and Ngoc Thang Vu. Integrating distributional lexical contrast into word embeddings for antonym-synonym distinction. In ACL, 2016.
  • [Ohta et al.2002] Tomoko Ohta, Yuka Tateisi, and Jin-Dong Kim. The GENIA Corpus: An Annotated Research Abstract Corpus in Molecular Biology Domain. In HLT, 2002.
  • [Pennington et al.2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global Vectors for Word Representation. In EMNLP, 2014.
  • [Song and Shi2018] Yan Song and Shuming Shi. Complementary learning of word embeddings. In JICAI, 2018.
  • [Song et al.2017] Yan Song, Chia-Jung Lee, and Fei Xia. Learning word representations with regularization from prior knowledge. In CoNLL 2017, 2017.
  • [Song et al.2018] Yan Song, Shuming Shi, Jing Li, and Haisong Zhang. Directional skip-gram: Explicitly distinguishing left and right context for word embeddings. In NAACL, 2018.
  • [Stymne et al.2018] Sara Stymne, Miryam de Lhoneux, Aaron Smith, and Joakim Nivre. Parser Training with Heterogeneous Treebanks. In ACL, 2018.
  • [Turney and Pantel2010] Peter D. Turney and Patrick Pantel. From Frequency to Meaning: Vector Space Models of Semantics. JAIR, 2010.
  • [Wendlandt et al.2018] Laura Wendlandt, Jonathan K. Kummerfeld, and Rada Mihalcea. Factors Influencing the Surprising Instability of Word Embeddings. In NAACL, 2018.
  • [Weston et al.2015] Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. arXiv preprint, 2015.
  • [Wieting et al.2015] John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. From Paraphrase Database to Compositional Paraphrase Model and Back. TACL, 2015.
  • [Xu et al.2018] Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. Lifelong Domain Word Embedding via Meta-Learning. 2018.
  • [Yang and Eisenstein2015] Yi Yang and Jacob Eisenstein. Unsupervised Multi-Domain Adaptation with Feature Embeddings. In NAACL-HLT, 2015.
  • [Yang et al.2017] Wei Yang, Wei Lu, and Vincent Zheng. A Simple Regularization-based Algorithm for Learning Cross-Domain Word Embeddings. In EMNLP, 2017.
  • [Yin and Shen2018] Zi Yin and Yuanyuan Shen. On the dimensionality of word embedding. NIPS, 2018.

Appendix A Experiment Details

a.1 Text Classification Setup

The ATIS corpus contains 5871 utterances with 26 different intents777It is the conventional setting that in the 26 intents, some of the intents are combinations of multiple single intents. such as flight, aircraft, airfare, etc. The task is thus to assign one of the 26 intents to an given utterance. In total there are 4978 utterances for training and 893 for testing. We use all the data in training and test data without labels as the unlabeled data set for training embeddings from the target domain.

In the total 100k reviews, the IMDB dataset has 50K labeled ones with equal number of positive and negative reviews, which is pre-divided into training and test sets, with each set containing 25K reviews. Identical to the qualitative analysis, we use the entire 100k reviews to train word embeddings from the target domain.

a.2 Sequence Labeling Setup

In the NER experiments on GENIA, we follow the standard preprocessing procedure [Muis and Lu2017], by splitting the dataset with the first 90% sentences for training and the rest for test. We keep cell line and cell type entities, and collapse all DNA, RNA, and protein subtypes into DNA, and RNA , and protein, respectively, and remove all other entities. As a result, there are five entities remaining in the dataset888The preprocessing follows muis2017-EMNLP muis2017-EMNLP.. Note that, following [Yang et al.2017] setup, we also use the last 10% of training data as the development set to tune hyper-parameters of our model.

The ARK dataset has a standard split containing 1,000/327/500 tweets as training/development/test set, respectively.