Supervised Complementary Entity Recognition with Augmented Key-value Pairs of Knowledge

05/29/2017 ∙ by Hu Xu, et al. ∙ University of Illinois at Chicago 0

Extracting opinion targets is an important task in sentiment analysis on product reviews and complementary entities (products) are one important type of opinion targets that may work together with the reviewed product. In this paper, we address the problem of Complementary Entity Recognition (CER) as a supervised sequence labeling with the capability of expanding domain knowledge as key-value pairs from unlabeled reviews, by automatically learning and enhancing knowledge-based features. We use Conditional Random Field (CRF) as the base learner and augment CRF with knowledge-based features (called the Knowledge-based CRF or KCRF for short). We conduct experiments to show that KCRF effectively improves the performance of supervised CER task.



There are no comments yet.


page 1

page 2

page 3

page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

Aspect extraction (or opinion target extraction) is an important task in sentiment analysis (Pang et al., 2002) on product reviews (Hu and Liu, 2004; Popescu and Etzioni, 2007; Liu, 2015). Besides opinion targets about the reviewed product itself (e.g., the “screen” in “iPhone’s screen is great.”), products that are compatible or incompatible with the reviewed product are also important opinion targets. Extracting those kinds of opinion targets are studied as Complementary Entity Recognition (CER) in (Xu et al., 2016b, a). For example, if the sold product is a tablet stand, then the “iPhone” in “This tablet stand works with my iPhone” is a complementary entity. Note that CER highly relies on entity’s contextual information (e.g., the word “works” in the previous example) and such information can be domain dependent. For example, “holds” in “This tablet stand holds my iPhone well” is a context verb particular for tablet stand. A traditional supervised method like Conditional Random Field (CRF) (Lafferty et al., 2001) may have good precision on such extraction yet suffer from low recall due to unseen context words appear in the test data but not in the training data. To solve this problem, instead of using supervised method, (Xu et al., 2016b) uses an unsupervised method by leveraging manually-crafted high precision dependency rules (Bach and Badaskar, 2007; Culotta and Sorensen, 2004; Bunescu and Mooney, 2005; Joshi and Penstein-Rosé, 2009) to expand (bootstrap) context words as knowledge on a large amount of unlabeled data and combine those context words with another set of manually-crafted high recall dependency rules for CER. However, crafting dependency rules for both context words and CER can be time-consuming and such rules may be domain dependent and subject to change for new domains.

To benefit from both the supervised and unsupervised methods, we consider to automatically learn patterns for both CER and knowledge expansion (of context words) from training data and expand context word knowledge on unlabeled data. So when making predictions on the test data, the model can leverage more contextual knowledge from unlabeled data to make better predictions. Or put it another way, we wish the prediction behavior of a supervised model can change after training when it sees more unlabeled data. This framework is inspired by the lifelong sequence labeling method proposed in (Shu et al., 2017, 2016). However, we do not expand knowledge for lifelong learning here and we make one step further: we automatically learn knowledge-based features and knowledge (or context words) as key-value pairs rather than manually crafting them. We use CRF as the base learner and augment CRF with knowledge-based features (a modified dependency relation) that are automatically learned from the training data. The augmented CRF is called Knowledge-based CRF (KCRF).

The proposed method has the following steps:
Pre-training We first train a CRF as a traditional sequence labeling training process using hand-crafted features, including primitive features (defined later) such as dependency relation based features. Then we automatically select from those primitive features as knowledge-based features to build a group of key-value pairs as initial knowledge, where keys are selected feature types and values are feature values (e.g., context words) appear in the training data.
Knowledge-based Training Then we train a Knowledge-based CRF (KCRF) based on the initial knowledge. So KCRF knows which features (as keys) can be used to expand knowledge (get more values for the same key).
Knowledge Expansion We expand the values in initial knowledge by iteratively collecting reliable knowledge from reliable predictions on plenty of unlabeled reviews. Experiments demonstrate that the expanded knowledge is effective in predicting test data.

2 Preliminaries

We briefly review the terms used throughout this paper. We use dependency relations as the major type of knowledge-based features since a dependency relation associates one word (current word) with another word (context word), which can be viewed as a piece of context knowledge.

Definition 1 (Dependency Relation)

A dependency relation is a typed relation between two words in a sentence with the following format:

where type is the type of a dependency relation, gov is the governor word, govpos is the POS (Part-Of-Speech) tag of the governor word, dep is the dependent word and deppos is the POS tag of the dependent word.

Definition 2 (Dependency Feature)

A dependency feature for the -th word is a simplified dependency relation with the following attributes:

where role can be either “GOV” or “DEP” indicating whether the -th word is a governor word or a dependent word; type is the type of the original dependency relation; gov/dep is the other word associated with the -th word via the original dependency relation and govpos/deppos is the POS tag of the other word.

Note here we omit the -th word, its POS-tag in a dependency relation and define them as separate features since they are the same for all dependency features of the -th word.

Definition 3 (Primitive Feature)

A primitive feature can be either a dependency feature or a current word feature (taking current word as a feature). Primitive features are used to generate knowledge base.

Definition 4 (Knowledge Base)

A knowledge base is a set of key-value pairs , where is the knowledge type and is the knowledge value belonging to that type. The same may have multiple knowledge values. We further define separate knowledge bases for each tag , where is the set of output labels for sequence labeing and .

Definition 5 (Knowledge-based Feature)

A knowledge-based feature is defined based on a knowledge type in a knowledge base. We use

to denote an index about a knowledge-based feature in a feature vector

. So indicates that the -th feature of the -th word is a knowledge-based feature of type with some found in KB. We use to denote all knowledge types in KB.

A primitive feature can generate a knowledge-based feature in the form of . Current word feature has a corresponding knowledge type and takes the current word as the knowledge value (e.g., we use to indicate “phone” is in the knowledge base as type [WORD] ). Dependency features have a corresponding knowledge type , which is similar to a dependency feature. The gov/dep part (the other word related to the current word in a dependency relation) is considered as the knowledge value . For example, if K={[WORD], [DEP, nmod:with, VBZ], [GOV, nmod:poss, PRP$]} and we have knowledge value “phone” and “works” for the first two types, we may have KB={([WORD], “phone”), ([DEP, nmod:with, VBZ], “works”) }. We describe how to automatically obtain all knowledge types and how to get initial knowledge values in the next section.

3 Pre-training

The role of pre-training is to identify knowledge types and initial knowledge values. It is important to obtain useful knowledge types and reliable knowledge values because some knowledge types may not help the prediction task and wrong knowledge values may be harmful to the performance of predictions. Fortunately, a trained CRF model can tell us which features are more useful for prediction and need to be enhanced with knowledge. The basic idea is to perform a traditional CRF training using primitive features and select knowledge-based features and initial knowledge values based on the weights of primitive features in the trained CRF model.

Let denote a feature vector of the -th word in an input sequence for pre-training. We use to denote an index about a primitive feature so means the -th word has a primitive feature (e.g., WORD=“phone” or (DEP, nmod:with, works, VBZ)) indexed by . We distinguish different feature functions according to the value of and the primitive features indexed by in . We care about the following type of feature function, which is a multiplication of 2 indicator functions:


It turns all combinations of primitive features and tag set into . Further we use a similar notation for the corresponding weight. A positive weight indicates a primitive feature indexed by has positive impact on predicting tag ; while a negative weight indicates a primitive feature indexed by has negative impact on predicting tag .

After training CRF using primitive features, we obtain the weights for and

, which are very important to know which primitive features are more useful for prediction and need to be expanded. We use entropy to measure the usefulness of a primitive feature. We compute the probability of each tag

for :


Based on Equation 2, we compute the entropy for a primitive feature indexed by :


The intuition of using entropy is that a salient primitive feature should favor some tags over the others so it has low entropy. We select primitive features that attain the maximum probability for tag and have entropies below (We set for ):


We obtain a set of primitive features indexed by and use it to generate for tag since each primitive feature can be interpreted as a . We group the same under to form the set and use the associated as initial knowledge value.

4 Knowledge-based Training

We train KCRF using knowledge-based features in this section. A knowledge-based feature simply tells whether a feature found in an example with a specified knowledge type has some values found in the current knowledge base (or KB). We use to denote the feature vectors with knowledge-based features for the -th word and use to denote a knowledge-based feature indexed by in . So indicates that the -th feature is a knowledge-based feature and the -th word has a knowledge with type and initial knowledge value found in KB:


For example, if ([DEP, nmod:with, VBZ], “works”) KB, the word “phone” has a dependency relation knowledge-based feature with type and and current word knowledge-based feature and . We denote the trained KCRF as and its parameters . It predicts on and generates probabilities for .

5 Knowledge Expansion

We perform sequence labeling on a large amount of unlabeled reviews under the same category as the target entity to expand the KB using . We assume that target entities under the same category share similar knowledge. Here the key point is to ensure the quality of the expanded knowledge since it is very easy to have harmful knowledge from unlabeled reviews without human supervision. We aggregate knowledge from reliable predictions on those reviews. To obtain a reliable prediction for the -th word, we marginalize over of other positions except as:


Then if a tag attains the maximum probability that is larger than a threshold: we consider it as a reliable prediction for tag at position . The knowledge-based features and potential knowledge values associated with such a reliable prediction are considered as candidate knowledge. We use as the set of candidate knowledge for tag . We further prune the knowledge since similar knowledge may appear in the knowledge base of another tag so this can make candidate knowledge from reliable predictions not reliable.

Input : , with ,
Output : , with updated KB
1 do
2       transform each into a sequence of knowledge-based feature vectors using KB for  do
3             use to predict use Equation 6 to compute for for  do
4                   for  do
5                         if  then
6                               add associated to for
7                         end if
9                   end for
11             end for
13       end for
14      for  do
16       end for
while return
Algorithm 1 Knowledge Expansion

Algorithm 1 is to maintain high-quality knowledge during expansion. We use to denote a set of unlabeled sequences and we transform to knowledge-based feature vectors based on current knowledge base KB (line 1). We apply KCRF and current KB on and get reliable prediction in line 1. We add associated knowledge in line 1) and prune it to get reliable knowledge and update KB in line 1-1. The whole process will stop once no reliable knowledge is available. Note that during knowledge expansion, KCRF itself is never re-trained.

6 Experimental Results

Product CRF(-)DR CRF CRF-Init KCRF
Stylus 0.5 0.54 0.52 0.75 0.50 0.60 0.84 0.64 0.73 0.66 0.84 0.74
Micro SD Card 0.63 0.51 0.56 0.89 0.44 0.59 0.87 0.57 0.69 0.77 0.70 0.74
Mouse 0.54 0.37 0.44 0.80 0.48 0.60 0.75 0.53 0.62 0.68 0.68 0.68
Tablet Stand 0.58 0.43 0.49 0.79 0.40 0.53 0.85 0.46 0.60 0.75 0.65 0.70
Keyboard 0.54 0.46 0.5 0.8 0.42 0.55 0.8 0.34 0.48 0.66 0.72 0.69
Notebook Sleeve 0.69 0.38 0.49 0.90 0.23 0.37 0.91 0.23 0.37 0.77 0.63 0.69
Compact Flash 0.75 0.61 0.67 0.92 0.46 0.62 0.89 0.51 0.65 0.82 0.73 0.77
Table 1: Comparison of different methods in precision, recall and F1-score

6.1 Dataset

We use the dataset111 from (Xu et al., 2016b), which includes 7 products. We take 50% reviews of the first 4 products as the training data for all methods that require supervised training. The remaining reviews of the first 4 products (for in-domain test) and all reviews of the last 3 products (for out-of-domain test) are test data. Similar to (Xu et al., 2016b), we also randomly select 6000 unlabeled reviews for each category from (McAuley et al., 2015) and use them as unlabeled reviews to expand knowledge.

6.2 Compared Methods

Since this paper proposes a supervised method on CER, we focus on the improvements of KCRF over CRF. We use CRFSuite222 as the base implementation of CRF.
CRF(-)DR: This is a very basic CRF without dependency relations as features to show that dependency relations are useful features. We use the following features: current word, POS-tags, 4 nearby words and POS-tags on the left and right, number of digits and whether current word has slash/dash.
CRF: This is the baseline with dependency relations as features. It is also the same learner as in the pre-training step of KCRF.
CRF-Init: This baseline uses the trained KCRF and initial KB directly on test data without knowledge expansion on unlabeled data.
KCRF: This is the proposed method that uses trained KCRF and initial KB to expand knowledge on unlabeled reviews under the same category as the target entity. We empirically set as the precisions of most predictions are around 0.8.

6.3 Evaluation Methods

We perform evaluation on each mention of complementary entities. We count true positive tp, false positive fp and false negative fn. For each sentence, one recognized complementary entity that is contained in the annotated complementary entities for that sentence is considered as one count for the tp; one recognized complementary entity that are not found contributes one count to the fp; any annotated complementary entity that can not be recognized contributes one count to the fn. Then we compute precision , recall and F1-score based on tp, fp and fn.

6.4 Result Analysis

From Table 1, we can see that KCRF performs well on F1-score. It significantly outperforms other methods on recall, which indicates that the expanded knowledge is helpful. CRF-Init performs better than CRF on most products, which indicates that knowledge-based features are better than primitive features in general. However, we notice that in order to get a higher recall, KCRF sacrifices its precision a lot. So how to further ensure that the expanded knowledge is of high quality to keep high precision is still an open problem.

The performance of KCRF does not drop much for the last 3 products even though we do not have any training data for those products. This is because KCRF can utilize unlabeled data to expand knowledge about the last 3 products separately from the knowledge of the first 4 products. One intuitive example is that “work” can be a frequent general verb knowledge that exists in the training data for some verb related knowledge type. Then later KCRF expands such a verb to other domain-specific verbs like “insert” for Compact Flash that does not have training data.

7 Conclusion

In this paper, we propose a supervised method for Complementary Entity Recognition (CER), called KCRF. KCRF can automatically identify knowledge-based features and expand knowledge as key-value pairs from plenty of unlabeled reviews. Experiments show that the expanded knowledge is useful in improving the performance of predictions, especially for products without training data.


  • Bach and Badaskar (2007) Nguyen Bach and Sameer Badaskar. 2007. A review of relation extraction. Literature review for Language and Statistics II .
  • Bunescu and Mooney (2005) Razvan C Bunescu and Raymond J Mooney. 2005. A shortest path dependency kernel for relation extraction. In

    Proceedings of the conference on human language technology and empirical methods in natural language processing

    . Association for Computational Linguistics, pages 724–731.
  • Culotta and Sorensen (2004) Aron Culotta and Jeffrey Sorensen. 2004. Dependency tree kernels for relation extraction. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, page 423.
  • Hu and Liu (2004) Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pages 168–177.
  • Joshi and Penstein-Rosé (2009) Mahesh Joshi and Carolyn Penstein-Rosé. 2009. Generalizing dependency features for opinion mining. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers. Association for Computational Linguistics, pages 313–316.
  • Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In

    Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 - July 1, 2001

    . pages 282–289.
  • Liu (2015) Bing Liu. 2015. Sentiment Analysis: Mining Opinions, Sentiments, and Emotions. Cambridge University Press.
  • McAuley et al. (2015) J. J. McAuley, R. Pandey, and J. Leskovec. 2015. Inferring networks of substitutable and complementary products. In KDD.
  • Pang et al. (2002) Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10. Association for Computational Linguistics, pages 79–86.
  • Popescu and Etzioni (2007) Ana-Maria Popescu and Orena Etzioni. 2007. Extracting product features and opinions from reviews. In Natural language processing and text mining, Springer, pages 9–28.
  • Shu et al. (2016) Lei Shu, Bing Liu, Hu Xu, and Annice Kim. 2016. Supervised opinion aspect extraction by exploiting past extraction results. arXiv preprint arXiv:1612.07940 .
  • Shu et al. (2017) Lei Shu, Hu Xu, and Bing Liu. 2017. Lifelong learning crf for supervised aspect extraction. arXiv preprint arXiv:1705.00251 .
  • Xu et al. (2016a) Hu Xu, Lei Shu, Jingyuan Zhang, and Philip S Yu. 2016a. Mining compatible/incompatible entities from question and answering via yes/no answer classification using distant label expansion. arXiv preprint arXiv:1612.04499 .
  • Xu et al. (2016b) Hu Xu, Sihong Xie, Lei Shu, and Philip S. Yu. 2016b. Cer: Complementary entity recognition via knowledge expansion on large unlabeled product reviews. In Proceedings of IEEE International Conference on Big Data.